Saturday, June 26, 2010

Clear memory cache in Linux

Read this article

http://linux-mm.org/Drop_Caches

Basically, run commands
    sync
    sudo echo 3 > /proc/sys/vm/drop_caches

If you want to control "swappiness" (to what extent apps are swapped), read following posts:
http://kerneltrap.org/node/3000
http://lwn.net/Articles/92892/

Basic command: sudo echo 0-100 > /proc/sys/vm/swappiness

Sunday, June 06, 2010

Steve Huffman talks about lessons learned at Reddit

Very interesting talk. Helpful to those who want to build scalable distributed systems.

Video: http://vimeo.com/10506751
Transcript is here: http://carsonified.com/blog/dev/steve-huffman-on-lessons-learned-at-reddit/

One interesting lesson is lesson 3: open schema.
They combined relational data model and key-value store. Relational data model is powerful (in some sense) and can naturally represent real-world data models. However, as data set size grows, it seems that relational databases cannot scale up easily. As a result, many large companies develop their own storage systems - Amazon's Dynamo, Google's BigTable, Cassandra, etc. Some use key-value model which seems to be able to scale better than relational model. To change schema does NOT incur much overhead. Different rows/entities can have different number of columns/attributes.

Another post on caching in web app: http://www.mysqlperformanceblog.com/2010/05/19/beyond-great-cache-hit-ratio/

Saturday, March 27, 2010

Create a file of a given size on linux

Let's say you want to create a file whose size is 1K. You can achieve that by using either of following two commands:

    dd if=/dev/zero of=file bs=1024 count=1

    dd of=file bs=1024 count=0 seek=1

In my test, the generated files are different. The second command generates a file with holes. You can try command stat file. The output in my machine is:

  File: `bigfile'
  Size: 1024            Blocks: 0          IO Block: 4096   regular file

The number of allocated blocks is 0 although size of the file is 1024.
Note: if sparse file is not supported, it may write 1024K zeros to the file.

As a user, you don't need to worry about whether the file is sparse or not. It will grow as needed.

Thursday, March 25, 2010

Ubuntu package downgrade

I tried to use some Karmic sources for my Intrepid. Of course, this is bad. The reason I had to do it was the package of new version I needed only exist in Karmic repository.
After installing a package, suddenly I got the following error when I tried to use gvim
"gvim: error while loading shared library: libgailutil.so.18: cannot open shared object file: No such file or directory"

Obviously, the old libgail18 was removed which is needed by Intrepid. Because I used unmatched sources, the apt-get did not detect any problem.

I tried to install libgail18 using command
    sudo apt-get install libgail18
It did not work and the error message is

Package libgail18 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  libgtk2.0-0
E: Package libgail18 has no installation candidate

Finally, I figured out the cause is package libgtk2.0-0, libgtk2.0-0-common, etc. These packages are too new for Intrepid. They are for Karmic. So I tried to remove these packages using command
    sudo apt-get remove libgtk2.0-0 libgtk2.0-0-common etc
However, apt-get always gave error messages which told me those packages were needed by lots of other packages. It is obviously true. Also I tried command
    sudo apt-get install --reinstall libgtk2.0-0
It also did not work. The error message is the package cannot be found. Again the cause I think was the those installed packages were too new and did not match version of ubuntu.

It turned out that I need to use dpkg command
    dpkg --remove –-depends libgtk2.0-0 libgtk2.0-0-common etc

Then use following command to fix the broken dependencies:
    sudo apt-get install –f
libgtk2.0-0, libgtk2.0-0-common of correct versions are downloaded and installed.

Probably following commands are needed to clean up:
    sudo dpkg --configure –a
    dpkg-reconfigure

One big difficult I encountered was it was hard to collect detailed log about which files are created/updated/removed by which package during installation. It makes much easier to locate those packages which remove needed files.

SSH, XAuth and X11 Forward after user switch at remote site

Problem

User A connects to server S using ssh. X11 forward is enabled using –Y option. Then X11 should work smoothly.
User A connects to server S using ssh. X11 forward is enabled. Then user A switches to another User B (using command su B or ssh localhost –l B). After that, X11 forward won't work. The error should look like
"SSH gateway: X11 authentication failed. Error: Can't open display:" or
"Error: Can't open display:".
Readers may ask why user A does not directly connect to server S as user B given user A knows password of user B. The reason is that sometimes user B is a restricted user account so that he cannot log in remotely.

Solution

  1. Run command
        echo ${DISPLAY}
    Sample result:
        localhost:11.0
  2. Command: xauth list
    The output should be like:   
    your_host_name/unix:11  MIT-MAGIC-COOKIE-1  d1e63de6fd7bc3800d868c3b64ca4531
    your_host_name/unix:0  MIT-MAGIC-COOKIE-1  e044d47b672dcade1362cd632236f919
    your_host_name/unix:10  MIT-MAGIC-COOKIE-1  2aa3bc47d1c209fd06577f4b45f83383
    Pick the entry with the same display number as the output in step 1)
    In this example, display number is 11, so the entry we pick is
    your_host_name/unix:11  MIT-MAGIC-COOKIE-1  d1e63de6fd7bc3800d868c3b64ca4531
  3. switch to another user using either of the following ways
    1) su user_name
    or su – user_name
    read "man su" for difference between these two commands.
    2) ssh localhost –l user_name
  4. In step 3), if you ran command "su – user_name" or "ssh localhost –l user_name", you should run command
        export DISPLAY=localhost:11.0
    Value of DISPLAY should be the same as the output in step 1).

    add the entry obtained in step 2) to the .Xauthority file. You can either add it manually to the file or use tool xauth to do it. The way to use xauth to add an entry:
        xauth add :11 . d1e63de6fd7bc3800d868c3b64ca4531
    The cookie string (long string) must match the one in step 2). The display number (:11) must match the result in step 1)
  5. Try command
        xclock

Or you can combine step 1), 2), 3) and 4) into one long command:

    (tmpfile=/tmp/xauth_tmp_entry; \
    xauth extract ${tmpfile} :$(echo $DISPLAY|cut -d : -f 2 ); \
    chmod a+r ${tmpfile}; \
    su user_name -c "xauth merge ${tmpfile}"; \
    rm ${tmpfile} )

Note: replace user_name with the real target user name.

Disadvantage

Each time the user reconnects the remote machine using ssh, the whole process described above must be redone :-( The reason is that sshd may choose another display number and cookie value.

How SSH X authorization works?

From ssh manual:

"ssh will also automatically set up Xauthority data on the server machine.  For this purpose, it
will generate a random authorization cookie, store it in Xauthority on the server, and verify
that any forwarded connections carry this cookie and replace it by the real cookie when the
connection is opened.  The real authentication cookie is never sent to the server machine (and
no cookies are sent in the plain)."

http://blogs.gnome.org/markmc/2005/02/25/ssh-x-forwarding-and-xauth/

Every time a user connects to a remote server using ssh, a proxy X server is created. And that X server is used by the sshd process (a new process is forked each time a new connection comes in). The process is like:

  user ---> server ---> fork a new process, 
create a proxy X server
create pseudo terminal, etc. |
                                       |
                                       V
                        a program that needs X is used
                                       |
                                       |
                                       V
  local display <--- verify <--- the X output is forwarded by sshd to client

It seems that after a user connects to a remote server using ssh, another proxy X server is not created when the user ssh to localhost or 127.0.0.1.

Sunday, February 21, 2010

Android Development Tips

Logging

http://groups.google.com/group/android-ndk/browse_thread/thread/d6c5e8f25f1ee543

#include <android/log.h> 
__android_log_print(ANDROID_LOG_INFO, "your_tag", "fmt string", parameter1, parameter2);

Then you will see it in logcat output.

Read file build/platforms/android-1.5/common/include/android/log.h in ndk for more info.

Syntax Highlighter syntax conversion

http://alexgorbatchev.com/wiki/SyntaxHighlighter 

I found that usage pattern was changed in recent releases.

Original sample usage:

<pre class="java:nocontrols" name="code">
    your code
</pre>

New sample usage:

<pre class="brush:java">
    your code
</pre>

Change your posts to use new version of Syntax Highlighter:

Invoke following command in vim:
%s/<pre \(.*\)class="\(\w\+\)*\(:\w\+\)*" \(.*\)>/<pre class="brush:\2" \1 \4>/g

Google Blogger/Blogspot line break conversion

Google Blogger/Blogspot new line conversion

Google blogger/blogspot provides an option to let users decide whether hard-returns in the post are converted to <br/>

Set it by clicking:
    Your blog admin page –> Settings -> Formatting -> Convert line breaks

Originally, I set it to "yes". Latter I decided to manually insert <br/> instead of using automatical hard-return conversion. So I set the option to "no". As a result, my previous posts were messed up because different lines were concatenated into a single line.

I used vim to convert those posts to new format.

Append <br/> to each line
    %s/$/<br\/>/g

Sometimes, you don't want to append new lines to block-level elements such as div, ol.
Use following vim commands:

    %s/\(<\/div>\s*\)\@<!$/<br\/>/g 
    %s/\(<\/ul>\s*\)\@<!$/<br\/>/g 
    %s/\(<\/ol>\s*\)\@<!$/<br\/>/g 
    %s/\(<\/dl>\s*\)\@<!$/<br\/>/g
    ......

Vim regular expression: http://vimdoc.sourceforge.net/htmldoc/pattern.html#pattern-overview

Another option is to write a script to download posts, adjust returns/newlines, and publish them.

Monday, January 11, 2010

Eucalyptus Code Reading


Some notes on Eucalyptus source



Architecture

See: http://open.eucalyptus.com/wiki/EucalyptusInstall_v1.6

Clound Controller (CLC)

The most complex component in terms of implementation and nuber of lines of code. Implemented in Java and code is located under directory *clc*. It seems it supports both ant and maven build.

Packages used:
  • mule
Modules:
  • authentication
  • bootstrap
    Built to generate an executable named eucalyptus-cloud which can be used to start up the cloud.
  • cloud
  • cluster-manager
    /edu/ucsb/eucalyptus/cloud/cluster
    Allocator.java
    ClusterAllocator.java
    ClusterEndpoint.java
    ClusterEnvelope.java

    QueuedEvent.java
    QueuedLogEvent.java

    Reservation.java
    Reservations.java
    VmInstance.java
    VmInstances.java

    VmAllocationTransaction.java
    VmTypeAvailability.java
    VmTypeVerify.java
    VmTypes.java

    Callbacks
    UnassignAddressCallback.java
    AssignAddressCallback.java
    ConfigureNetworkCallback.java
    ConsoleOutputCallback.java
    MultiClusterCallback.java
    QueuedEventCallback.java
    RebootCallback.java
    StartNetworkCallback.java
    StopNetworkCallback.java
    TerminateCallback.java
    VmRunCallback.java
    VolumeAttachCallback.java
    VolumeDetachCallback.java

    /edu/ucsb/eucalyptus/cloud/cluster
    AddressManager.java
    CreateVmInstances.java
    StateSnapshot.java
    SystemState.java
    VmAdmissionControl.java
    VmControl.java
    VmMetadata.java
    VmReplyTransform.java


    /com/eucalyptus/cluster
    Cluster.java
    ClusterBootstrapper.java
    ClusterMessageQueue.java
    ClusterNodeState.java
    ClusterState.java
    ClusterThreadGroup.java
    Clusters.java
    Networks.java

    /com/eucalyptus/cluster/handlers
    AbstractClusterMessageDispatcher.java
    AddressStateHandler.java
    ClusterCertificateHandler.java
    LogStateHandler.java
    NetworkStateHandler.java
    ResourceStateHandler.java
    VmStateHandler.java
  • configuration
  • core
    edu/ucsb/eucalyptus/cloud/entities: Includes basic beans which represent various objects in the system (e.g. vmtype, user, bucket, image)
  • distribution
  • dns
  • group-manager
  • image-manager
  • interface
  • key-manager
  • msgs
    com/eucalyptus/bootstrap:
    Depends, Provides, Resource
    Component: each component has a resource provider.
    ResourceProvider
    Bootstrapper
    SystemBootstrapper: includes methods to load and start all bootstrappers contained in jars.
    BootstrapFactory: provides static methods to initialize bootstrappers, configuration resources and resource providers.
    com/eucalyptus/event
    AbstractNamedRegistry.java
    ClockTick.java: clock tick event
    Event.java: event abstraction
    EventListener.java: event listener abstration
    GenericEvent.java: adds message to an event
    ListenerRegistry.java: a event listener registry
    ReentrantListenerRegistry.java
    StateEvent.java: represents an event that has state.
    StatefulNamedRegistry.java
    SystemClock.java: periodically triggers clocktick event.
    com/eucalyptus/util

    edu/ucsb/eucalyptus/msgs/
    Files under this directory defines various message types.
    Configuration.groovy
    DNS.groovy
    EventRecord.java
    Messages.groovy
    StorageController.groovy
    Unimplemented.groovy
    VPN.groovy
    VmAddresses.groovy
    VmBlockDevice.groovy
    VmControl.groovy
    VmImages.groovy
    VmKeys.groovy
    VmLocation.groovy
    VmNetwork.groovy
    VmSecurity.groovy
    Walrus.groovy
  • storage-common
    edu/ucsb/eucalyptus/cloud/ws:
    ChunkedDataFile.java
    CompressedChunkedFile.java

    edu/ucsb/eucalyptus/storage
    StorageManager.java: interface for S3?

    edu/ucsb/eucalyptus/storage/fs:
    FileIO.java
    FileReader.java
    FileSystemStorageManager.java: impl of StorageManager.
    FileWriter.java

  • storage-controller
    com/eucalyptus/bootstrap
    BlockStorageBootstrapper: bootstrappers for block storage.
    edu/ucsb/eucalyptus/cloud/ws
    BlockStorage: EBS (to read). It uses StorageManager from storage-common as underlying impl.
    StorageEventListener
    /edu/ucsb/eucalyptus/storage/
    AOEManager.java: implements StorageExportManager
    BlockStorageChecker.java
    BlockStorageManagerFactory.java: factory for LVM2Manager.
    LVM2Manager.java: impls LogicalStorageManager
    LogicalStorageManager.java: interface for logical volume management
    StorageExportManager.java: interface for export of volume
    /edu/ucsb/eucalyptus/ic
    StorageController.java: handles messages.

  • walrus (S3 impl)
    edu/ucsb/eucalyptus/cloud/ws:
    ObjectReader.java
    TorrentClient.java
    TorrentCreator.java
    Torrents.java
    Tracker.java
    WalrusControl.java
    WalrusEventListener.java: event listener. Currently, listens to StopComponentEvent.
    WalrusImageManager.java
    WalrusManager.java
    WalrusBlockStorageManager.java
    WalrusStatistics.java
    edu/ucsb/eucalyptus/ic:
    Walrus.java: handles messages.

  • wsstack
  • www

Cluster Controller (CC)

It interacts with two parts: cloud controller and node controllers. Its interface is described in file wsdl/eucalyptus_cc.wsdl. Axis2/C is used to generate C stub functions from wsdl.
It has two parts in the code: client and server. It provides services using SOAP-based web service.

client

It relies on
  • generated CC client stub functions from *wsdl/eucalyptus_cc.wsdl*
Code:
cluster/CCclient.c
cluster/cc-client-marshal-adb.c (main file)
cluster/cc-client-marshal.h
Client functions make use of stub functions generated from wsdl using axis2/C to construct SOAP request messages. The generated tool (by make) can be used to interact with cluster controller services.

server

When you build this part, a shared object is generated and deployed to Axis2/C environment. The axis2/C service name is "EucalyptusCC".
To fulfill the requests, sometimes it needs to interact with node controller (e.g. get information of the instances running on a specific node). So this part relies on generated stubs from both wsdl/eucalyptus_cc.wsdl and wsdl/eucalyptus_nc.wsdl. It acts as a server to provide services for external parties while it also acts as a service client to interact with other services.
So, it relies on
  • generated NC client stub functions *wsdl/eucalyptus_nc.wsdl*
  • generated CC server stub functions *wsdl/eucalyptus_cc.wsdl*
Code:
cluster/handlers.c
cluster/handlers.h
cluster/server-marshal.c
cluster/server-marshal.h
Functions in *handlers.c* do the real work when a service request is received. If necessary, it sends service requests to node controller.
File *server-marshal.c* extracts the request parameters using generated stub functions and invokes corresponding functions in file *handlers.c*. So it acts like a mediator which converts web service request to function name and parameters.

Storage Controller

Provides storage services for other parts. However, the form is not web service, just functions. Other parts directly call storage controller functions.
Code
  • storage/walrus.c, storage/walrus.h
    Provides functions:
    int walrus_object_by_url (const char * url, const char * outfile, const int do_compress);
    int walrus_object_by_path (const char * path, const char * outfile, const int do_compress);
    int walrus_image_by_manifest_url (const char * url, const char * outfile, const int do_compress);
    int walrus_image_by_manifest_path (const char * manifest_path, const char * outfile, const int do_compress);
    int walrus_verify_digest (const char * url, const char * digest_path);
    These functions are used to interact with walrus services (using REST interfaces instead of SOAP-based interfaces).
  • storage/storage.c, storage/storage.h
    provides higher-level abstraction which eases interaction of application with storage system.
    It is used by Node Controller and Cluster controller(? it is included as header file, but seems none of its functions are used).

A executable named Wclient is generated when you "make all".

Node Controller (NC)

Node controller manages the instances running on a physical machine. It also provides services in the form of web services.
Its interface is described in file wsdl/eucalyptus_nc.wsdl

Server

Provides various services to other parts.
Code:
node/handlers.c
node/handlers_default.c
node/handlers.h
node/handlers_kvm.c
node/handlers_xen.c
node/server-marshal.c
node/server-marshal.h
Similar to implementation of cluster controller, files *handlers.c* does the real work to interact with virtual machine monitoring. Currently, both kvm and xen are supported. The concrete work involved to interact with a specific type of virtualization implementation is delegated to functions in file handlers_kvm.c, handlers_xen.c or handlers_default.c based on the user configuration. Under the hood, libvirt is used to interact with virtualization implementation.
server-marshal.c just extracts information from the incoming service requests and invokes corresponding functions in file *handlers.c*.
When a service needs to interact with storage system (e.g. download an image, create an image), it makes use of Storage Controller.

Client

Access services provided by NC server.
Code:
node/client-marshal-adb.c
node/client-marshal.h
node/client-marshal-local.c
node/NCclient.c
NCclient.c contains the main function. It parses the command and dispatch the requests to corresponding functions in file *client-marshal-adb.c* or *client-marshal-local.c*.
If the service is not local, client-marshal-adb.c is used. Basically the request is transformed to SOAP message and sent to remote NC service.
If the service is local, client-marshal-local.c is used. The corresponding implementation functions are directly invoked without hassle of web service stuff.

Misc.

network

See http://open.eucalyptus.com/wiki/EucalyptusNetworking_v1.6 for networking configuration.
In managed mode,
Manipulates the dhcpd configuration file and use "brctl" command from package *bridge-utils*. Also it supports VLAN.

gatherlog

Another sets of services described in file wsdl/eucalyptus_gl.wsdl. Code is located under directory *gatherlog*.

util

tools



Friday, January 08, 2010

Do you know Windows 7 GodMode?

See this post for details:

http://www.osnews.com/story/22691/Activate_Windows_7_s_Hidden_God_Mode_

Just try:

  1. Create a new folder
  2. Name it: GodMode.{ED7BA470-8E54-465E-825C-99712043E01C}
  3. That's it