Vdbench notes

Two types of supported testings.  Following table shows the configuration groups for these testings.

Storage device testing	File system testing
General - optional	General - optional
Host Definition(HD) - optional	Host Definition (HD) - optional
Storage Definition(SD)	File System Definition (FSD)
Workload Definition(WD)	File System Workload Definition(FSWD)
Run Definition (RD)	Run Definition (RD)

Different configuration groups may be specified in one single file or multiple files (use multiple -f options to specify all of them).

Comments: a line starting with '/', '#', or '*' is a comment.
Continuation: end a line with a comma plus a whitespace.

Output file
-o dirname+   # if dirname has existed, use dirname001.
-o dirname.tod    # the directory name will be dirname.yymmdd.hhmmss

General parameters

Host definition

Storage definition

Workload definition

Run definition

File System Definition

File System Workload Definition

Run definition for file system testing

Resources:
Official doc: http://downloads.sourceforge.net/project/vdbench/vdbench502.pdf?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fvdbench%2Ffiles%2F&ts=1315925901&use_mirror=superb-sea2

R notes

Construct a list of lists

a = c(1,2,3)
b = c(4,5,6)
c = list(a, b)
d = list(c, list(a))    # different from list(c, a)
e = list(d, name="gerald")

Append to a list:

mylist[[length(mylist) + 1]] = obj

Append to a vector:

append(vec, val)

 

Type information

typeof(var),  class(var)

Resources

• http://www.statmethods.net/input/datatypes.html

How to get network information in Linux

Following files/dirs can be directly read:

/proc/net/dev
/sys/class/net/<if_name>/
/sys/class/net/<if_name>/statistics

Tool netstat is your friend to get network-related information:

netstat -I  # display interface info
netstat -s  # display statistics info
netstat -r  # display routing info
netstat -tlnp  # display TCP listening info

Other tools: sar, ifconfig, iftop

How to see network card and disk speed

Sometimes, you may want to know the hardware speed limit for network interface cards and disks.

Network Interface Card

I use commands

dmesg | grep -i ethernet
dmesg | grep -i infiniband

The output looks like

Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2-1
igb 0000:0b:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:0b:00.1: Intel(R) Gigabit Ethernet Network Connection

and

mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
Registered RDS/infiniband transport

I also use command

/sbin/lspci

The output has following useful information

0b:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
0b:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
10:00.0 InfiniBand: Mellanox Technologies MT26418 [ConnectX VPI PCIe 2.0 5GT/s - IB DDR / 10GigE] (rev a0)

Disk

/sbin/lspci | grep -i ata

Other useful tools

mii-tools
hdparm

They need root privilege usually.

java.lang.LinkageError in Tomcat 6

Usually the cause of this error is you may unintentionally include following two jars into your war file. 
    el-api.jar jasper-el.jar
They are provided by tomcat 6.

In Tomcat 6, you will get following error:

java.lang.LinkageError: loader constraint violation: loader …

You can solve the problem by adding following snippet into your pom file.

<dependency>
        <groupId>org.apache.tomcat</groupId>
        <artifactId>el-api</artifactId>
        <version>[1,)</version>
        <scope>provided</scope>
</dependency>
<dependency>
        <groupId>org.apache.tomcat</groupId>
        <artifactId>jasper-el</artifactId>
        <version>[1,)</version>
        <scope>provided</scope>
</dependency>

Read this post for more information.

Hadoop datanode version control

Sometimes, when you upgrade your Hadoop, you may get following error in your namenode log:

Incompatible build versions: namenode BV = Unknown; datanode BV =

./common-0.21.0/src/saveVersion.sh generates package-info.java which includes version information.  The content looks like

@HadoopVersionAnnotation(version="0.21.1-SNAPSHOT", revision="1", branch="",
                         user="username", date="Mon Nov 15 12:28:49 EST 2010",
                         url="your_domain/path",
                         srcChecksum="a1aeb15b4854808d152989ba76f90fac")
saveVersion.sh is executed when you build Hadoop using ant.  It is specified in build.xml (target "init").

In Java code, class org.apache.hadoop.util.VersionInfo manages version.  It gets version information from package-info.java generated by saveVersion.sh.

In class org.apache.hadoop.hdfs.server.datanode.DataNode, method handshake checks whether build version is equal.  How build version is calculated is shown below.

public static String getBuildVersion(){
  return VersionInfo.getVersion() +
  " from " + VersionInfo.getRevision() +
  " by " + VersionInfo.getUser() +
  " source checksum " + VersionInfo.getSrcChecksum();
}
So, the quick solution is that you upgrade all installations of Hadoop on different nodes.

How to install user-provided jars to Hadoop

If you write a MapReduce program and compile it to a jar, you usually run it with following command:

./bin/hadoop jar your_jar_name

If you want to your jar loaded when Hadoop is started (e.g. you add a new service which should be initiated and started by Hadoop), you can follow steps shown below.

In file bin/hadoop-config.sh, you can find following snippet:

for f in $HADOOP_COMMON_HOME/hadoop-*.jar; do
  CLASSPATH=${CLASSPATH}:$f;
done

So only jars whose names starting with "hadoop-" are loaded by default.

Drop your jar to the directory where Hadoop is used, and change file bin/hadoop-config.sh to add

CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/your_jar_name

Debug/profile heap/gc in Java

HPROF

Profiler agent.
Examples:
java -agentlib:hprof=help
java -agentlib:hprof=heap=sites
java -agentlib:hprof=heap=dump
java -agentlib:hprof=cpu=samples

" By default, heap profiling information (sites and dump) is written out to java.hprof.txt (in ASCII) in the current working directory.

The output is normally generated when the VM exits, although this can be disabled by setting the “dump on exit” option to “n” ( doe=n). In addition, a profile is generated when Ctrl-\ or Ctrl-Break (depending on platform) is pressed. On Solaris OS and Linux a profile is also generated when a QUIT signal is received ( kill -QUIT pid). If Ctrl-\ or Ctrl-Break is pressed multiple times, multiple profiles are generated to the one file.  "

jmap

The jmap command-line utility prints memory related statistics for a running VM or core file.

Commands:

jmap -histo <pid>                          #show histogram of objects
jmap -dump:format=b,file=<file>    #dump heap in HPROF format (can be processed by jhat)

jstat

"The jstat utility uses the built-in instrumentation in the HotSpot VM to provide information on performance and resource consumption of running applications. "

show garbage collection info, class loading info, compilation info, etc.

visualgc

GUI to show results of jstat.

Java VisualVM

http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/index.html
command:  jvisualvm

"Java VisualVM is a tool that provides a visual interface for viewing detailed information about Java applications while they are running on a Java Virtual Machine (JVM), and for troubleshooting and profiling these applications."

JConsole

"This tool is compliant with Java Management Extensions (JMX). The tool uses the built-in JMX instrumentation in the Java Virtual Machine to provide information on the performance and resource consumption of running applications."

jhat (java heap analysis tool)

"The jhat tool provides a convenient means to browse the object topology in a heap snapshot. This tool was introduced in the Java SE 6 release to replace the Heap Analysis Tool (HAT). "

Command:

jhat <hprof_file_name>

Eclipse MAT

 

jdb

Misc.

"As of Java SE 5.0 update 7, the -XX:+HeapDumpOnOutOfMemoryError command-line option
tells the HotSpot VM to generate a heap dump when an OutOfMemoryError occurs (see
section 1.9).
As of Java SE 5.0 update 14, the -XX:+HeapDumpOnCtrlBreak command-line option tells the
HotSpot VM to generate a heap dump when a Ctrl-Break or SIGQUIT signal is received (see
section 1.10). "

Resources

http://www.oracle.com/technetwork/java/javase/index-137495.html

Eucalyptus notes

https://portal.futuregrid.org/tutorials/eucalyptus

Bundle images

http://open.eucalyptus.com/participate/wiki/modifying-prepackaged-image

http://open.eucalyptus.com/wiki/EucalyptusImageManagement_v1.6

 

Bundle running instance

http://megam.info/2010/04/28/euca-bundle-vol-solved/
http://open.eucalyptus.com/forum/how-create-snapshot-running-vm-instance-euca2ools-or-eucalyptus-web-interface

Basic commands

euca-bundle-vol  \
    --kernel <kernel_id> \
    --ramdisk <randisk_i> \
    -d <dir_to_put_bundled_img> \
    -p <prefix> \
    -r <arch>

euca-upload-bundle -b <bucket_name> -m <path_to_manifest_file>

euca-register <manifest_file>

 

Euca2tools and its dependency download

http://open.eucalyptus.com/downloads

http://eucalyptussoftware.com/downloads/releases/euca2ools-1.3.1-src-deps.tar.gz

http://eucalyptussoftware.com/downloads/releases/euca2ools-1.3.1.tar.gz

How to decommission nodes/blacklist nodes

HDFS

Put following config in conf/hdfs-site.xml:

<property>
  <name>dfs.hosts.exclude</name>
  <value>/full/path/of/host/exclude/file</value>
</property>

Use following command to ask HDFS to re-read host exclude file and decommission nodes accordingly.

./bin/hadoop dfsadmin -refreshNodes

MapReduce

Put following config in conf/mapred-site.xml

<property>
  <name>mapred.hosts.exclude</name>
  <value>/full/path/of/host/exclude/file</value>
</property>

Use following command to ask Hadoop MapReduce to reconfigure nodes.

./bin/hadoop mradmin -refreshNodes

Whitelist/Recommission

Also you can "whitelist" nodes. In other words, you can specify which nodes are allowed to connect to namenode/jobtracker. 

HDFS

Put following config in conf/hdfs-site.xml:

<property>
  <name>dfs.hosts</name>
  <value>/full/path/to/whitelisted/node/file</value>
</property>

Use following command to ask Hadoop to refresh node status to based on configuration.

./bin/hadoop dfsadmin -refreshNodes

MapReduce

Put following config in conf/mapred-site.xml

<property>
  <name>mapred.hosts</name>
  <value>>/full/path/to/whitelisted/node/file</value>
</property>

Use following command to ask Hadoop MapReduce to reconfigure nodes.

./bin/hadoop mradmin -refreshNodes

 

Support of -mradmin was added in 0.21.0. See JIRA issue https://issues.apache.org/jira/browse/HADOOP-5643 for details.