Hadoop datanode version control

Sometimes, when you upgrade your Hadoop, you may get following error in your namenode log:

Incompatible build versions: namenode BV = Unknown; datanode BV =

./common-0.21.0/src/saveVersion.sh generates package-info.java which includes version information.  The content looks like

@HadoopVersionAnnotation(version="0.21.1-SNAPSHOT", revision="1", branch="",
                         user="username", date="Mon Nov 15 12:28:49 EST 2010",
                         url="your_domain/path",
                         srcChecksum="a1aeb15b4854808d152989ba76f90fac")
saveVersion.sh is executed when you build Hadoop using ant.  It is specified in build.xml (target "init").

In Java code, class org.apache.hadoop.util.VersionInfo manages version.  It gets version information from package-info.java generated by saveVersion.sh.

In class org.apache.hadoop.hdfs.server.datanode.DataNode, method handshake checks whether build version is equal.  How build version is calculated is shown below.

public static String getBuildVersion(){
  return VersionInfo.getVersion() +
  " from " + VersionInfo.getRevision() +
  " by " + VersionInfo.getUser() +
  " source checksum " + VersionInfo.getSrcChecksum();
}
So, the quick solution is that you upgrade all installations of Hadoop on different nodes.

How to install user-provided jars to Hadoop

If you write a MapReduce program and compile it to a jar, you usually run it with following command:

./bin/hadoop jar your_jar_name

If you want to your jar loaded when Hadoop is started (e.g. you add a new service which should be initiated and started by Hadoop), you can follow steps shown below.

In file bin/hadoop-config.sh, you can find following snippet:

for f in $HADOOP_COMMON_HOME/hadoop-*.jar; do
  CLASSPATH=${CLASSPATH}:$f;
done

So only jars whose names starting with "hadoop-" are loaded by default.

Drop your jar to the directory where Hadoop is used, and change file bin/hadoop-config.sh to add

CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/your_jar_name

Debug/profile heap/gc in Java

HPROF

Profiler agent.
Examples:
java -agentlib:hprof=help
java -agentlib:hprof=heap=sites
java -agentlib:hprof=heap=dump
java -agentlib:hprof=cpu=samples

" By default, heap profiling information (sites and dump) is written out to java.hprof.txt (in ASCII) in the current working directory.

The output is normally generated when the VM exits, although this can be disabled by setting the “dump on exit” option to “n” ( doe=n). In addition, a profile is generated when Ctrl-\ or Ctrl-Break (depending on platform) is pressed. On Solaris OS and Linux a profile is also generated when a QUIT signal is received ( kill -QUIT pid). If Ctrl-\ or Ctrl-Break is pressed multiple times, multiple profiles are generated to the one file.  "

jmap

The jmap command-line utility prints memory related statistics for a running VM or core file.

Commands:

jmap -histo <pid>                          #show histogram of objects
jmap -dump:format=b,file=<file>    #dump heap in HPROF format (can be processed by jhat)

jstat

"The jstat utility uses the built-in instrumentation in the HotSpot VM to provide information on performance and resource consumption of running applications. "

show garbage collection info, class loading info, compilation info, etc.

visualgc

GUI to show results of jstat.

Java VisualVM

http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/index.html
command:  jvisualvm

"Java VisualVM is a tool that provides a visual interface for viewing detailed information about Java applications while they are running on a Java Virtual Machine (JVM), and for troubleshooting and profiling these applications."

JConsole

"This tool is compliant with Java Management Extensions (JMX). The tool uses the built-in JMX instrumentation in the Java Virtual Machine to provide information on the performance and resource consumption of running applications."

jhat (java heap analysis tool)

"The jhat tool provides a convenient means to browse the object topology in a heap snapshot. This tool was introduced in the Java SE 6 release to replace the Heap Analysis Tool (HAT). "

Command:

jhat <hprof_file_name>

Eclipse MAT

 

jdb

Misc.

"As of Java SE 5.0 update 7, the -XX:+HeapDumpOnOutOfMemoryError command-line option
tells the HotSpot VM to generate a heap dump when an OutOfMemoryError occurs (see
section 1.9).
As of Java SE 5.0 update 14, the -XX:+HeapDumpOnCtrlBreak command-line option tells the
HotSpot VM to generate a heap dump when a Ctrl-Break or SIGQUIT signal is received (see
section 1.10). "

Resources

http://www.oracle.com/technetwork/java/javase/index-137495.html

Eucalyptus notes

https://portal.futuregrid.org/tutorials/eucalyptus

Bundle images

http://open.eucalyptus.com/participate/wiki/modifying-prepackaged-image

http://open.eucalyptus.com/wiki/EucalyptusImageManagement_v1.6

 

Bundle running instance

http://megam.info/2010/04/28/euca-bundle-vol-solved/
http://open.eucalyptus.com/forum/how-create-snapshot-running-vm-instance-euca2ools-or-eucalyptus-web-interface

Basic commands

euca-bundle-vol  \
    --kernel <kernel_id> \
    --ramdisk <randisk_i> \
    -d <dir_to_put_bundled_img> \
    -p <prefix> \
    -r <arch>

euca-upload-bundle -b <bucket_name> -m <path_to_manifest_file>

euca-register <manifest_file>

 

Euca2tools and its dependency download

http://open.eucalyptus.com/downloads

http://eucalyptussoftware.com/downloads/releases/euca2ools-1.3.1-src-deps.tar.gz

http://eucalyptussoftware.com/downloads/releases/euca2ools-1.3.1.tar.gz

How to decommission nodes/blacklist nodes

HDFS

Put following config in conf/hdfs-site.xml:

<property>
  <name>dfs.hosts.exclude</name>
  <value>/full/path/of/host/exclude/file</value>
</property>

Use following command to ask HDFS to re-read host exclude file and decommission nodes accordingly.

./bin/hadoop dfsadmin -refreshNodes

MapReduce

Put following config in conf/mapred-site.xml

<property>
  <name>mapred.hosts.exclude</name>
  <value>/full/path/of/host/exclude/file</value>
</property>

Use following command to ask Hadoop MapReduce to reconfigure nodes.

./bin/hadoop mradmin -refreshNodes

Whitelist/Recommission

Also you can "whitelist" nodes. In other words, you can specify which nodes are allowed to connect to namenode/jobtracker. 

HDFS

Put following config in conf/hdfs-site.xml:

<property>
  <name>dfs.hosts</name>
  <value>/full/path/to/whitelisted/node/file</value>
</property>

Use following command to ask Hadoop to refresh node status to based on configuration.

./bin/hadoop dfsadmin -refreshNodes

MapReduce

Put following config in conf/mapred-site.xml

<property>
  <name>mapred.hosts</name>
  <value>>/full/path/to/whitelisted/node/file</value>
</property>

Use following command to ask Hadoop MapReduce to reconfigure nodes.

./bin/hadoop mradmin -refreshNodes

 

Support of -mradmin was added in 0.21.0. See JIRA issue https://issues.apache.org/jira/browse/HADOOP-5643 for details.

Japan earthquake GPS data visualization gadget

I made a gadget version of QuakeSim Japan earthquake data visualization portal. It shows data (longitude, latitude and height) collected by GPS stations during Japan earthquake.

You can click http://www.google.com/ig/adde?synd=open&source=ggyp&moduleurl=hosting.gmodules.com%2Fig%2Fgadgets%2Ffile%2F105322631994749779353%2Fquakesim-japan.xml to add it to your iGoogle. After it is added, maximize it by clicking the icon near top right corner of the gadget

Link for the gadget is

http://www.google.com/ig/directory?url=hosting.gmodules.com%2Fig%2Fgadgets%2Ffile%2F105322631994749779353%2Fquakesim-japan.xml

Thank Xiaoming Gao for providing service pages.

Install ns2 (ns-2.33) on Ubuntu Maverick

Install prerequisite:

sudo apt-get install \  
    tcl tcl-dev \ 
    libotcl1 libotcl1-dev  \ 
    tclcl-dev tclcl \ 
    tk tk-dev

./configure failed complaining that some tcl, tk related files cannot be found. It turns out those packages have been installed but file locations are different than what the configure script expects. Following is a fix.

You need to change two variables in file configure : TCL_TCL_PLACES and TK_TCL_PLACES.

Add
    /usr/share/tcltk/tcl$TCL_VERS \
    /usr/share/tcltk/tcl$TCL_HI_VERS
to variable TCL_TCL_PLACES.

Add
    /usr/share/tcltk/tk$TK_HI_VERS \
    /usr/share/tcltk/tk$TK_VERS"
to variable TK_TCL_PLACES.

execute ./configure

Official page: http://www.isi.edu/nsnam/ns/ns-build.html

Hadoop tips

• Change logging level
• For each daemon, there is a service at http://daemon_address:port/logLevel through which you can get and set logging level.
• Use command line
    hadoop daemonlog -getLevel daemon_address:port fullQualifiedClassName
  hadoop daemonlog -setLevel daemon_address:port fullQualifiedClassName logLevel
• Permanent change
  Change file log4j.properties. Example
      log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG
      log4j.logger.org.apache.hadoop.mapred.TaskTracker=DEBUG
      log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG
  
• Commission and decommission nodes
  Following four config parameters are related:
  dfs.hosts
  dfs.hosts.exclude
  mapreduce.jobtracker.hosts.filename (mapred.hosts for old version)
  mapreduce.jobtracker.hosts.exclude.filename (mapred.hosts.exclude for old version)
  
  For HDFS, execute "hadoop dfsadmin -refreshNodes" after you change the include file or exclude file.
  From the mailing list, I know "mradmin -refreshNodes was added in 0.21".  So for MapReduce, you can use "hadoop mradmin -refreshNodes" after you change the include file or exclude file to commission or decommission a node respectively.
  To permanently add or remove a node, you also need to change slave file conf/slaves.
  
• Block scanner report
  http://datanode_address:50075/blockScannerReport
  
• If you want to check blocks and block locations of a specific file, use following command:
    hadoop fsck file_to_check -files -blocks -locations -racks
  Note: you should execute it on master node.
  Use "hadoop fsck /" to check health of the whole file system.

Extract some continuous lines from a file

Sometimes, I want to extract some continuous lines from a file, e.g. line 10 to line 100. I was thinking whether there is any linux command to do that. Unfortunately, I did not find the command in vanilla Linux distros. Suddenly, I found that it can achieved by combining commands head and tail.

Let's say you want to extract line min to line max, both inclusive.
Calculate nlines=(max-min+1). Then use following command:

cat <filename>|head -n <max>|tail -n <nlines>

vim quickfix and location list

Commands	Description
copen	open quickfix window
cclose	close quickfix window
cwindow	open quickfix window if its content is not empty.
cc [nr]	display error [nr]
cr	display the first error.
cfirst	display the first error.
clast	display the last error.
[count]cn	display [count] next error
[count]cp	display [count] previous error
[count]cnf	display first error in the [count] next file
[count]cpf	display first error in the [count] previous file

For commands related to location list, just replace first 'c' with 'l' in above commands.