How to build Hadoop in Eclipse

Subclipse

1. Install SVN
   I installed "Slik-SVN" because it provides JavaHL lib for 64bit Windows.
2. Install Subclipse plugin to Eclipse.
3. Change eclipse.ini to add following parameters after -vmargs:
   -Djava.library.path=/usr/share/jni/lib (For linux)
   -Djava.library.path=<svn_install_dir>/bin (For Windows)
4. Start eclipse
5. Goto WIndow --> Preference --> Team --> SVN
   In section "SVN Interface", it should say something like "JavaHL (JNI) … SlikSvn". If it says "JavaHL(JNI) not available", it means subclipse cannot find JavaHL library. Check step 3).

Code checkout

Example: svn co http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.21.0/
You can also check out code within Eclipse using Subclipse plugin.

Install Ant and Ivy

Read http://ant.apache.org/ for how to install Ant.

Download ivy jar and put it into directory ANT_HOME/lib/. If ANT_HOME is not specified explicitly, it is the installation directory.

IvyDE

Hadoop is managed by ivy. You need IvyIDE Eclipse plugin. Read http://ant.apache.org/ivy/ivyde/ for more info. IvyDE includes ivy jar file itself. So it does not use the ivy jar you installed in last step. Also it seems that ANT_HOME variable is set to <eclipse_dir>\plugins\org.apache.ant_1.7.1.v20090120-1145 (version number may vary for you).

Shell and Unix commands on Windows

Hadoop build file invokes sh and some other linux commands such as tr, sed to build the project. Of course those commands don't exist on Windows.

Following two projects port linux tools to Windows:

1. http://sourceforge.net/projects/win-bash/files/win-bash/
2. http://unxutils.sourceforge.net/

I use the first one. You just download the tarball and decompress it to a directory. This directory must be passed to Ant. The usual way is you put it into environment variable "PATH". Ant will pick it up automatically. It's true for command line use of Ant. It does not work well for Ant within Eclipse. Following sections include instructions about how to pass PATH to Ant in Eclipse.
For command line use, try command "ant compile".

Create Eclipse Project for Hadoop

1. New --> Java Project
   Select "Create project from existing source". Then select the directory where code is located.
   Click "Next"
2. CHANGE OUTPUT DIRECTORY TO <workspace_name>/build.
   The default directory bin is used by Hadoop for different purposes.
   Click "Finish"
3. Add JDK's tools.jar to build path.  It is not included in JRE.
4. Change source directories to tell Eclipse which directories include Java source code.
   Right click project name --> Build Path --> Configure Build Path… --> Source
5. Make IvyDE to manage ivy dependencies.
   Right click project name --> Build Path --> Configure Build Path… --> Libraries --> Add Library --> IvyDE Managed Dependencies --> Next --> (couple of  IvyDE setting steps) --> OK.
   It may take some time for IvyDE to resolve dependencies.

Create Run Configuration

1. Right click "build.xml" --> Run As --> Ant Build … (not "Ant Build")
2. A dialog should pop up
1. Switch to "Targets" tab: select corresponding target (e.g. compile) you want to execute.
2. Switch to "JRE" tab: select "separate JRE"
3. This step is for Windows users.
   Switch to Environment Tab: set PATH. (to include where those linux tools are included on Window)
   You can click "Select" and choose variable "Path". But in my case, its value does NOT include all of the content of the variable (use "path" command in command line). probably, Eclipse has some restriction about length of value of environment variable. If it's too long, it will be truncated.
4. Click "Run"
3. See Console for messages.

Customize project builder

It's more convenient to use "Builder" than right click "build.xml" --> Run As --> Ant Build … --> Run. Following steps tell you how to use ant as default builder. Then you can use "Project-->Build Project" to build your project (same as any regular native Eclipse Java project).

1. Right click project name --> Properties --> Builder --> New --> Ant Builder
1. Select the build file (usually "build.xml").
2. Switch to "Targets" tab.  Specify which targets are executed when the project is built or cleaned.
3. This step is for Windows users.
   Switch to "Environment" tab. Add PATH environment variable if needed. (to include where those linux tools are included on Window)
2. Deselect default java builder.

package installation log on Ubuntu (dpkg, apt-get, aptitude)

Dpkg log

All deb package operations must go through deb system. So no matter you use apt-get install or deb -I, it will be logged in /var/log/dpkg.log.

lesspipe can show content of .gz files directly. But it cannot show normal text file .

Show install and upgrade history for dkg.log.#.gz files:

ls /var/log/dpkg.log*|sort -r|xargs -I{} lesspipe {}|egrep "^[0-9\-]+[[:space:]][0-9:]+[[:space:]](install|upgrade)[[:space:]]"

Show install and upgrade history for dkg.log and dpkg.log.# files:

ls /var/log/dpkg.log*|sort -r|grep -v ".gz"|xargs -I{} cat {}|egrep "^[0-9\-]+[[:space:]][0-9:]+[[:space:]](install|upgrade)[[:space:]]"

apt-get log

apt-get logges to /var/log/apt/term.log

aptitude

/var/log/aptitude

Resources

http://superuser.com/questions/6338/how-do-you-track-which-packages-were-installed-on-ubuntu-linux

As far as logs, apt-get notoriously doesn't have one;
dpkg does (at /var/log/dpkg.log) but it's famously hard to parse and can only be read with root privileges;
aptitude has one at /var/log/aptitude and you can page through it with regular user privileges.

paratrac on Ubuntu

Compile ftrack

Dependencies

Depends on: fuse-dev, glib-dev, gthread-dev

    sudo apt-get install libfuse-dev  libglib2.0-dev

Use following commands to check whether they are installed successfully

    pkg-config --libs --cflags glib-2.0
    pkg-config --libs --cflags fuse

Build

cd fuse/ftrac/
./configure prefix=your_prefix
make
make install

Add ftrack to PATH:
  export PATH=your_prefix/bin:$PATH
  which ftrac

Use fusetrac.py

add parent directory of paratrac to PYTHONPATH

  python fusetrac.py -t /tmp/fuse/

FUSE FS is mounted and a monitoring page is shown. From now on, when you access /tmp/fuse, data on monitoring page will be changed.

  cd /tmp/fuse

Python in Ubuntu/Debian

http://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html

site module

http://docs.python.org/library/site.html

I assume sys.prefix and sys.exec_prefix are /usr. It may be different on your machine.

/usr/lib/pythonX.Y/site-packages

/usr/lib/site-python

"It sees if it refers to an existing directory, and if so, adds it to sys.path and also inspects the newly added path for configuration files."

Note: sub-directories are not added.

If .pth files exist in those directories, its contents are additional items (one per line) to be added to sys.path.

local admin

A special directory is dedicated to public Python modules installed by the local administrator, /usr/local/lib/pythonX.Y/dist-packages for python2.6 and later, and /usr/local/lib/pythonX.Y/site-packages for python2.5 and earlier. For a local installation by the administrator of python2.6 and later, a special directory is reserved to Python modules which should only be available to this Python, /usr/local/lib/pythonX.Y/site-packages. Unfortunately, for python2.5 and earlier this directory is also visible to the system Python. Additional information on appending site-specific paths to the module search path is available in the official documentation of the site module.

Central repository

It seems all (not all?) modules are installed into directory /usr/share/pyshared. It's a central module repository for python. Python modules in other system directories are symbolic references to files in this directory.

/usr/lib/pyshared/python2.6/: .so python extensions?

python-central

python-central is a tool for Python module management.

pycentral:  register and build utility for Python packages. It manages python modules you installed.

pyversions  prints  information  about installed, supported python runtimes,

py_compilefiles: compiles Python .py source files into .pyc or .pyo bytecode format.

It adds hooks for runtime change:

/usr/share/python/runtime.d/pycentral.rtinstall
/usr/share/python/runtime.d/pycentral.rtremove
/usr/share/python/runtime.d/pycentral.rtupdate

python-support

python-central is another tool for Python module management.

modules managed by python-support are installed in another directory which is added to the sys.path using the .pth mechanism. The .pth mechanism is documented in the Python documentation of the site module. 

During installation, it adds a file /usr/lib/python2.6/dist-packages/python-support.pth which contains /usr/lib/pymodules/pythonX.Y/. This directory also contains the byte-compiled modules for version pythonX.Y.

update-python-modules can be used to rebuild those modules.

It also adds hooks for runtime change:

    /usr/share/python/runtime.d/python-support.rtinstall
    /usr/share/python/runtime.d/python-support.rtremove
    /usr/share/python/runtime.d/python-support.rtupdate

euca2tools

Resources:

http://wiki.debian.org/euca2ools

http://open.eucalyptus.com/wiki/Euca2oolsUsing

 

If you choose to use REST APIs, following options are necessary:

-U: endpoint to which requests are sent

-a: access key ID

-s: secret key

Note: for some tools, access key ID is specified via option "-A" instead of "-a", "-S" instead of "-s".

Install a package from Debian repository for Ubuntu

You can directly use Debian repository. But Debian packages may or may not be compatible with Ubuntu. So you take your own risks by doing so. Another way is to download source and build the package, which is described below.

1) Add a line to /etc/apt/sources.list

    deb-src repo-url

2) Update package index

    sudo apt-get update

3) Install dependencies: (These dependencies are downloaded from Ubuntu repository, not Debian repository)

    sudo apt-get build-dep pkg_name

4) Download source and build package

    apt-get -b source pkg_name

Now you should have a pkg_name.deb file generated in current directory.

5) Install the package

    sudo dpkg -I pkg_name.deb

6) Revert /etc/apt/sources.list file by removing the line added in step 1).

7) Rebuild package index

    sudo apt-get update

 

You are done!

"Twitter is over capacity"

Arrays in Bash (Cheatsheet)

Description	Indexed array	Associative array
Declare an array	declare -a array	declare -A array
Assignment	array=(value1 value2 … valuen)	array=([sub1]=value1 … [subN]=valueN) Note:You must declare it first you can use declare -A array=([sub1]=value1 … [subN]=valueN)
Expand to all values	${array[@]}  or ${array[*]}	${array[@]}  or ${array[*]}
Expand to all keys	${!array[@]}  or ${!array[*]}	${!array[@]}  or ${!array[*]}
length of an array	${#array[@]} or ${#array[*]}	${#array[@]} or ${#array[*]}
Remove entire array	unset array or unset array[@] or unset array[*]	unset array or unset array[@] or unset array[*]
Remove an element	unset array[index]	unset array[index]
Array join	array1+=(${array2[@]}) or array1+=(v1 v2 … vN)	array1+=([sub1]=v1 [sub2]=v2 … [subN]=vN)
Length of an element:	${#array[index]}	${#array[index]}

 

Resources

http://tldp.org/LDP/abs/html/arrays.html

Bash config file loading order

interactive	login	files sourced	Related option
Y	Y	/etc/profile, {~/.bash_profile| ~/.bash_login| ~/.profile } Only one of the files in brackets will be executed.	--noprofile
N	Y
Y	N	/etc/bash.bashrc, ~/.bashrc	--norc

Kindle 3rd generation

 

I got problems when I tried to read some books in Chinese.

I will basically investigate following questions

1. What encodings are supported by Kindle
2. What fonts are supported
3. If the fonts used in the PDF cannot be found, how Kindle handles it.
4. What formats are supported?