Change ports used by Hadoop

Edit file conf/hdfs-site.xml to change ports used by HDFS

    <property>
        <name>dfs.secondary.http.address</name>
        <value>0.0.0.0:51090</value>
    </property>
    <property>
        <name>dfs.datanode.address</name>
        <value>0.0.0.0:51010</value>
    </property>
    <property>
        <name>dfs.datanode.http.address</name>
        <value>0.0.0.0:51075</value>
    </property>
    <property>
        <name>dfs.datanode.https.address</name>
        <value>0.0.0.0:51475</value>
    </property>
    <property>
        <name>dfs.datanode.ipc.address</name>
        <value>0.0.0.0:51020</value>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>0.0.0.0:51070</value>
    </property>
    <property>
        <name>dfs.https.address</name>
        <value>0.0.0.0:51470</value>
    </property>

Edit file conf/mapred-site.xml o change ports used by MapReduce

    <property>
        <name>mapred.job.tracker.http.address</name>
        <value>0.0.0.0:51030</value>
    </property>

    <property>
        <name>mapred.task.tracker.http.address</name>
        <value>0.0.0.0:51060</value>
    </property>

exclude directories when using GNU tar

tar zvcf name.tar.gz --exclude path/to/dir1 --exclude path/to/dir2 path/to/tar

Note:

1. Do not include a trailing '/' in the path of excluded directories.  Otherwise, it won't work.
2. Put --exclude before the directory/file to be tarred.

Install RPM packages with non-root account

Recently, I need to install RPM to RedHat Linux, but I don't have root access.  I found this post: http://ajaya.name/?p=6353.  However, some commands in the command are not correct or need more clarification.  So I wrote down my experience below.

• create file ~/.rpmmacros and add following line
      %_rpmlock_path lib/rpm/__db.000
• Initialize the database by running command:

rpm --initdb \

    --root /home/<user_name>/rpm-local/ \

    --dbpath /home/<user_name>/rpm-local/lib/rpm

• Check the dependence

rpm --root /home/<user_name>/rpm-local/ \
    --dbpath /home/<user_name>/rpm-local/lib/rpm \
    -ivh package.rpm

• Install package

rpm --root /home/<user_name>/rpm-local \
    --dbpath /home/<user_name>/rpm-local/lib/rpm \ 
    --relocate /usr=/home/<user_name>/rpm-local \
    --nodeps \
    -ivh package.rpm

Resources

• https://bugzilla.redhat.com/show_bug.cgi?id=222194
• http://www.redhat.com/archives/rpm-list/2007-May/msg00037.html
• https://rhn.redhat.com/errata/RHBA-2006-0440.html

Add "Edit with Vim" item to context-menu in Windows for Vim

This post shows how to add "Edit with Vim" and "Tab Edit with Vim" items to context menu (pop out when you right click a file) in Windows.

Run regedit.exe, go to HKEY_LOCAL_MACHINE/SOFTWARE/Classes/*/shell/

1. Create new key "Tab Edit with &Vim" (Right click parent entry -> New -> Key)
2. Create new key "Edit with Vim"
3. Create new key "command" under "Tab Edit with &Vim"
4. Edit the entry with name "(Default)", change its data to
   <vim_dir>\gvim.exe" -p --remote-tab-silent "%1" "%*"
5. Create new key "command" under "Edit with Vim".
6. Edit the entry with name "(Default)", change its data to
   <vim_dir>\gvim.exe "%1"

It seems that latest versions of vim automatically create the registry entry:

• HKEY_LOCAL_MACHINE\SOFTWARE\Vim\Gvim and the data of entry path points to the vim executable.
• HKEY_CLASSES_ROOT\*\shellex\ContextmenuHandlers\gvim. Data of the entry "(Default)" is the CLSID (51EEE242-AD87-11d3-9C1E-0090278BBD99).
  You can delete this entry because we have already added menu item "Edit with Vim".
• HKEY_CLASSES_ROOT\Applications\gvim.exe\shell\edit\command. Data of the entry "(Default)" is "<vim_dir>\gvim.exe" "%1"
• HKEY_CLASSES_ROOT\CLSID\{51EEE242-AD87-11d3-9C1E-0090278BBD99}
  It has a key named InProcServer32. The data of "(Default)" is <vim_dir>\gvimext.dll
• Some other keys

Logging in Hadoop

Hadoop uses log4j via Apache common logging.  The config file is conf/log4j.properties.

Some important variables are set in the command line.  Following is a snippet cut from the whole command line used to launch HDFS name node.

-Dhadoop.root.logger=INFO,DRFA
-Dhadoop.log.dir=/N/u/hdfs/programs/hadoop-0.21.0/bin/../logs
-Dhadoop.log.file=hadoop-hdfs-namenode-b009.log
-Dhadoop.home.dir=/N/u/hdfs/programs/hadoop-0.21.0/bin/..
-Dhadoop.id.str=hdfs

You can see that log dir, log file, log level, logger are set.  DRFA is defined in conf/log4j.properties.

pdf to eps conversion

Currently, I need to convert pdf files to eps so that they can be included in latex files.

1. Use Acrobat Pro open the pdf file. 
   Click File -> export -> PostScript -> Encapsulated PostScript or use "Save As" and change "Save As Type"
   However, the bounding box is NOT correctly calculated.
   You can use gsview to correct it.  Use gsview open the eps file, click "File -> PS to EPS", select "Automatically calculate Bounding Box" and save the output file.

2. Use ghostscript.  Execute following command:
   gswin32 -sDEVICE=epswrite -sOutputFile=<filename>.eps <filename>.pdf
   This works well and bounding box is correctly calculated.

3. Use Xpdf (http://www.foolabs.com/xpdf/download.html)
   pdftops -eps <filename>.pdf <filename>.eps
   However, the bounding box is NOT correctly calculated.

Brief Latex notes for equations

In-line: $…$

Single line, without equation number: \[ … \] or \begin{equation*} … \end{equation*}

Single line, with equation number: \begin{equation} … \end{equation}

Multi-line, without equation number: \begin{align*} ... \end{align*}

Multi-line, with equation number: \begin{align} ... \end{align}

"a double backslash (\\) is used to separate the lines, and an ampersand symbol (&) is used to indicate the place at which the formulas should be aligned."

For align, \label must be put in front of each equation. For equation, it does not matter you put it in front or in the end.

Make vim-latex to generate output in a specified directory

Recently I started to use latex to write papers.  I want to edit latex in my favorite editor - vim. I found the project vim-latex: http://vim-latex.sourceforge.net/.  It is powerful and convenient to use. 

However, one feature I want is to generate output files (.div, .ps, .log, etc) into a separate directory rather than the same directory as tex files.  It turns out that vim-latex does not support it natively.  So I hacked into the source code to make it work on Windows.

1. Edit file ~/vimfiles/ftplugin/tex.vim

   Add following config:

     set iskeyword+=: 
     let g:Tex_Outdir='out' 
     let g:Tex_ViewRule_pdf='"Foxit Reader.exe" ' 
     let g:Tex_CompileRule_dvi='mkdir '.g:Tex_Outdir.' & latex -output-directory='.g:Tex_Outdir.' -src-specials --interaction=nonstopmode $*' 
     let g:Tex_CompileRule_pdf=g:Tex_CompileRule_dvi.' & cd '.g:Tex_Outdir.' & dvipdfm $*.dvi'

   Basically above config specifies

   1. output directory of latex compilation
   2. How to view PDF files (if you don't give full path, the command needs to be in env variable PATH)
   3. How to compile dvi: create the output directory and put output there
   4. How to compile pdf: first compile tex to dvi, and then call dvipdfm to generate pdf

2. Change ~/vimfiles/ftplugin/latex-suite/compiler.vim

   Change line 252 to (this line adds the full path of output directory):
     let execString = 'start '.s:viewer.' "'.expand('%:p:h').'/'.g:Tex_Outdir.'/$*.'.s:target.'"'

   Change line 405 to (this line adds the full path of output directory):
   let execString = 'silent! !'.viewer.' "'.expand('%:p:h').'/'.g:Tex_Outdir.'/'.mainfnameRoot.'.'.s:target.'" '.line('.').' "'.expand('%').'"'

   Note: the text in blue is what I added (diff against the original code)

Use command 'TTarget' to switch amont dvi, ps, pdf, etc.

How to publish artifacts to ivy local repository and use it in another project

Build artifacts that are required by another project

ivy.xml: Attributes organisation, module and revision of element ivy-module/info.

1) Add a new resolver to ivy by adding following snippet to ivysettings.xml:

<filesystem name="gerald-local-ivy" m2compatible="false" force="false" local="true">
    <artifact pattern="${ivy.default.ivy.user.dir}/local/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
</filesystem>

This will create a directory local under <user.home>/.ivy2/.

2) Add following snippet to your build.xml:

<target name="publish" depends="jar" description="Publish">
    <!-- following property defines the version to publish -->
    <property name="ivy.deliver.revision" value="${version}"/>
    <ivy:publish resolver="gerald-local-ivy" forcedeliver="true"
    	 settingsRef="${ant.project.name}.ivy.settings" overwrite="true">
    	<artifacts pattern="${build.dir}/[artifact]-[revision](-[classifier]).[ext]" />
    </ivy:publish>
</target>

The resolver attribute of ivy:publish must match name attribute specified in step 1). Change pattern attribute for artifacts element to match where you put the artifact.

3) Use command ant publish to publish your jar.

Build main project that depends on the artifacts built above.

1) Add the same resolver to ivy by adding following snippet to ivysettings.xml:

<filesystem name="gerald-local-ivy" m2compatible="false" force="false" local="true">
    <artifact pattern="${ivy.default.ivy.user.dir}/local/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
</filesystem>

2) Add it to your effective resolver chain.

3) Add dependency declaration to ivy.xml

<dependency org="<organization>" name="<module>" 

            rev="<version>" conf="<common->master>">

The attributes org, name, rev must match the values specified when you built the dependency jar.

4) Manually remove the artifacts that exist in local cache (<user.home>/.ivy2/cache).

5) Build your project

Resources

http://ant.apache.org/ivy/history/2.2.0/use/publish.html
http://ant.apache.org/ivy/history/latest-milestone/resolver/chain.html
http://stackoverflow.com/questions/353336/how-does-ivypublish-work
http://mail-archives.apache.org/mod_mbox/ant-ivy-user/201002.mbox/%3C27714488.post@talk.nabble.com%3E

Configure ivy to use local Maven repository

This post shows how to configure ivy to use specific local Maven repository.

1) Add following config as a child element of tag "resolvers" in you ivy settings file:

    <filesystem name="local-maven-2" m2compatible="true" force="false" local="true">
       <artifact pattern="${gerald.repo.dir}/[organisation]/[module]/[revision]/[module]-[revision].[ext]"/>
       <ivy pattern="${gerald.repo.dir}/[organisation]/[module]/[revision]/[module]-[revision].pom"/>
    </filesystem>

2) Then add it to your chain resolver config. Example: 

    <chain name="internal" dual="true">
      <resolver ref="local-maven-2"/>
      <resolver ref="apache-snapshot"/> 
      <resolver ref="maven2"/>
    </chain>

If you have multiple resolver chain, make sure that the correct one, which is effective for you build, is changed.

3) After ivy caches the artifacts in its own local repo (the first time the dependency is resolved), it will not pick changes you made to the artifacts in the original Maven repository.  In other words, if you use "mvn clean install" to re-publish the artifact, the new version will NOT propagate to ivy.
You can change the default behavior by tweaking parameters:
    checkmodified, changingPattern,  changingMatcher, alwaysCheckExactRevision
Read this article for details: http://ant.apache.org/ivy/history/trunk/settings/resolvers.html
One example:

    <chain name="default" dual="true" 
	     checkmodified="true" changingPattern=".*SNAPSHOT">
     ......
    </chain>

Resources:

http://mail-archives.apache.org/mod_mbox/ant-ivy-user/200807.mbox/raw/%3C94bda3fa0807202250y446a818eodb527dba96c8ac93@mail.gmail.com%3E/