Monday, October 17, 2011

Logging in Hadoop

Hadoop uses log4j via Apache common logging.  The config file is conf/log4j.properties.

Some important variables are set in the command line.  Following is a snippet cut from the whole command line used to launch HDFS name node.

-Dhadoop.root.logger=INFO,DRFA
-Dhadoop.log.dir=/N/u/hdfs/programs/hadoop-0.21.0/bin/../logs
-Dhadoop.log.file=hadoop-hdfs-namenode-b009.log
-Dhadoop.home.dir=/N/u/hdfs/programs/hadoop-0.21.0/bin/..
-Dhadoop.id.str=hdfs

You can see that log dir, log file, log level, logger are set.  DRFA is defined in conf/log4j.properties.