Sending it to the hadoop mailing list - I think this is a hadoop related problem and not related to Cloudera distribution.
Raj ----- Forwarded Message ----- >From: Raj V <rajv...@yahoo.com> >To: CDH Users <cdh-u...@cloudera.org> >Sent: Friday, September 30, 2011 5:21 PM >Subject: pointing mapred.local.dir to a ramdisk > > >Hi all > > >I have been trying some experiments to improve performance. One of the >experiments involved pointing mapred.local.dir to a RAM disk. To this end I >created a 128MB RAM disk ( each of my map outputs are smaller than this) but I >have not been able to get the task tracker to start. > > >I am running CDH3B3 ( hadoop-0.20.2+737) and here the error message from the >task tracker log. > > >Tasktracker logs > > >2011-09-30 16:50:00,689 INFO org.mortbay.log: Logging to >org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog >2011-09-30 16:50:00,930 INFO org.apache.hadoop.http.HttpServer: Added global >filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) >2011-09-30 16:50:01,000 INFO org.apache.hadoop.http.HttpServer: Port returned >by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening >the listener on 50060 >2011-09-30 16:50:01,023 INFO org.apache.hadoop.http.HttpServer: >listener.getLocalPort() returned 50060 >webServer.getConnectors()[0].getLocalPort() returned 50060 >2011-09-30 16:50:01,024 INFO org.apache.hadoop.http.HttpServer: Jetty bound to >port 50060 >2011-09-30 16:50:01,024 INFO org.mortbay.log: jetty-6.1.14 >2011-09-30 16:50:02,388 INFO org.mortbay.log: Started >SelectChannelConnector@0.0.0.0:50060 >2011-09-30 16:50:02,400 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 >2011-09-30 16:50:02,422 INFO org.apache.hadoop.mapred.TaskTracker: Starting >tasktracker with owner as mapred >2011-09-30 16:50:02,493 ERROR org.apache.hadoop.mapred.TaskTracker: Can not >start task tracker because java.lang.NullPointerException > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:213) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157) > at >org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:253) > at >org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:404) > at >org.apache.hadoop.util.MRAsyncDiskService.moveAndDeleteRelativePath(MRAsyncDiskService.java:255) > at >org.apache.hadoop.util.MRAsyncDiskService.cleanupAllVolumes(MRAsyncDiskService.java:311) > at >org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:618) > at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1351) > at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3504) > > >2011-09-30 16:50:02,497 INFO org.apache.hadoop.mapred.TaskTracker: >SHUTDOWN_MSG: >/************************************************************ >SHUTDOWN_MSG: Shutting down TaskTracker at HADOOP52-4/10.52.1.5 > > >and here is my mapred-site.xml file > > ><property> > <name>mapred.local.dir</name> > <value>/ramdisk1</value> > </property> > > >If I have a regular directory on a regular drive such as below - it works. If >I don't mount the ramdisk - it works. > > ><property> > <name>mapred.local.dir</name> > <value>/hadoop-dsk0/local,/hadoop-dsk1/local</value> > </property> > > > > > >The NullPointerException does not tell me what the error is or how to fix it. > > >From the logs it looks like some disk based operation failed. I can't guess I >must also confess that this is the first time I am using an ext2 file system. > > >Any ideas? > > > > >Raj > > > > > > > >
