MiniMRCluster does not utilize multiple local directories in "mapred.local.dir"
-------------------------------------------------------------------------------
Key: HADOOP-2462
URL: https://issues.apache.org/jira/browse/HADOOP-2462
Project: Hadoop
Issue Type: Bug
Components: test
Affects Versions: 0.15.0
Reporter: Konstantin Shvachko
Fix For: 0.16.0
My hadoop-site.xml specifies 4 local directories
{code}
<property>
<name>mapred.local.dir</name>
<value>${hadoop.tmp.dir}/mapred/local1, ${hadoop.tmp.dir}/mapred/local2,
${hadoop.tmp.dir}/mapred/local3,
${hadoop.tmp.dir}/mapred/local4</value>
</property>
{code}
and I am looking at MiniMRCluster.TaskTrackerRunner
There are several things here:
# localDirBase value is set to
{code}
"/tmp/h/mapred/local1, /tmp/h/mapred/local2, /tmp/h/mapred/local3,
/tmp/h/mapred/local4"
{code}
and I get a hierarchy of directories with commas and spaces in the names.
I think this was not designed to work with multiple dirs.
# Further down, all new directories are generated with the same name
{code}
File ttDir = new File(localDirBase,
Integer.toString(trackerId) + "_" + 0);
{code}
So in fact only one directory is created. I think the intension was to have i
instead of 0
{code}
File ttDir = new File(localDirBase,
Integer.toString(trackerId) + "_" + i);
{code}
# On windows MiniMRCluster.TaskTrackerRunner in this case throws an
IOException,
which is silently ignored by all but the TestMiniMRMapRedDebugScript MiniMR
tests.
{code}
java.io.IOException: Mkdirs failed to create
/tmp/h/mapred/local1, /tmp/h/mapred/local2, /tmp/h/mapred/local3,
/tmp/h/mapred/local4/0_0
at
org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.<init>(MiniMRCluster.java:124)
at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:293)
at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:244)
at
org.apache.hadoop.mapred.TestMiniMRClasspath.testClassPath(TestMiniMRClasspath.java:163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:478)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:344)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
{code}
I am marking it as "Major" because we actually do not test multiple local
directories.
Looks like it was introduced rather recently by HADOOP-1819.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
