[
https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782813#action_12782813
]
Vinod K V commented on MAPREDUCE-1245:
--------------------------------------
Here's the problem as mentioned
[here|https://issues.apache.org/jira/browse/MAPREDUCE-1239?focusedCommentId=12782742&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12782742]:
{code}
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker up at: 32801
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker webserver:
44348
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: Cleaning up the system
directory
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: problem cleaning system
directory: file:/tmp/hadoop-sreerama/mapred/system
[junit] java.io.IOException: Cannot run program "chmod":
java.io.IOException: error=24, Too many open files
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:474)
[junit] at org.apache.hadoop.util.Shell.runCommand(Shell.java:188)
[junit] at org.apache.hadoop.util.Shell.run(Shell.java:170)
[junit] at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:363)
[junit] at org.apache.hadoop.util.Shell.execCommand(Shell.java:449)
[junit] at org.apache.hadoop.util.Shell.execCommand(Shell.java:432)
[junit] at
org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
[junit] at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
[junit] at
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:281)
[junit] at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:295)
[junit] at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1477)
[junit] at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1306)
[junit] at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1299)
[junit] at
org.apache.hadoop.mapred.UtilsForTests.getJobTracker(UtilsForTests.java:712)
[junit] at
org.apache.hadoop.mapred.TestFairScheduler.testPoolAssignment(TestFairScheduler.java:2546)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at java.lang.reflect.Method.invoke(Method.java:616)
[junit] at junit.framework.TestCase.runTest(TestCase.java:168)
[junit] at junit.framework.TestCase.runBare(TestCase.java:134)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
[junit] at junit.framework.TestResult.run(TestResult.java:113)
[junit] at junit.framework.TestCase.run(TestCase.java:124)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:232)
[junit] at junit.framework.TestSuite.run(TestSuite.java:227)
[junit] at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
[junit] at
junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
[junit] Caused by: java.io.IOException: java.io.IOException: error=24, Too
many open files
[junit] at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
[junit] at java.lang.ProcessImpl.start(ProcessImpl.java:81)
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:467)
[junit] ... 31 more
{code}
This happens because every {{submitJob()}} creates a new {{JobTracker}} object.
The same problem plagues other tests too and drastically slows them down.
Creating a single object and use it repetitively should suffice.
> TestFairScheduler fails with "too many open files" error
> --------------------------------------------------------
>
> Key: MAPREDUCE-1245
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Vinod K V
> Fix For: 0.21.0
>
>
> This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
