[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782795#action_12782795 ] Giridharan Kesavan commented on MAPREDUCE-1239: --- tested m-1239.patch and hit the same exception as sreekant mentioned. Otherwise this patch looks good. +1 Mapreduce test build is broken after HADOOP-5107 Key: MAPREDUCE-1239 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0, 0.22.0 Reporter: Vinod K V Assignee: Giridharan Kesavan Priority: Blocker Fix For: 0.21.0 Attachments: m-1239.patch, mapred-1239.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782796#action_12782796 ] Giridharan Kesavan commented on MAPREDUCE-1239: --- let me commit this to fix the trunk build failure and file a jira for the fairscheduler test failure TestFairScheduler Mapreduce test build is broken after HADOOP-5107 Key: MAPREDUCE-1239 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0, 0.22.0 Reporter: Vinod K V Assignee: Giridharan Kesavan Priority: Blocker Fix For: 0.21.0 Attachments: m-1239.patch, mapred-1239.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1009) Forrest documentation needs to be updated to describes features provided for supporting hierarchical queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782800#action_12782800 ] rahul k singh commented on MAPREDUCE-1009: -- +1 with the patch Forrest documentation needs to be updated to describes features provided for supporting hierarchical queues --- Key: MAPREDUCE-1009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1009 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 0.21.0 Reporter: Hemanth Yamijala Assignee: Vinod K V Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-1009-20091008.txt, MAPREDUCE-1009-20091116.txt, MAPREDUCE-1009-20091124.txt Forrest documentation must be updated for describing how to set up and use hierarchical queues in the framework and the capacity scheduler. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1110) Currently QueueManager API's return JobQueueInfo , instead it should return QueueInfo, similarly wher ever we are using JobQueueInfo as view , we should use QueueInfo
[ https://issues.apache.org/jira/browse/MAPREDUCE-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1110: - Attachment: MAPREDUCE-1110-20091126.1.txt Uploading a new patch that has to be applied over the latest patches at MAPREDUCE-1082 and MAPREDUCE-1075 in that order. This patch also fixes the incompatibility issues with JobQueueInfo. Currently QueueManager API's return JobQueueInfo , instead it should return QueueInfo, similarly wher ever we are using JobQueueInfo as view , we should use QueueInfo --- Key: MAPREDUCE-1110 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1110 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.21.0 Reporter: rahul k singh Assignee: Vinod K V Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-1110-20091119.1.txt, MAPREDUCE-1110-20091119.txt, MAPREDUCE-1110-20091126.1.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
eclipse-plugin fails with missing dependencies -- Key: MAPREDUCE-1244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1244: -- Attachment: mapred-1244.patch eclipse-plugin fails with missing dependencies -- Key: MAPREDUCE-1244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: mapred-1244.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1009) Forrest documentation needs to be updated to describes features provided for supporting hierarchical queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1009: - Status: Patch Available (was: Open) Forrest documentation needs to be updated to describes features provided for supporting hierarchical queues --- Key: MAPREDUCE-1009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1009 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 0.21.0 Reporter: Hemanth Yamijala Assignee: Vinod K V Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-1009-20091008.txt, MAPREDUCE-1009-20091116.txt, MAPREDUCE-1009-20091124.txt Forrest documentation must be updated for describing how to set up and use hierarchical queues in the framework and the capacity scheduler. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1244: -- Affects Version/s: 0.22.0 eclipse-plugin fails with missing dependencies -- Key: MAPREDUCE-1244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: mapred-1244.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1244: -- Component/s: build updated components and affected versions eclipse-plugin fails with missing dependencies -- Key: MAPREDUCE-1244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: mapred-1244.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1245) TestFairScheduler fails with too many open files error
TestFairScheduler fails with too many open files error Key: MAPREDUCE-1245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Vinod K V Fix For: 0.21.0 This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1245) TestFairScheduler fails with too many open files error
[
https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782813#action_12782813
]
Vinod K V commented on MAPREDUCE-1245:
--
Here's the problem as mentioned
[here|https://issues.apache.org/jira/browse/MAPREDUCE-1239?focusedCommentId=12782742page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12782742]:
{code}
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker up at: 32801
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker webserver:
44348
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: Cleaning up the system
directory
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: problem cleaning system
directory: file:/tmp/hadoop-sreerama/mapred/system
[junit] java.io.IOException: Cannot run program chmod:
java.io.IOException: error=24, Too many open files
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:474)
[junit] at org.apache.hadoop.util.Shell.runCommand(Shell.java:188)
[junit] at org.apache.hadoop.util.Shell.run(Shell.java:170)
[junit] at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:363)
[junit] at org.apache.hadoop.util.Shell.execCommand(Shell.java:449)
[junit] at org.apache.hadoop.util.Shell.execCommand(Shell.java:432)
[junit] at
org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
[junit] at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
[junit] at
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:281)
[junit] at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:295)
[junit] at
org.apache.hadoop.mapred.JobTracker.init(JobTracker.java:1477)
[junit] at
org.apache.hadoop.mapred.JobTracker.init(JobTracker.java:1306)
[junit] at
org.apache.hadoop.mapred.JobTracker.init(JobTracker.java:1299)
[junit] at
org.apache.hadoop.mapred.UtilsForTests.getJobTracker(UtilsForTests.java:712)
[junit] at
org.apache.hadoop.mapred.TestFairScheduler.testPoolAssignment(TestFairScheduler.java:2546)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at java.lang.reflect.Method.invoke(Method.java:616)
[junit] at junit.framework.TestCase.runTest(TestCase.java:168)
[junit] at junit.framework.TestCase.runBare(TestCase.java:134)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
[junit] at junit.framework.TestResult.run(TestResult.java:113)
[junit] at junit.framework.TestCase.run(TestCase.java:124)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:232)
[junit] at junit.framework.TestSuite.run(TestSuite.java:227)
[junit] at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
[junit] at
junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
[junit] Caused by: java.io.IOException: java.io.IOException: error=24, Too
many open files
[junit] at java.lang.UNIXProcess.init(UNIXProcess.java:164)
[junit] at java.lang.ProcessImpl.start(ProcessImpl.java:81)
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:467)
[junit] ... 31 more
{code}
This happens because every {{submitJob()}} creates a new {{JobTracker}} object.
The same problem plagues other tests too and drastically slows them down.
Creating a single object and use it repetitively should suffice.
TestFairScheduler fails with too many open files error
Key: MAPREDUCE-1245
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.21.0
Reporter: Vinod K V
Fix For: 0.21.0
This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-3-ydist.txt Patch renaming uniqueSubDir. I think the log will be useful in debugging, so leaving it as is. Verified that chmod is not done on baseDir, by manually verifying logs. Reproduced the issue by putting about 1Lakh files in base dir, and running a job using 5 files on distributed cache. Verified that the same job, under same environment, finishes faster with the patch. Verified that streaming and pipes jobs, which use executables from distributed cache, work fine (with both DefaultTaskController and LinuxTaskController). Thanks Karthikeyan for helping in testing. test-patch and ant test passed on local machine, except TestHdfsProxy. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with too many open files error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Attachment: 1245_v1.patch Use a lightweight JT for tests. TestFairScheduler fails with too many open files error Key: MAPREDUCE-1245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Vinod K V Fix For: 0.21.0 Attachments: 1245_v1.patch This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with too many open files error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Status: Patch Available (was: Open) TestFairScheduler fails with too many open files error Key: MAPREDUCE-1245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Vinod K V Fix For: 0.21.0 Attachments: 1245_v1.patch This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1239: -- Resolution: Fixed Fix Version/s: (was: 0.21.0) 0.22.0 Status: Resolved (was: Patch Available) I just committed this to trunk, thanks Owen Mapreduce test build is broken after HADOOP-5107 Key: MAPREDUCE-1239 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0, 0.22.0 Reporter: Vinod K V Assignee: Giridharan Kesavan Priority: Blocker Fix For: 0.22.0 Attachments: m-1239.patch, mapred-1239.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan resolved MAPREDUCE-1244. --- Resolution: Fixed Fix Version/s: 0.22.0 eclipse-plugin fails with missing dependencies -- Key: MAPREDUCE-1244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Fix For: 0.22.0 Attachments: mapred-1244.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V reopened MAPREDUCE-1239: -- This has to be fixed in 0.21 too as HADOOP-5107 was pulled into 0.21 branch. Mapreduce test build is broken after HADOOP-5107 Key: MAPREDUCE-1239 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0, 0.22.0 Reporter: Vinod K V Assignee: Giridharan Kesavan Priority: Blocker Fix For: 0.22.0 Attachments: m-1239.patch, mapred-1239.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-28) TestQueueManager takes too long and times out some times
[ https://issues.apache.org/jira/browse/MAPREDUCE-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782861#action_12782861 ] Hudson commented on MAPREDUCE-28: - Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) TestQueueManager takes too long and times out some times Key: MAPREDUCE-28 URL: https://issues.apache.org/jira/browse/MAPREDUCE-28 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, test Affects Versions: 0.21.0 Reporter: Amareshwari Sriramadasu Assignee: V.V.Chaitanya Krishna Fix For: 0.21.0 Attachments: MAPREDUCE-28-1.txt, MAPREDUCE-28-2.txt, MAPREDUCE-28-3.txt, MAPREDUCE-28-4.txt, MAPREDUCE-28-5.txt, MAPREDUCE-28-6.txt, MAPREDUCE-28-7.txt, MAPREDUCE-28-8.patch TestQueueManager takes long time for the run and timeouts sometimes. See the failure at http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3875/testReport/. Looking at the console output, before the test finsihes, it was timed-out. On my machine, the test takes about 5 minutes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782865#action_12782865 ] Hudson commented on MAPREDUCE-1244: --- Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) . Fix eclipse-plugin's build dependencies. eclipse-plugin fails with missing dependencies -- Key: MAPREDUCE-1244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Fix For: 0.22.0 Attachments: mapred-1244.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782862#action_12782862 ] Hudson commented on MAPREDUCE-1239: --- Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) Mapreduce test build is broken after HADOOP-5107 Key: MAPREDUCE-1239 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0, 0.22.0 Reporter: Vinod K V Assignee: Giridharan Kesavan Priority: Blocker Fix For: 0.22.0 Attachments: m-1239.patch, mapred-1239.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1182) Reducers fail with OutOfMemoryError while copying Map outputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782864#action_12782864 ] Hudson commented on MAPREDUCE-1182: --- Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) Reducers fail with OutOfMemoryError while copying Map outputs - Key: MAPREDUCE-1182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1182 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chandra Prakash Bhagtani Assignee: Chandra Prakash Bhagtani Priority: Blocker Fix For: 0.20.2 Attachments: HADOOP-6357.patch, M1182-0.patch, M1182-0v20.patch, M1182-1.patch, M1182-1v20.patch Reducers fail while copying Map outputs with following exception java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1539) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216) ,Error: Reducer's memory usage keeps on increasing and ultimately exceeds -Xmx value I even tried with -Xmx6.5g to each reducer but it's still failing While looking into the reducer logs, I found that reducers were doing shuffleInMemory every time, rather than doing shuffleOnDisk -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1245) TestFairScheduler fails with too many open files error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782909#action_12782909 ] Hadoop QA commented on MAPREDUCE-1245: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426216/1245_v1.patch against trunk revision 884531. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/console This message is automatically generated. TestFairScheduler fails with too many open files error Key: MAPREDUCE-1245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Vinod K V Fix For: 0.21.0 Attachments: 1245_v1.patch This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved MAPREDUCE-1239. -- Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] I committed this to 0.21 also. Mapreduce test build is broken after HADOOP-5107 Key: MAPREDUCE-1239 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0, 0.22.0 Reporter: Vinod K V Assignee: Giridharan Kesavan Priority: Blocker Fix For: 0.21.0, 0.22.0 Attachments: m-1239.patch, mapred-1239.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12782965#action_12782965 ] Hudson commented on MAPREDUCE-1239: --- Integrated in Hadoop-Mapreduce-trunk-Commit #135 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/135/]) . Fix contrib components build dependencies. (Giridharan Kesavan and omalley) Move the change message into 0.21.0. Mapreduce test build is broken after HADOOP-5107 Key: MAPREDUCE-1239 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0, 0.22.0 Reporter: Vinod K V Assignee: Giridharan Kesavan Priority: Blocker Fix For: 0.21.0, 0.22.0 Attachments: m-1239.patch, mapred-1239.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-1231: - Status: Open (was: Patch Available) Distcp is very slow --- Key: MAPREDUCE-1231 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.20.1 Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, mapred-1231-y20-v3.patch, mapred-1231-y20.patch, mapred-1231.patch Currently distcp does a checksums check in addition to file length check to decide if a remote file has to be copied. If the number of files is high (thousands), this checksum check is proving to be fairly costly leading to a long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-1231: - Status: Patch Available (was: Open) Trying to pass through Hudson once more Distcp is very slow --- Key: MAPREDUCE-1231 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.20.1 Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, mapred-1231-y20-v3.patch, mapred-1231-y20.patch, mapred-1231.patch Currently distcp does a checksums check in addition to file length check to decide if a remote file has to be copied. If the number of files is high (thousands), this checksum check is proving to be fairly costly leading to a long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-787: -- Status: Open (was: Patch Available) -files, -archives should honor user given symlink path -- Key: MAPREDUCE-787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, patch-787-ydist.txt, patch-787.txt Currently, if user gives an option such as -files hdfs://host:fs_port/user/testfile.txt#testlink The symlink name testlink is not honored. It alwasys creates symlink with name testfile.txt in cwd of the task. If the user has given a symlink name, it should be honored. If no symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-787: -- Status: Patch Available (was: Open) -files, -archives should honor user given symlink path -- Key: MAPREDUCE-787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, patch-787-3.txt, patch-787-ydist.txt, patch-787.txt Currently, if user gives an option such as -files hdfs://host:fs_port/user/testfile.txt#testlink The symlink name testlink is not honored. It alwasys creates symlink with name testfile.txt in cwd of the task. If the user has given a symlink name, it should be honored. If no symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-787: -- Attachment: patch-787-3.txt Re-submitting the same patch for hudson, since the common jars will be picked up from maven. -files, -archives should honor user given symlink path -- Key: MAPREDUCE-787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, patch-787-3.txt, patch-787-ydist.txt, patch-787.txt Currently, if user gives an option such as -files hdfs://host:fs_port/user/testfile.txt#testlink The symlink name testlink is not honored. It alwasys creates symlink with name testfile.txt in cwd of the task. If the user has given a symlink name, it should be honored. If no symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783019#action_12783019 ] Amareshwari Sriramadasu commented on MAPREDUCE-787: --- Re-ran ant test in branch 0.21, with the patch. All the tests passed except TestGridmixSubmission (due to MAPREDUCE-1124) -files, -archives should honor user given symlink path -- Key: MAPREDUCE-787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, patch-787-3.txt, patch-787-ydist.txt, patch-787.txt Currently, if user gives an option such as -files hdfs://host:fs_port/user/testfile.txt#testlink The symlink name testlink is not honored. It alwasys creates symlink with name testfile.txt in cwd of the task. If the user has given a symlink name, it should be honored. If no symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with too many open files error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Status: Open (was: Patch Available) Retrying hudson TestFairScheduler fails with too many open files error Key: MAPREDUCE-1245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Vinod K V Fix For: 0.21.0 Attachments: 1245_v1.patch This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with too many open files error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Status: Patch Available (was: Open) TestFairScheduler fails with too many open files error Key: MAPREDUCE-1245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Vinod K V Fix For: 0.21.0 Attachments: 1245_v1.patch This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: patch-1186-3-ydist.txt I dither, I preferred both 'unique' and 'parent' prefixes for CacheStatus.parentDir, so I renamed it to be CacheStatus.uniqueParentDir... no other changes. *grazing at my navel* While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-28) TestQueueManager takes too long and times out some times
[ https://issues.apache.org/jira/browse/MAPREDUCE-28?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-28: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed this to 0.21 and trunk. Thanks Chaitanya and Rahul. TestQueueManager takes too long and times out some times Key: MAPREDUCE-28 URL: https://issues.apache.org/jira/browse/MAPREDUCE-28 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, test Affects Versions: 0.21.0 Reporter: Amareshwari Sriramadasu Assignee: V.V.Chaitanya Krishna Fix For: 0.21.0 Attachments: MAPREDUCE-28-1.txt, MAPREDUCE-28-2.txt, MAPREDUCE-28-3.txt, MAPREDUCE-28-4.txt, MAPREDUCE-28-5.txt, MAPREDUCE-28-6.txt, MAPREDUCE-28-7.txt, MAPREDUCE-28-8.patch TestQueueManager takes long time for the run and timeouts sometimes. See the failure at http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3875/testReport/. Looking at the console output, before the test finsihes, it was timed-out. On my machine, the test takes about 5 minutes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: (was: patch-1186-3-ydist.txt) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: patch-1186-3-ydist.txt Forgot --no-prefix While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783029#action_12783029 ] Amareshwari Sriramadasu commented on MAPREDUCE-1186: Patch looks fine to me. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783030#action_12783030 ] Hemanth Yamijala commented on MAPREDUCE-1231: - +1 for the change. Distcp is very slow --- Key: MAPREDUCE-1231 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.20.1 Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, mapred-1231-y20-v3.patch, mapred-1231-y20-v4.patch, mapred-1231-y20.patch, mapred-1231.patch Currently distcp does a checksums check in addition to file length check to decide if a remote file has to be copied. If the number of files is high (thousands), this checksum check is proving to be fairly costly leading to a long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work with retired jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1062: -- Status: Patch Available (was: Open) Re running thro' Hudson MRReliability test does not work with retired jobs -- Key: MAPREDUCE-1062 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Sreekanth Ramakrishnan Assignee: Sreekanth Ramakrishnan Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch Currently the MRReliability uses job clients get all job api which also includes retired jobs. In case we have retired jobs in cluster, The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work with retired jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1062: -- Status: Open (was: Patch Available) MRReliability test does not work with retired jobs -- Key: MAPREDUCE-1062 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Sreekanth Ramakrishnan Assignee: Sreekanth Ramakrishnan Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, mapreduce-ydist-20-1.patch Currently the MRReliability uses job clients get all job api which also includes retired jobs. In case we have retired jobs in cluster, The retired jobs are appended at the end of the job list, this results in Test always getting completed job and not spawning off KillTask thread and KillTracker threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783046#action_12783046 ] Hadoop QA commented on MAPREDUCE-1231: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426190/mapred-1231-v3.patch against trunk revision 884628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/272/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/272/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/272/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/272/console This message is automatically generated. Distcp is very slow --- Key: MAPREDUCE-1231 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.20.1 Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, mapred-1231-y20-v3.patch, mapred-1231-y20-v4.patch, mapred-1231-y20.patch, mapred-1231.patch Currently distcp does a checksums check in addition to file length check to decide if a remote file has to be copied. If the number of files is high (thousands), this checksum check is proving to be fairly costly leading to a long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783047#action_12783047 ] Jothi Padmanabhan commented on MAPREDUCE-1231: -- Test failures are unrelated. See MAPREDUCE-1124 and MAPREDUCE-1245 Distcp is very slow --- Key: MAPREDUCE-1231 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.20.1 Reporter: Jothi Padmanabhan Assignee: Jothi Padmanabhan Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, mapred-1231-y20-v3.patch, mapred-1231-y20-v4.patch, mapred-1231-y20.patch, mapred-1231.patch Currently distcp does a checksums check in addition to file length check to decide if a remote file has to be copied. If the number of files is high (thousands), this checksum check is proving to be fairly costly leading to a long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783048#action_12783048 ] Hadoop QA commented on MAPREDUCE-787: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426262/patch-787-3.txt against trunk revision 884628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/console This message is automatically generated. -files, -archives should honor user given symlink path -- Key: MAPREDUCE-787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, patch-787-3.txt, patch-787-ydist.txt, patch-787.txt Currently, if user gives an option such as -files hdfs://host:fs_port/user/testfile.txt#testlink The symlink name testlink is not honored. It alwasys creates symlink with name testfile.txt in cwd of the task. If the user has given a symlink name, it should be honored. If no symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783050#action_12783050 ] Amareshwari Sriramadasu commented on MAPREDUCE-787: --- Test failure is due to MAPREDUCE-1245 -files, -archives should honor user given symlink path -- Key: MAPREDUCE-787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, patch-787-3.txt, patch-787-ydist.txt, patch-787.txt Currently, if user gives an option such as -files hdfs://host:fs_port/user/testfile.txt#testlink The symlink name testlink is not honored. It alwasys creates symlink with name testfile.txt in cwd of the task. If the user has given a symlink name, it should be honored. If no symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
