[jira] Commented: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783050#action_12783050 ] Amareshwari Sriramadasu commented on MAPREDUCE-787: --- Test failure is due to MAPREDUCE-1245 > -files, -archives should honor user given symlink path > -- > > Key: MAPREDUCE-787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, > patch-787-3.txt, patch-787-ydist.txt, patch-787.txt > > > Currently, if user gives an option such as > -files hdfs://host:fs_port/user/testfile.txt#testlink > The symlink name "testlink" is not honored. It alwasys creates symlink with > name testfile.txt in cwd of the task. > If the user has given a symlink name, it should be honored. If no > symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783048#action_12783048 ] Hadoop QA commented on MAPREDUCE-787: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426262/patch-787-3.txt against trunk revision 884628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/console This message is automatically generated. > -files, -archives should honor user given symlink path > -- > > Key: MAPREDUCE-787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, > patch-787-3.txt, patch-787-ydist.txt, patch-787.txt > > > Currently, if user gives an option such as > -files hdfs://host:fs_port/user/testfile.txt#testlink > The symlink name "testlink" is not honored. It alwasys creates symlink with > name testfile.txt in cwd of the task. > If the user has given a symlink name, it should be honored. If no > symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783047#action_12783047 ] Jothi Padmanabhan commented on MAPREDUCE-1231: -- Test failures are unrelated. See MAPREDUCE-1124 and MAPREDUCE-1245 > Distcp is very slow > --- > > Key: MAPREDUCE-1231 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 0.20.1 >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, > mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, > mapred-1231-y20-v3.patch, mapred-1231-y20-v4.patch, mapred-1231-y20.patch, > mapred-1231.patch > > > Currently distcp does a checksums check in addition to file length check to > decide if a remote file has to be copied. If the number of files is high > (thousands), this checksum check is proving to be fairly costly leading to a > long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783046#action_12783046 ] Hadoop QA commented on MAPREDUCE-1231: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426190/mapred-1231-v3.patch against trunk revision 884628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/272/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/272/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/272/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/272/console This message is automatically generated. > Distcp is very slow > --- > > Key: MAPREDUCE-1231 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 0.20.1 >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, > mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, > mapred-1231-y20-v3.patch, mapred-1231-y20-v4.patch, mapred-1231-y20.patch, > mapred-1231.patch > > > Currently distcp does a checksums check in addition to file length check to > decide if a remote file has to be copied. If the number of files is high > (thousands), this checksum check is proving to be fairly costly leading to a > long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work with retired jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1062: -- Status: Patch Available (was: Open) Re running thro' Hudson > MRReliability test does not work with retired jobs > -- > > Key: MAPREDUCE-1062 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Sreekanth Ramakrishnan >Assignee: Sreekanth Ramakrishnan > Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, > mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, > mapreduce-ydist-20-1.patch > > > Currently the MRReliability uses job clients get all job api which also > includes retired jobs. > In case we have retired jobs in cluster, > The retired jobs are appended at the end of the job list, this results in > Test always getting completed job and not spawning off KillTask thread and > KillTracker threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1062) MRReliability test does not work with retired jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1062: -- Status: Open (was: Patch Available) > MRReliability test does not work with retired jobs > -- > > Key: MAPREDUCE-1062 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1062 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Sreekanth Ramakrishnan >Assignee: Sreekanth Ramakrishnan > Attachments: mapreduce-1062-1.patch, mapreduce-1062-2.patch, > mapreduce-1062-3-ydist.patch, mapreduce-1062-3.patch, > mapreduce-ydist-20-1.patch > > > Currently the MRReliability uses job clients get all job api which also > includes retired jobs. > In case we have retired jobs in cluster, > The retired jobs are appended at the end of the job list, this results in > Test always getting completed job and not spawning off KillTask thread and > KillTracker threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1075) getQueue(String queue) in JobTracker would return NPE for invalid queue name
[ https://issues.apache.org/jira/browse/MAPREDUCE-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] V.V.Chaitanya Krishna updated MAPREDUCE-1075: - Status: Patch Available (was: Open) Running through hudson. > getQueue(String queue) in JobTracker would return NPE for invalid queue name > > > Key: MAPREDUCE-1075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1075 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: V.V.Chaitanya Krishna >Assignee: V.V.Chaitanya Krishna > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1075-1.patch, MAPREDUCE-1075-2.patch, > MAPREDUCE-1075-3.patch, MAPREDUCE-1075-4.patch, MAPREDUCE-1075-5.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783030#action_12783030 ] Hemanth Yamijala commented on MAPREDUCE-1231: - +1 for the change. > Distcp is very slow > --- > > Key: MAPREDUCE-1231 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 0.20.1 >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, > mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, > mapred-1231-y20-v3.patch, mapred-1231-y20-v4.patch, mapred-1231-y20.patch, > mapred-1231.patch > > > Currently distcp does a checksums check in addition to file length check to > decide if a remote file has to be copied. If the number of files is high > (thousands), this checksum check is proving to be fairly costly leading to a > long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783029#action_12783029 ] Amareshwari Sriramadasu commented on MAPREDUCE-1186: Patch looks fine to me. > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, > patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: patch-1186-3-ydist.txt Forgot --no-prefix > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, > patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-1231: - Attachment: mapred-1231-y20-v4.patch Thanks for the review, Hemanth. And for the good catch. Here is a patch with the option name changed to .skip.crc.check > Distcp is very slow > --- > > Key: MAPREDUCE-1231 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 0.20.1 >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, > mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, > mapred-1231-y20-v3.patch, mapred-1231-y20-v4.patch, mapred-1231-y20.patch, > mapred-1231.patch > > > Currently distcp does a checksums check in addition to file length check to > decide if a remote file has to be copied. If the number of files is high > (thousands), this checksum check is proving to be fairly costly leading to a > long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-28) TestQueueManager takes too long and times out some times
[ https://issues.apache.org/jira/browse/MAPREDUCE-28?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-28: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed this to 0.21 and trunk. Thanks Chaitanya and Rahul. > TestQueueManager takes too long and times out some times > > > Key: MAPREDUCE-28 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-28 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker, test >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: V.V.Chaitanya Krishna > Fix For: 0.21.0 > > Attachments: MAPREDUCE-28-1.txt, MAPREDUCE-28-2.txt, > MAPREDUCE-28-3.txt, MAPREDUCE-28-4.txt, MAPREDUCE-28-5.txt, > MAPREDUCE-28-6.txt, MAPREDUCE-28-7.txt, MAPREDUCE-28-8.patch > > > TestQueueManager takes long time for the run and timeouts sometimes. > See the failure at > http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3875/testReport/. > Looking at the console output, before the test finsihes, it was timed-out. > On my machine, the test takes about 5 minutes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: (was: patch-1186-3-ydist.txt) > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-1186-3-ydist.txt, patch-1186-ydist.txt, > patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: patch-1186-3-ydist.txt I dither, I preferred both 'unique' and 'parent' prefixes for CacheStatus.parentDir, so I renamed it to be CacheStatus.uniqueParentDir... no other changes. *grazing at my navel* > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, > patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783023#action_12783023 ] Hemanth Yamijala commented on MAPREDUCE-1231: - I looked at the Yahoo! Hadoop 0.20 patch. One minor nit is that the internal config option name is different between this and the trunk patch. In the trunk patch, the option is distcp.skip.crc.check. In the internal patch it is distcp.skip.crc. Since this is a jobconf option, it may be better to keep these in sync. At the very least, it avoids confusion when Hadoop is upgraded to the trunk version. Other than this, the 20 patch looks good. Another point, (unrelated to this JIRA), is that the way the post-copy validation is done between trunk and 20 seems different. In trunk, this is done by a call to the API sameFile(). Hence, it includes CRC checks by default. In the internal 20 patch, this check is done only on file lengths irrespective of the option to skip crc checks. It is unclear whether this is by design. At any rate, this inconsistency is not related to this patch. > Distcp is very slow > --- > > Key: MAPREDUCE-1231 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 0.20.1 >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, > mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, > mapred-1231-y20-v3.patch, mapred-1231-y20.patch, mapred-1231.patch > > > Currently distcp does a checksums check in addition to file length check to > decide if a remote file has to be copied. If the number of files is high > (thousands), this checksum check is proving to be fairly costly leading to a > long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Status: Open (was: Patch Available) Retrying hudson > TestFairScheduler fails with "too many open files" error > > > Key: MAPREDUCE-1245 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Vinod K V > Fix For: 0.21.0 > > Attachments: 1245_v1.patch > > > This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Status: Patch Available (was: Open) > TestFairScheduler fails with "too many open files" error > > > Key: MAPREDUCE-1245 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Vinod K V > Fix For: 0.21.0 > > Attachments: 1245_v1.patch > > > This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783019#action_12783019 ] Amareshwari Sriramadasu commented on MAPREDUCE-787: --- Re-ran ant test in branch 0.21, with the patch. All the tests passed except TestGridmixSubmission (due to MAPREDUCE-1124) > -files, -archives should honor user given symlink path > -- > > Key: MAPREDUCE-787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, > patch-787-3.txt, patch-787-ydist.txt, patch-787.txt > > > Currently, if user gives an option such as > -files hdfs://host:fs_port/user/testfile.txt#testlink > The symlink name "testlink" is not honored. It alwasys creates symlink with > name testfile.txt in cwd of the task. > If the user has given a symlink name, it should be honored. If no > symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-787: -- Attachment: patch-787-3.txt Re-submitting the same patch for hudson, since the common jars will be picked up from maven. > -files, -archives should honor user given symlink path > -- > > Key: MAPREDUCE-787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, > patch-787-3.txt, patch-787-ydist.txt, patch-787.txt > > > Currently, if user gives an option such as > -files hdfs://host:fs_port/user/testfile.txt#testlink > The symlink name "testlink" is not honored. It alwasys creates symlink with > name testfile.txt in cwd of the task. > If the user has given a symlink name, it should be honored. If no > symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-787: -- Status: Patch Available (was: Open) > -files, -archives should honor user given symlink path > -- > > Key: MAPREDUCE-787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, > patch-787-3.txt, patch-787-ydist.txt, patch-787.txt > > > Currently, if user gives an option such as > -files hdfs://host:fs_port/user/testfile.txt#testlink > The symlink name "testlink" is not honored. It alwasys creates symlink with > name testfile.txt in cwd of the task. > If the user has given a symlink name, it should be honored. If no > symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-787) -files, -archives should honor user given symlink path
[ https://issues.apache.org/jira/browse/MAPREDUCE-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-787: -- Status: Open (was: Patch Available) > -files, -archives should honor user given symlink path > -- > > Key: MAPREDUCE-787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-787-1.txt, patch-787-2.txt, patch-787-3.txt, > patch-787-ydist.txt, patch-787.txt > > > Currently, if user gives an option such as > -files hdfs://host:fs_port/user/testfile.txt#testlink > The symlink name "testlink" is not honored. It alwasys creates symlink with > name testfile.txt in cwd of the task. > If the user has given a symlink name, it should be honored. If no > symlink-name is given, then the path.getName() can be used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-1231: - Status: Patch Available (was: Open) Trying to pass through Hudson once more > Distcp is very slow > --- > > Key: MAPREDUCE-1231 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 0.20.1 >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, > mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, > mapred-1231-y20-v3.patch, mapred-1231-y20.patch, mapred-1231.patch > > > Currently distcp does a checksums check in addition to file length check to > decide if a remote file has to be copied. If the number of files is high > (thousands), this checksum check is proving to be fairly costly leading to a > long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1231) Distcp is very slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-1231: - Status: Open (was: Patch Available) > Distcp is very slow > --- > > Key: MAPREDUCE-1231 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1231 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 0.20.1 >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapred-1231-v1.patch, mapred-1231-v2.patch, > mapred-1231-v3.patch, mapred-1231-v3.patch, mapred-1231-y20-v2.patch, > mapred-1231-y20-v3.patch, mapred-1231-y20.patch, mapred-1231.patch > > > Currently distcp does a checksums check in addition to file length check to > decide if a remote file has to be copied. If the number of files is high > (thousands), this checksum check is proving to be fairly costly leading to a > long time before the copy is started. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782965#action_12782965 ] Hudson commented on MAPREDUCE-1239: --- Integrated in Hadoop-Mapreduce-trunk-Commit #135 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/135/]) . Fix contrib components build dependencies. (Giridharan Kesavan and omalley) Move the change message into 0.21.0. > Mapreduce test build is broken after HADOOP-5107 > > > Key: MAPREDUCE-1239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V >Assignee: Giridharan Kesavan >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: m-1239.patch, mapred-1239.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved MAPREDUCE-1239. -- Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] I committed this to 0.21 also. > Mapreduce test build is broken after HADOOP-5107 > > > Key: MAPREDUCE-1239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V >Assignee: Giridharan Kesavan >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: m-1239.patch, mapred-1239.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782909#action_12782909 ] Hadoop QA commented on MAPREDUCE-1245: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426216/1245_v1.patch against trunk revision 884531. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/console This message is automatically generated. > TestFairScheduler fails with "too many open files" error > > > Key: MAPREDUCE-1245 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Vinod K V > Fix For: 0.21.0 > > Attachments: 1245_v1.patch > > > This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1182) Reducers fail with OutOfMemoryError while copying Map outputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782864#action_12782864 ] Hudson commented on MAPREDUCE-1182: --- Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) > Reducers fail with OutOfMemoryError while copying Map outputs > - > > Key: MAPREDUCE-1182 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1182 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chandra Prakash Bhagtani >Assignee: Chandra Prakash Bhagtani >Priority: Blocker > Fix For: 0.20.2 > > Attachments: HADOOP-6357.patch, M1182-0.patch, M1182-0v20.patch, > M1182-1.patch, M1182-1v20.patch > > > Reducers fail while copying Map outputs with following exception > java.lang.OutOfMemoryError: Java heap space at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1539) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216) > ,Error: > Reducer's memory usage keeps on increasing and ultimately exceeds -Xmx value > I even tried with -Xmx6.5g to each reducer but it's still failing > While looking into the reducer logs, I found that reducers were doing > shuffleInMemory every time, rather than doing shuffleOnDisk -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-28) TestQueueManager takes too long and times out some times
[ https://issues.apache.org/jira/browse/MAPREDUCE-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782861#action_12782861 ] Hudson commented on MAPREDUCE-28: - Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) > TestQueueManager takes too long and times out some times > > > Key: MAPREDUCE-28 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-28 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker, test >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: V.V.Chaitanya Krishna > Fix For: 0.21.0 > > Attachments: MAPREDUCE-28-1.txt, MAPREDUCE-28-2.txt, > MAPREDUCE-28-3.txt, MAPREDUCE-28-4.txt, MAPREDUCE-28-5.txt, > MAPREDUCE-28-6.txt, MAPREDUCE-28-7.txt, MAPREDUCE-28-8.patch > > > TestQueueManager takes long time for the run and timeouts sometimes. > See the failure at > http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3875/testReport/. > Looking at the console output, before the test finsihes, it was timed-out. > On my machine, the test takes about 5 minutes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782865#action_12782865 ] Hudson commented on MAPREDUCE-1244: --- Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) . Fix eclipse-plugin's build dependencies. > eclipse-plugin fails with missing dependencies > -- > > Key: MAPREDUCE-1244 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Fix For: 0.22.0 > > Attachments: mapred-1244.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782862#action_12782862 ] Hudson commented on MAPREDUCE-1239: --- Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) > Mapreduce test build is broken after HADOOP-5107 > > > Key: MAPREDUCE-1239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V >Assignee: Giridharan Kesavan >Priority: Blocker > Fix For: 0.22.0 > > Attachments: m-1239.patch, mapred-1239.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V reopened MAPREDUCE-1239: -- This has to be fixed in 0.21 too as HADOOP-5107 was pulled into 0.21 branch. > Mapreduce test build is broken after HADOOP-5107 > > > Key: MAPREDUCE-1239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V >Assignee: Giridharan Kesavan >Priority: Blocker > Fix For: 0.22.0 > > Attachments: m-1239.patch, mapred-1239.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan resolved MAPREDUCE-1244. --- Resolution: Fixed Fix Version/s: 0.22.0 > eclipse-plugin fails with missing dependencies > -- > > Key: MAPREDUCE-1244 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Fix For: 0.22.0 > > Attachments: mapred-1244.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1239: -- Resolution: Fixed Fix Version/s: (was: 0.21.0) 0.22.0 Status: Resolved (was: Patch Available) I just committed this to trunk, thanks Owen > Mapreduce test build is broken after HADOOP-5107 > > > Key: MAPREDUCE-1239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V >Assignee: Giridharan Kesavan >Priority: Blocker > Fix For: 0.22.0 > > Attachments: m-1239.patch, mapred-1239.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Status: Patch Available (was: Open) > TestFairScheduler fails with "too many open files" error > > > Key: MAPREDUCE-1245 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Vinod K V > Fix For: 0.21.0 > > Attachments: 1245_v1.patch > > > This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Attachment: 1245_v1.patch Use a lightweight JT for tests. > TestFairScheduler fails with "too many open files" error > > > Key: MAPREDUCE-1245 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Vinod K V > Fix For: 0.21.0 > > Attachments: 1245_v1.patch > > > This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-3-ydist.txt Patch renaming uniqueSubDir. I think the log will be useful in debugging, so leaving it as is. Verified that chmod is not done on baseDir, by manually verifying logs. Reproduced the issue by putting about 1Lakh files in base dir, and running a job using 5 files on distributed cache. Verified that the same job, under same environment, finishes faster with the patch. Verified that streaming and pipes jobs, which use executables from distributed cache, work fine (with both DefaultTaskController and LinuxTaskController). Thanks Karthikeyan for helping in testing. test-patch and ant test passed on local machine, except TestHdfsProxy. > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-1186-3-ydist.txt, patch-1186-ydist.txt, > patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[
https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782813#action_12782813
]
Vinod K V commented on MAPREDUCE-1245:
--
Here's the problem as mentioned
[here|https://issues.apache.org/jira/browse/MAPREDUCE-1239?focusedCommentId=12782742&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12782742]:
{code}
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker up at: 32801
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker webserver:
44348
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: Cleaning up the system
directory
[junit] 09/11/25 16:34:08 INFO mapred.JobTracker: problem cleaning system
directory: file:/tmp/hadoop-sreerama/mapred/system
[junit] java.io.IOException: Cannot run program "chmod":
java.io.IOException: error=24, Too many open files
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:474)
[junit] at org.apache.hadoop.util.Shell.runCommand(Shell.java:188)
[junit] at org.apache.hadoop.util.Shell.run(Shell.java:170)
[junit] at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:363)
[junit] at org.apache.hadoop.util.Shell.execCommand(Shell.java:449)
[junit] at org.apache.hadoop.util.Shell.execCommand(Shell.java:432)
[junit] at
org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
[junit] at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
[junit] at
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:281)
[junit] at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:295)
[junit] at
org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1477)
[junit] at
org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1306)
[junit] at
org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1299)
[junit] at
org.apache.hadoop.mapred.UtilsForTests.getJobTracker(UtilsForTests.java:712)
[junit] at
org.apache.hadoop.mapred.TestFairScheduler.testPoolAssignment(TestFairScheduler.java:2546)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at java.lang.reflect.Method.invoke(Method.java:616)
[junit] at junit.framework.TestCase.runTest(TestCase.java:168)
[junit] at junit.framework.TestCase.runBare(TestCase.java:134)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
[junit] at junit.framework.TestResult.run(TestResult.java:113)
[junit] at junit.framework.TestCase.run(TestCase.java:124)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:232)
[junit] at junit.framework.TestSuite.run(TestSuite.java:227)
[junit] at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
[junit] at
junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
[junit] Caused by: java.io.IOException: java.io.IOException: error=24, Too
many open files
[junit] at java.lang.UNIXProcess.(UNIXProcess.java:164)
[junit] at java.lang.ProcessImpl.start(ProcessImpl.java:81)
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:467)
[junit] ... 31 more
{code}
This happens because every {{submitJob()}} creates a new {{JobTracker}} object.
The same problem plagues other tests too and drastically slows them down.
Creating a single object and use it repetitively should suffice.
> TestFairScheduler fails with "too many open files" error
>
>
> Key: MAPREDUCE-1245
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0
>Reporter: Vinod K V
> Fix For: 0.21.0
>
>
> This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782810#action_12782810 ] Vinod K V commented on MAPREDUCE-1239: -- MAPREDUCE-1245 is created for the TestFairScheduler failure. > Mapreduce test build is broken after HADOOP-5107 > > > Key: MAPREDUCE-1239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V >Assignee: Giridharan Kesavan >Priority: Blocker > Fix For: 0.21.0 > > Attachments: m-1239.patch, mapred-1239.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
TestFairScheduler fails with "too many open files" error Key: MAPREDUCE-1245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Vinod K V Fix For: 0.21.0 This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1244: -- Component/s: build updated components and affected versions > eclipse-plugin fails with missing dependencies > -- > > Key: MAPREDUCE-1244 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: mapred-1244.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1244: -- Affects Version/s: 0.22.0 > eclipse-plugin fails with missing dependencies > -- > > Key: MAPREDUCE-1244 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: mapred-1244.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1244: -- Attachment: mapred-1244.patch > eclipse-plugin fails with missing dependencies > -- > > Key: MAPREDUCE-1244 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: mapred-1244.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1009) Forrest documentation needs to be updated to describes features provided for supporting hierarchical queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1009: - Status: Patch Available (was: Open) > Forrest documentation needs to be updated to describes features provided for > supporting hierarchical queues > --- > > Key: MAPREDUCE-1009 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1009 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.21.0 >Reporter: Hemanth Yamijala >Assignee: Vinod K V >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1009-20091008.txt, > MAPREDUCE-1009-20091116.txt, MAPREDUCE-1009-20091124.txt > > > Forrest documentation must be updated for describing how to set up and use > hierarchical queues in the framework and the capacity scheduler. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1244) eclipse-plugin fails with missing dependencies
eclipse-plugin fails with missing dependencies -- Key: MAPREDUCE-1244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1244 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1110) Currently QueueManager API's return JobQueueInfo , instead it should return QueueInfo, similarly wher ever we are using JobQueueInfo as view , we should use QueueInfo
[ https://issues.apache.org/jira/browse/MAPREDUCE-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1110: - Attachment: MAPREDUCE-1110-20091126.1.txt Uploading a new patch that has to be applied over the latest patches at MAPREDUCE-1082 and MAPREDUCE-1075 in that order. This patch also fixes the incompatibility issues with JobQueueInfo. > Currently QueueManager API's return JobQueueInfo , instead it should return > QueueInfo, similarly wher ever we are using JobQueueInfo as view , we should > use QueueInfo > --- > > Key: MAPREDUCE-1110 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1110 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: rahul k singh >Assignee: Vinod K V >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1110-20091119.1.txt, > MAPREDUCE-1110-20091119.txt, MAPREDUCE-1110-20091126.1.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1009) Forrest documentation needs to be updated to describes features provided for supporting hierarchical queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782800#action_12782800 ] rahul k singh commented on MAPREDUCE-1009: -- +1 with the patch > Forrest documentation needs to be updated to describes features provided for > supporting hierarchical queues > --- > > Key: MAPREDUCE-1009 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1009 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.21.0 >Reporter: Hemanth Yamijala >Assignee: Vinod K V >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1009-20091008.txt, > MAPREDUCE-1009-20091116.txt, MAPREDUCE-1009-20091124.txt > > > Forrest documentation must be updated for describing how to set up and use > hierarchical queues in the framework and the capacity scheduler. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782796#action_12782796 ] Giridharan Kesavan commented on MAPREDUCE-1239: --- let me commit this to fix the trunk build failure and file a jira for the fairscheduler test failure "TestFairScheduler" > Mapreduce test build is broken after HADOOP-5107 > > > Key: MAPREDUCE-1239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V >Assignee: Giridharan Kesavan >Priority: Blocker > Fix For: 0.21.0 > > Attachments: m-1239.patch, mapred-1239.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1239) Mapreduce test build is broken after HADOOP-5107
[ https://issues.apache.org/jira/browse/MAPREDUCE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782795#action_12782795 ] Giridharan Kesavan commented on MAPREDUCE-1239: --- tested m-1239.patch and hit the same exception as sreekant mentioned. Otherwise this patch looks good. +1 > Mapreduce test build is broken after HADOOP-5107 > > > Key: MAPREDUCE-1239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1239 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V >Assignee: Giridharan Kesavan >Priority: Blocker > Fix For: 0.21.0 > > Attachments: m-1239.patch, mapred-1239.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
