[jira] [Commented] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539527#comment-13539527 ] Hadoop QA commented on MAPREDUCE-4839: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562377/partitioner8.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3172//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3172//console This message is automatically generated. TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Attachments: partitioner8.txt, textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
Robert Joseph Evans created MAPREDUCE-4901: -- Summary: JobHistoryEventHandler errors should be fatal Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha, 0.23.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539581#comment-13539581 ] Robert Joseph Evans commented on MAPREDUCE-4901: Oh so looking at the code a bit more, it also appears that if an error does happen the thread that writes to the log dies and the event queue then grows without bound, so technically there is a memory leak involved here too. JobHistoryEventHandler errors should be fatal - Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[ https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-4902: -- Status: Patch Available (was: Open) Fix typo receievd should be received in log output -- Key: MAPREDUCE-4902 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Albert Chu Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-4902.patch Noticed a typo in the log output, receievd should be received org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0 sent hash and receievd reply -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[ https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539632#comment-13539632 ] Hadoop QA commented on MAPREDUCE-4902: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562391/MAPREDUCE-4902.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3173//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3173//console This message is automatically generated. Fix typo receievd should be received in log output -- Key: MAPREDUCE-4902 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Albert Chu Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-4902.patch Noticed a typo in the log output, receievd should be received org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0 sent hash and receievd reply -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4894) Renewal / cancellation of JobHistory tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539651#comment-13539651 ] Siddharth Seth commented on MAPREDUCE-4894: --- bq. I'm not sure if there is a problem canceling JH tokens when the task completes You're right. Scratch my last two comments. Uploading a patch for branch 23. Renewal / cancellation of JobHistory tokens --- Key: MAPREDUCE-4894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4894 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.4 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MAPREDUCE-4894_wip.txt, MR-4894.txt Equivalent of YARN-50 for JobHistory tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4894) Renewal / cancellation of JobHistory tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4894: -- Attachment: MR-4894_branch0.23.txt Patch for branch-0.23 Renewal / cancellation of JobHistory tokens --- Key: MAPREDUCE-4894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4894 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.4 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MAPREDUCE-4894_wip.txt, MR-4894_branch0.23.txt, MR-4894.txt Equivalent of YARN-50 for JobHistory tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4896) mapred queue -info spits out ugly exception when queue does not exist
[ https://issues.apache.org/jira/browse/MAPREDUCE-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539658#comment-13539658 ] Sandy Ryza commented on MAPREDUCE-4896: --- Ah, I was testing with the fair scheduler. I just tried it with the capacity scheduler and got the trace you mentioned. This is because the capacity scheduler's getQueueInfo throws an IOException when the queue isn't found, and the fair scheduler's returns null. Does it make more sense for me to modify the capacity scheduler to return null or to catch the exception? My inclination would be to do the former because a non-existent queue isn't related to IO. Also, I'll add a test case. mapred queue -info spits out ugly exception when queue does not exist --- Key: MAPREDUCE-4896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4896 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4896-1.patch, MAPREDUCE-4896.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4901: --- Attachment: MR-4901-trunk.txt This patch should also work on branch-2. On branch-0.23 HADOOP-9196 will need to be applied first, but after that it works just fine there too. JobHistoryEventHandler errors should be fatal - Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Attachments: MR-4901-trunk.txt To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4901: --- Status: Patch Available (was: Open) JobHistoryEventHandler errors should be fatal - Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha, 0.23.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Attachments: MR-4901-trunk.txt To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539681#comment-13539681 ] Hadoop QA commented on MAPREDUCE-4901: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562403/MR-4901-trunk.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3174//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3174//console This message is automatically generated. JobHistoryEventHandler errors should be fatal - Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Attachments: MR-4901-trunk.txt To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[ https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned MAPREDUCE-4902: - Assignee: Albert Chu +1, lgtm. Fix typo receievd should be received in log output -- Key: MAPREDUCE-4902 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Albert Chu Assignee: Albert Chu Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-4902.patch Noticed a typo in the log output, receievd should be received org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0 sent hash and receievd reply -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[ https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4902: -- Resolution: Fixed Fix Version/s: (was: 3.0.0) 0.23.6 2.0.3-alpha Target Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Albert. I committed this to trunk, branch-2, and branch-0.23. Note that in the future, please don't set the Fix Version/s field, as that is reserved for noting which versions actually contain the fix once committed. The Target Version/s field is what should be set instead, as it indicates which versions are targeted to be fixed. Fix typo receievd should be received in log output -- Key: MAPREDUCE-4902 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Albert Chu Assignee: Albert Chu Priority: Trivial Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4902.patch Noticed a typo in the log output, receievd should be received org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0 sent hash and receievd reply -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit
[ https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539698#comment-13539698 ] Robert Joseph Evans commented on MAPREDUCE-4813: A quick look at the patch looks OK to me. I need to dig into it in more detail. Also the patch no longer compiles on trunk. Could you please upmerge. The generated state transition charts make looking at the events a lot simpler, but that does not work if the code does not compile. AM timing out during job commit --- Key: MAPREDUCE-4813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 0.23.3, 2.0.1-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch The AM calls the output committer's {{commitJob}} method synchronously during JobImpl state transitions, which means the JobImpl write lock is held the entire time the job is being committed. Holding the write lock prevents the RM allocator thread from heartbeating to the RM. Therefore if committing the job takes too long (e.g.: the job has tons of files to commit and/or the namenode is bogged down) then the AM appears to be unresponsive to the RM and the RM kills the AM attempt. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated MAPREDUCE-4839: --- Attachment: (was: partitioner8.txt) TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar reassigned MAPREDUCE-4839: -- Assignee: Radim Kolar TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Assignee: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated MAPREDUCE-4839: --- Attachment: textpartitioner8.txt TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Assignee: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539699#comment-13539699 ] Radim Kolar commented on MAPREDUCE-4839: attached wrong patch, retry. TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Assignee: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4813) AM timing out during job commit
[ https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4813: -- Attachment: MAPREDUCE-4813-2.patch Updated patch to trunk and cleaned up some unused import warnings. AM timing out during job commit --- Key: MAPREDUCE-4813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 0.23.3, 2.0.1-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch The AM calls the output committer's {{commitJob}} method synchronously during JobImpl state transitions, which means the JobImpl write lock is held the entire time the job is being committed. Holding the write lock prevents the RM allocator thread from heartbeating to the RM. Therefore if committing the job takes too long (e.g.: the job has tons of files to commit and/or the namenode is bogged down) then the AM appears to be unresponsive to the RM and the RM kills the AM attempt. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539709#comment-13539709 ] Hadoop QA commented on MAPREDUCE-4839: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562408/textpartitioner8.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3175//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3175//console This message is automatically generated. TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Assignee: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit
[ https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539710#comment-13539710 ] Hadoop QA commented on MAPREDUCE-4813: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562409/MAPREDUCE-4813-2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3176//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3176//console This message is automatically generated. AM timing out during job commit --- Key: MAPREDUCE-4813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 0.23.3, 2.0.1-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch The AM calls the output committer's {{commitJob}} method synchronously during JobImpl state transitions, which means the JobImpl write lock is held the entire time the job is being committed. Holding the write lock prevents the RM allocator thread from heartbeating to the RM. Therefore if committing the job takes too long (e.g.: the job has tons of files to commit and/or the namenode is bogged down) then the AM appears to be unresponsive to the RM and the RM kills the AM attempt. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4900) Dynamic configuration for task slots on TT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-4900: - Target Version/s: 1.2.0 Affects Version/s: 1.1.1 Dynamic configuration for task slots on TT -- Key: MAPREDUCE-4900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4900 Project: Hadoop Map/Reduce Issue Type: New Feature Components: resourcemanager, tasktracker Affects Versions: 1.1.1 Reporter: Junping Du Assignee: Binglin Chang The current Hadoop MRV1 resource management logic assumes per node slot number is static during the lifetime of the TT process. Allowing run-time configuration on per node slot will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. For more background or design details of this effort, please refer proposal in HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4900) Dynamic configuration for task slots on TT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-4900: - Attachment: MAPREDUCE-4900-demo.patch A demo patch to demonstrate how this can be done, not the final solution. Dynamic configuration for task slots on TT -- Key: MAPREDUCE-4900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4900 Project: Hadoop Map/Reduce Issue Type: New Feature Components: resourcemanager, tasktracker Affects Versions: 1.1.1 Reporter: Junping Du Assignee: Binglin Chang Attachments: MAPREDUCE-4900-demo.patch The current Hadoop MRV1 resource management logic assumes per node slot number is static during the lifetime of the TT process. Allowing run-time configuration on per node slot will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. For more background or design details of this effort, please refer proposal in HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira