[jira] [Commented] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[
https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539527#comment-13539527
]
Hadoop QA commented on MAPREDUCE-4839:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562377/partitioner8.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3172//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3172//console
This message is automatically generated.
TextPartioner for hashing Text with good hashing function to get better
distribution
Key: MAPREDUCE-4839
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Radim Kolar
Attachments: partitioner8.txt, textpartitioner1.txt,
textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt,
textpartitioner6.txt, textpartitioner7.txt
partitioner for Text keys using util.Hash framework for hashing function
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
Robert Joseph Evans created MAPREDUCE-4901: -- Summary: JobHistoryEventHandler errors should be fatal Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha, 0.23.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539581#comment-13539581 ] Robert Joseph Evans commented on MAPREDUCE-4901: Oh so looking at the code a bit more, it also appears that if an error does happen the thread that writes to the log dies and the event queue then grows without bound, so technically there is a memory leak involved here too. JobHistoryEventHandler errors should be fatal - Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[ https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-4902: -- Status: Patch Available (was: Open) Fix typo receievd should be received in log output -- Key: MAPREDUCE-4902 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Albert Chu Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-4902.patch Noticed a typo in the log output, receievd should be received org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0 sent hash and receievd reply -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[
https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539632#comment-13539632
]
Hadoop QA commented on MAPREDUCE-4902:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562391/MAPREDUCE-4902.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3173//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3173//console
This message is automatically generated.
Fix typo receievd should be received in log output
--
Key: MAPREDUCE-4902
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Albert Chu
Priority: Trivial
Fix For: 3.0.0
Attachments: MAPREDUCE-4902.patch
Noticed a typo in the log output, receievd should be received
org.apache.hadoop.mapreduce.task.reduce.Fetcher: for
url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0
sent hash and receievd reply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4894) Renewal / cancellation of JobHistory tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539651#comment-13539651 ] Siddharth Seth commented on MAPREDUCE-4894: --- bq. I'm not sure if there is a problem canceling JH tokens when the task completes You're right. Scratch my last two comments. Uploading a patch for branch 23. Renewal / cancellation of JobHistory tokens --- Key: MAPREDUCE-4894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4894 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.4 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MAPREDUCE-4894_wip.txt, MR-4894.txt Equivalent of YARN-50 for JobHistory tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4894) Renewal / cancellation of JobHistory tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4894: -- Attachment: MR-4894_branch0.23.txt Patch for branch-0.23 Renewal / cancellation of JobHistory tokens --- Key: MAPREDUCE-4894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4894 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.4 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MAPREDUCE-4894_wip.txt, MR-4894_branch0.23.txt, MR-4894.txt Equivalent of YARN-50 for JobHistory tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4896) mapred queue -info spits out ugly exception when queue does not exist
[ https://issues.apache.org/jira/browse/MAPREDUCE-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539658#comment-13539658 ] Sandy Ryza commented on MAPREDUCE-4896: --- Ah, I was testing with the fair scheduler. I just tried it with the capacity scheduler and got the trace you mentioned. This is because the capacity scheduler's getQueueInfo throws an IOException when the queue isn't found, and the fair scheduler's returns null. Does it make more sense for me to modify the capacity scheduler to return null or to catch the exception? My inclination would be to do the former because a non-existent queue isn't related to IO. Also, I'll add a test case. mapred queue -info spits out ugly exception when queue does not exist --- Key: MAPREDUCE-4896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4896 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4896-1.patch, MAPREDUCE-4896.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4901: --- Attachment: MR-4901-trunk.txt This patch should also work on branch-2. On branch-0.23 HADOOP-9196 will need to be applied first, but after that it works just fine there too. JobHistoryEventHandler errors should be fatal - Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Attachments: MR-4901-trunk.txt To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4901: --- Status: Patch Available (was: Open) JobHistoryEventHandler errors should be fatal - Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha, 0.23.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Attachments: MR-4901-trunk.txt To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[
https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539681#comment-13539681
]
Hadoop QA commented on MAPREDUCE-4901:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562403/MR-4901-trunk.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3174//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3174//console
This message is automatically generated.
JobHistoryEventHandler errors should be fatal
-
Key: MAPREDUCE-4901
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 2.0.0-alpha
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Attachments: MR-4901-trunk.txt
To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we
need a 2 phase commit where a subsequent AM can be sure that at a specific
point in time it knows exactly if any tasks/jobs are committing. The job
history log is already used for similar functionality so we would like to
reuse this, but we need to be sure that errors while writing out to the job
history log are now fatal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[ https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned MAPREDUCE-4902: - Assignee: Albert Chu +1, lgtm. Fix typo receievd should be received in log output -- Key: MAPREDUCE-4902 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Albert Chu Assignee: Albert Chu Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-4902.patch Noticed a typo in the log output, receievd should be received org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0 sent hash and receievd reply -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[ https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4902: -- Resolution: Fixed Fix Version/s: (was: 3.0.0) 0.23.6 2.0.3-alpha Target Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Albert. I committed this to trunk, branch-2, and branch-0.23. Note that in the future, please don't set the Fix Version/s field, as that is reserved for noting which versions actually contain the fix once committed. The Target Version/s field is what should be set instead, as it indicates which versions are targeted to be fixed. Fix typo receievd should be received in log output -- Key: MAPREDUCE-4902 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Albert Chu Assignee: Albert Chu Priority: Trivial Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4902.patch Noticed a typo in the log output, receievd should be received org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0 sent hash and receievd reply -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit
[
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539698#comment-13539698
]
Robert Joseph Evans commented on MAPREDUCE-4813:
A quick look at the patch looks OK to me. I need to dig into it in more detail.
Also the patch no longer compiles on trunk. Could you please upmerge. The
generated state transition charts make looking at the events a lot simpler, but
that does not work if the code does not compile.
AM timing out during job commit
---
Key: MAPREDUCE-4813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch,
MAPREDUCE-4813-2.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch,
MAPREDUCE-4813.patch
The AM calls the output committer's {{commitJob}} method synchronously during
JobImpl state transitions, which means the JobImpl write lock is held the
entire time the job is being committed. Holding the write lock prevents the
RM allocator thread from heartbeating to the RM. Therefore if committing the
job takes too long (e.g.: the job has tons of files to commit and/or the
namenode is bogged down) then the AM appears to be unresponsive to the RM and
the RM kills the AM attempt.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated MAPREDUCE-4839: --- Attachment: (was: partitioner8.txt) TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar reassigned MAPREDUCE-4839: -- Assignee: Radim Kolar TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Assignee: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated MAPREDUCE-4839: --- Attachment: textpartitioner8.txt TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Assignee: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539699#comment-13539699 ] Radim Kolar commented on MAPREDUCE-4839: attached wrong patch, retry. TextPartioner for hashing Text with good hashing function to get better distribution Key: MAPREDUCE-4839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Assignee: Radim Kolar Attachments: textpartitioner1.txt, textpartitioner2.txt, textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt, textpartitioner7.txt, textpartitioner8.txt partitioner for Text keys using util.Hash framework for hashing function -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4813) AM timing out during job commit
[
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4813:
--
Attachment: MAPREDUCE-4813-2.patch
Updated patch to trunk and cleaned up some unused import warnings.
AM timing out during job commit
---
Key: MAPREDUCE-4813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch,
MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813.patch,
MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
The AM calls the output committer's {{commitJob}} method synchronously during
JobImpl state transitions, which means the JobImpl write lock is held the
entire time the job is being committed. Holding the write lock prevents the
RM allocator thread from heartbeating to the RM. Therefore if committing the
job takes too long (e.g.: the job has tons of files to commit and/or the
namenode is bogged down) then the AM appears to be unresponsive to the RM and
the RM kills the AM attempt.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4839) TextPartioner for hashing Text with good hashing function to get better distribution
[
https://issues.apache.org/jira/browse/MAPREDUCE-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539709#comment-13539709
]
Hadoop QA commented on MAPREDUCE-4839:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562408/textpartitioner8.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3175//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3175//console
This message is automatically generated.
TextPartioner for hashing Text with good hashing function to get better
distribution
Key: MAPREDUCE-4839
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4839
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Radim Kolar
Assignee: Radim Kolar
Attachments: textpartitioner1.txt, textpartitioner2.txt,
textpartitioner3.txt, textpartitioner4.txt, textpartitioner6.txt,
textpartitioner7.txt, textpartitioner8.txt
partitioner for Text keys using util.Hash framework for hashing function
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit
[
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539710#comment-13539710
]
Hadoop QA commented on MAPREDUCE-4813:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562409/MAPREDUCE-4813-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3176//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3176//console
This message is automatically generated.
AM timing out during job commit
---
Key: MAPREDUCE-4813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch,
MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813.patch,
MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
The AM calls the output committer's {{commitJob}} method synchronously during
JobImpl state transitions, which means the JobImpl write lock is held the
entire time the job is being committed. Holding the write lock prevents the
RM allocator thread from heartbeating to the RM. Therefore if committing the
job takes too long (e.g.: the job has tons of files to commit and/or the
namenode is bogged down) then the AM appears to be unresponsive to the RM and
the RM kills the AM attempt.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4900) Dynamic configuration for task slots on TT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-4900: - Target Version/s: 1.2.0 Affects Version/s: 1.1.1 Dynamic configuration for task slots on TT -- Key: MAPREDUCE-4900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4900 Project: Hadoop Map/Reduce Issue Type: New Feature Components: resourcemanager, tasktracker Affects Versions: 1.1.1 Reporter: Junping Du Assignee: Binglin Chang The current Hadoop MRV1 resource management logic assumes per node slot number is static during the lifetime of the TT process. Allowing run-time configuration on per node slot will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. For more background or design details of this effort, please refer proposal in HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4900) Dynamic configuration for task slots on TT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-4900: - Attachment: MAPREDUCE-4900-demo.patch A demo patch to demonstrate how this can be done, not the final solution. Dynamic configuration for task slots on TT -- Key: MAPREDUCE-4900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4900 Project: Hadoop Map/Reduce Issue Type: New Feature Components: resourcemanager, tasktracker Affects Versions: 1.1.1 Reporter: Junping Du Assignee: Binglin Chang Attachments: MAPREDUCE-4900-demo.patch The current Hadoop MRV1 resource management logic assumes per node slot number is static during the lifetime of the TT process. Allowing run-time configuration on per node slot will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. For more background or design details of this effort, please refer proposal in HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
