[jira] [Updated] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-3360: -- Attachment: MAPREDUCE-3360-5.patch Addressing above comments Provide information about lost nodes in the UI. --- Key: MAPREDUCE-3360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Environment: NA Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Priority: Critical Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, MAPREDUCE-3360-5.patch, MAPREDUCE-3360.patch, lostNodes.png Currently there is no information provided about *lost nodes*. Provide information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-3360: -- Status: Open (was: Patch Available) Provide information about lost nodes in the UI. --- Key: MAPREDUCE-3360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Environment: NA Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Priority: Critical Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, MAPREDUCE-3360-5.patch, MAPREDUCE-3360.patch, lostNodes.png Currently there is no information provided about *lost nodes*. Provide information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2640) The maxRunningTasks property of the LimitTasksPerJob scheduler is ambiguous in its description, and must be updated
[
https://issues.apache.org/jira/browse/MAPREDUCE-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-2640:
---
Resolution: Not A Problem
Fix Version/s: (was: 0.24.0)
Status: Resolved (was: Patch Available)
No longer a problem in MR2. I guess we can close this.
The maxRunningTasks property of the LimitTasksPerJob scheduler is ambiguous
in its description, and must be updated
---
Key: MAPREDUCE-2640
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2640
Project: Hadoop Map/Reduce
Issue Type: Task
Components: documentation
Affects Versions: 0.20.2
Reporter: Harsh J
Assignee: Harsh J
Labels: doc
Attachments: 0.22-MAPREDUCE-2640.r1.diff,
0.23-MAPREDUCE-2640.r1.diff, MAPREDUCE-2640.r1.diff
The property's entry in mapred-default.xml is like so:
{code}
property
namemapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob/name
value/value
descriptionThe maximum number of running tasks for a job before
it gets preempted. No limits if undefined.
/description
/property
{code}
There is no mention that this is a property exclusive to the LimitTasksPerJob
scheduler alone. The doc ought to be updated to note that unless there's a
plan of reusing such a property (I do not see fair or capacity schedulers
utilizing this, and they use their own configs).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182469#comment-13182469
]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-3354:
---
Jonathan,
minor comments from the patch.
{noformat}usage=Usage: mapreduce-daemon.sh [--config conf-dir] [--hosts
hostlistfile] (start|stop) yarn-command {noformat}
I *think*, instead of mapreduce-daemon.sh, it should be mapred-daemon.sh in
mapred-daemon.sh. Moreover after inclusion of this patch, it will be nomore a
yarn-command, it will be a mapred-command.
{noformat}$ $YARN_HOME/bin/mapred stop historyserver --config $HADOOP_CONF_DIR
{noformat}
I *think*, we should replace HADOOP_CONF_DIR by YARN_CONF_DIR and mapred by
mapred-daemon.sh in ClusterSetup.apt.vm.
JobHistoryServer should be started by bin/mapred and not by bin/yarn
Key: MAPREDUCE-3354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Jonathan Eagles
Priority: Blocker
Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch,
MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch,
MAPREDUCE-3354.patch
JobHistoryServer belongs to mapreduce land.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3619) Change streaming code to use new mapreduce api.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liyin Liang resolved MAPREDUCE-3619.
Resolution: Duplicate
Change streaming code to use new mapreduce api.
---
Key: MAPREDUCE-3619
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3619
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1
Reporter: Liyin Liang
If we run a streaming job with following python script as mapper or reducer,
the job will throws NullPointerException.
{code:}
#!/usr/bin/python
import sys,os
class MyTask:
def __init__(self, file=sys.stdin):
self.file = file
print sys.stderr, reporter:counter:spam,disp_flag_record,0
print sys.stderr, reporter:counter:spam,spam_record,0
def process(self):
while True:
line = self.file.readline()
if not line:
break;
print line
if __name__ == __main__:
task = MyTask()
task.process()
{code}
Here is the NPE related log:
2011-12-22 14:14:06,310 WARN org.apache.hadoop.streaming.PipeMapRed:
java.lang.NullPointerException
at
org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.incrCounter(PipeMapRed.java:502)
at
org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:444)
This is because the above script's print sys.stderr will invoke
reporter.incrCounter() during PipeMapper|PipeReducer.configure(). While we
can not get reporter in configure() function.
To fix this problem, we should change streaming code to use new-api. Then we
can call context.getCounter() in Mapper|Reducer.setup() function.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182544#comment-13182544 ] Jason Lowe commented on MAPREDUCE-3360: --- +1, looks good to me. Provide information about lost nodes in the UI. --- Key: MAPREDUCE-3360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Environment: NA Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Priority: Critical Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, MAPREDUCE-3360-5.patch, MAPREDUCE-3360.patch, lostNodes.png Currently there is no information provided about *lost nodes*. Provide information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3360: -- Target Version/s: 0.23.1, 0.24.0 (was: 0.23.1) Status: Patch Available (was: Open) Provide information about lost nodes in the UI. --- Key: MAPREDUCE-3360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Environment: NA Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Priority: Critical Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, MAPREDUCE-3360-5.patch, MAPREDUCE-3360.patch, lostNodes.png Currently there is no information provided about *lost nodes*. Provide information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182549#comment-13182549 ] Thomas Graves commented on MAPREDUCE-3299: -- Hey Jon, It looks like there is a pre-existing bug (that I introduced) in the HS attempts code that you picked up when you copied it over. The nodeId has the wrong port. It has the http port instead of the normal node manager port (should be using getNodeManagerPort()). Would you mind fixing both of those (app master and history server) here? Otherwise we can file a separate jira for the HS stuff. Everything else looks good. Thanks. Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182573#comment-13182573 ] Hadoop QA commented on MAPREDUCE-3360: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509882/MAPREDUCE-3360-5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1568//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1568//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1568//console This message is automatically generated. Provide information about lost nodes in the UI. --- Key: MAPREDUCE-3360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Environment: NA Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Priority: Critical Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, MAPREDUCE-3360-5.patch, MAPREDUCE-3360.patch, lostNodes.png Currently there is no information provided about *lost nodes*. Provide information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[
https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182592#comment-13182592
]
Robert Joseph Evans commented on MAPREDUCE-3511:
I have a few comments about the patch. Overall it looks very good.
If the old counter API is going to be the long term fix then perhaps we should
not mark is as deprecated any more. If not then can we file a new JIRA to
update the new Counters to be as space efficient as the old ones.
This is a minor performance improvement, but in many of the HistoryEvents there
is a datum, that has had most of its fields replicated inside the event itself.
I guess this is because they are in slightly different formats now. But we
still create an instance of datum in the constructor, and always populate its
fields inside getDatum. These event objects are imutable, outside of the
setDatum method that is only to be used when de-serializing the event.
getDatum, however, tends to be called repeatedly to pull out individual fields
from the datum. I would prefer to see the datum start out as null, and only
have its fields set if it is not null, inside getDatum.
Why was TokenCache.java modified at all? It does not seem to be related to this
JIRA.
You added a TODO in CompletedTask.java and CompletedJob.java {code}// TODO:
Make sure.{code} Did you make sure yet? if so please delete the TODO.
Also good catch on TestHsWebServicesTasks.java, TestAMWebServicesAttempts.java,
TestHsWebServicesTasks.java and the others. I also don't think we need any
more tests because, all we are doing is reducing memory usage, which is very
hard to write a unit test for.
Inside JobHistoryEventHandler.java you added in {code}// TODO: Only
job-counters is enough? How about the myriad clones in this code-path.{code} is
this TODO still needed?
Like I said before, overall the patch looks really good and I only have some
minor comments, thanks for doing this.
Counters occupy a good part of AM heap
--
Key: MAPREDUCE-3511
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.23.1
Attachments: MAPREDUCE-3511-20120107.1.txt
Per task counters seem to be occupying a good part of an AMs heap. Looks like
more than 50% of what's used by a TaskAttemptImpl object.
This could be optimized by interning strings or possibly using mrv1 counters
which are optimized. Currently counters are converted from mrv1 to mrv2
format for in memory storage. The conversion could be delayed till it's
actually required for RPC transfers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3299: --- Status: Open (was: Patch Available) Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3299: --- Attachment: MAPREDUCE-3299.patch Thanks, Tom. Fixed nodeid for AM and Hs with latest patch. Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3299: --- Status: Patch Available (was: Open) Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3643) Reducer Job always goes to pending state.
Reducer Job always goes to pending state. - Key: MAPREDUCE-3643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun Prakash Priority: Minor I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker. In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer. I have ran this job several times there was no issues but i am facing the problem right now. Any suggestions! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to pending state.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182608#comment-13182608 ] Robert Joseph Evans commented on MAPREDUCE-3643: What version is this filed against? Can you update the affects version field? Reducer Job always goes to pending state. - Key: MAPREDUCE-3643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun Prakash Priority: Minor I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker. In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer. I have ran this job several times there was no issues but i am facing the problem right now. Any suggestions! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to pending state.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182613#comment-13182613 ] Arun Prakash commented on MAPREDUCE-3643: - I am using the hadoop version 0.20,i couldn't find the exact version in the field. Reducer Job always goes to pending state. - Key: MAPREDUCE-3643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun Prakash Priority: Minor I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker. In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer. I have ran this job several times there was no issues but i am facing the problem right now. Any suggestions! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Attachment: MAPREDUCE-3354.patch JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Status: Open (was: Patch Available) JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Status: Patch Available (was: Open) JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to pending state.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182616#comment-13182616 ] Robert Joseph Evans commented on MAPREDUCE-3643: Is there any more information you can give on this? How reproducible is it? It sounds like it is kind of sporadic. So the reducer is the identity reducer, what about the mapper. Do you any logs that you can upload. The JT log for the time of the error would be very helpful. Reducer Job always goes to pending state. - Key: MAPREDUCE-3643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun Prakash Priority: Minor I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker. In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer. I have ran this job several times there was no issues but i am facing the problem right now. Any suggestions! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182615#comment-13182615 ] Jonathan Eagles commented on MAPREDUCE-3354: Thanks for the comments. I have fixed the issues you have addressed. I have left historyserver being started by mapred since i think that better reflects the recommended usage. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182618#comment-13182618 ] Hadoop QA commented on MAPREDUCE-3354: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509929/MAPREDUCE-3354.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1570//console This message is automatically generated. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182627#comment-13182627 ] Jonathan Eagles commented on MAPREDUCE-3354: Patch modifies hadoop-assemblies so patch apply failure is expected. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3643) Reducer Job always goes to pending state.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Prakash updated MAPREDUCE-3643: Attachment: snap.JPG Reducer Job always goes to pending state. - Key: MAPREDUCE-3643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun Prakash Priority: Minor Attachments: snap.JPG I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker. In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer. I have ran this job several times there was no issues but i am facing the problem right now. Any suggestions! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to pending state.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182631#comment-13182631 ] Arun Prakash commented on MAPREDUCE-3643: - I have attached the snapshot for the job tracker and below i mentioned the console output. 12/01/09 11:36:29 INFO streaming.StreamJob: map 0% reduce 0% 12/01/09 11:36:34 INFO streaming.StreamJob: map 100% reduce 0% Streaming command for step4 hadoop fs -rmr $OUTPUT hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-*.jar \ -D mapred.job.name=$NAME \ -D mapred.reduce.tasks=1 \ -D mapred.task.timeout=$TIMEOUT \ -input $INPUT1 \ -input $INPUT2 \ -mapper /bin/awk 'gsub(/\t/, \,\)' \ -reducer /bin/cat \ -output $OUTPUT i have used the /bin/cat instead of identity reducer and mapper Reducer Job always goes to pending state. - Key: MAPREDUCE-3643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun Prakash Priority: Minor Attachments: snap.JPG I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker. In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer. I have ran this job several times there was no issues but i am facing the problem right now. Any suggestions! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to pending state.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182633#comment-13182633
]
Robert Joseph Evans commented on MAPREDUCE-3643:
This is unrelated, but for your information having {code}-reducer NONE{code} or
not including a -reducer command is going to be faster then using
{code}-reducer /bin/cat{code} as the streaming code will not bother to stream
the data through cat which will reduce the IO and processing needed.
Reducer Job always goes to pending state.
-
Key: MAPREDUCE-3643
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Prakash
Priority: Minor
Attachments: snap.JPG
I am using hadoop streaming command for map-reduce.In my job there are 4
streaming commands,i have no issues with first 3 steps but step4 job once
mapper got 100% completed directly going to pending state,even there was no
error in the job tracker.
In the step4,i am merging the output(i.e step2 and step 3 output) as single
output using identity reducer.
I have ran this job several times there was no issues but i am facing the
problem right now.
Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182641#comment-13182641 ] Hadoop QA commented on MAPREDUCE-3299: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509926/MAPREDUCE-3299.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1569//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1569//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1569//console This message is automatically generated. Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182656#comment-13182656 ] Thomas Graves commented on MAPREDUCE-3360: -- A couple of comments: RMWebServices.java - minor nit: I would prefer the web services to output empty string rather then N/A for the nodeHttpAddress. It seems more programmatic and I tried to make that change consistent throughout. - I think you can use the nid.getHost() instead of doing the split on : to get the hostName. MetricsOverviewTable.java - I think we need to change ClusterMetricsInfo.java to match if this field is Active instead of Total. It has a field totalNodes which appears to really be active nodes. So I think we should add a new activeNodes field and then make totalNodes the true total nodes of the cluster - active + rebooted + lost + decommissioned. ClusterMetrics.java - do we want to change the heading output to match active nodes? @Metric(# of NMs) MutableGaugeInt numNMs; Provide information about lost nodes in the UI. --- Key: MAPREDUCE-3360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Environment: NA Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Priority: Critical Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, MAPREDUCE-3360-5.patch, MAPREDUCE-3360.patch, lostNodes.png Currently there is no information provided about *lost nodes*. Provide information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182661#comment-13182661 ] Jonathan Eagles commented on MAPREDUCE-3299: Findbugs ContainerRemoteLaunchEvent defines equals but not hashCode is pre-existing and not related to this patch Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182662#comment-13182662 ] Thomas Graves commented on MAPREDUCE-3299: -- +1 lgtm Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182691#comment-13182691
]
Alejandro Abdelnur commented on MAPREDUCE-3427:
---
Mahadev,
the tests you are see failing, those were tweaked and passing with HADOOP-7590.
Would you check if any of those changes have been mistakenly reverted?
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Priority: Blocker
Fix For: 0.23.1, 0.24.0
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3511: --- Status: Open (was: Patch Available) Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs to be overridable
[
https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182747#comment-13182747
]
Bruno Mahé commented on MAPREDUCE-3505:
---
+1 to the patch.
It solves a big issue I have had to hack around. This patch makes it way better
than the current situation and we can always improve it later on.
yarn APPLICATION_CLASSPATH needs to be overridable
--
Key: MAPREDUCE-3505
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Bruno Mahé
Assignee: Ahmed Radwan
Labels: bigtop
Attachments: MAPREDUCE-3505.patch
Right now MRApps sets the classpath to just being mrapp-generated-classpath,
its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing
and may change or disappear at any time
The list of hardcoded directories is defined in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
at line 92.
For convenience, here is its current content:
{noformat}
/**
* Classpath for typical applications.
*/
public static final String[] APPLICATION_CLASSPATH =
new String[] {
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$YARN_HOME/modules/*,
$YARN_HOME/lib/*
};
{noformat}
Not all deployment scenarii fit in this layout and therefore we need a
standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[
https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3511:
---
Attachment: MAPREDUCE-3511-20120109.txt
Thanks for looking into it, Robert!
bq. If the old counter API is going to be the long term fix then perhaps we
should not mark is as deprecated any more.
The counters that I am now using are from {{mapreduce.Counters}} which aren't
deprecated. Irrespective of this, it makes sense to undeprecate other old v1
stuff (mapped.*), I'll propose and merge MAPREDUCE-1735 into trunk/23 also.
bq. [...] a minor performance improvement [...] I would prefer to see the datum
start out as null, and only have its fields set if it is not null, inside
getDatum.
Not sure if {{getDatum()}} will be called multiple times as each event will be
logged only once. Makes sense to implement your proposal anyways just to be
sure. Done.
bq. Why was TokenCache.java modified at all? It does not seem to be related to
this JIRA.
It isn't. But without that, due to HADOOP-7963/MAPREDUCE-3639 the patch
couldn't be tested on cluster. I'll revert those changes from the patch.
bq. You added a TODO in CompletedTask.java and CompletedJob.java
Done, added that to remind myself to avoid any clones :) Removing the clones
now.
bq. good catch on TestHsWebServicesTasks.java, TestAMWebServicesAttempts.java,
TestHsWebServicesTasks.java
Am surprised Jenkins didn't catch these.
bq. I also don't think we need any more tests because, all we are doing is
reducing memory usage, which is very hard to write a unit test for.
Yes, +1 :)
bq. Inside JobHistoryEventHandler.java you added in // TODO: Only job-counters
is enough? How about the myriad clones in this code-path. is this TODO still
needed?
Avoided that extra clone for a finished Job.
Counters occupy a good part of AM heap
--
Key: MAPREDUCE-3511
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.23.1
Attachments: MAPREDUCE-3511-20120107.1.txt,
MAPREDUCE-3511-20120109.txt
Per task counters seem to be occupying a good part of an AMs heap. Looks like
more than 50% of what's used by a TaskAttemptImpl object.
This could be optimized by interning strings or possibly using mrv1 counters
which are optimized. Currently counters are converted from mrv1 to mrv2
format for in memory storage. The conversion could be delayed till it's
actually required for RPC transfers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs to be overridable
[
https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182753#comment-13182753
]
Roman Shaposhnik commented on MAPREDUCE-3505:
-
+1 on the current patch. Can we, please, have it committed? I understand the
interest in a more general solution, but committing this would be an
incremental positive step.
yarn APPLICATION_CLASSPATH needs to be overridable
--
Key: MAPREDUCE-3505
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Bruno Mahé
Assignee: Ahmed Radwan
Labels: bigtop
Attachments: MAPREDUCE-3505.patch
Right now MRApps sets the classpath to just being mrapp-generated-classpath,
its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing
and may change or disappear at any time
The list of hardcoded directories is defined in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
at line 92.
For convenience, here is its current content:
{noformat}
/**
* Classpath for typical applications.
*/
public static final String[] APPLICATION_CLASSPATH =
new String[] {
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$YARN_HOME/modules/*,
$YARN_HOME/lib/*
};
{noformat}
Not all deployment scenarii fit in this layout and therefore we need a
standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3511: --- Status: Patch Available (was: Open) Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182755#comment-13182755 ] Robert Joseph Evans commented on MAPREDUCE-3511: LGTM +1 (Non-binding) Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3644) Snapshot builds have confusing jar file names in share/hadoop/mapreduce in tarball
Snapshot builds have confusing jar file names in share/hadoop/mapreduce in
tarball
--
Key: MAPREDUCE-3644
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3644
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, mrv2
Affects Versions: 0.23.0
Reporter: Andrew Bayer
If you build a Hadoop tarball with a non-release version, the moduleSet used in
hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
results in jar files going into share/hadoop/mapreduce with unique snapshot
versions - i.e., the timestamp they were built. This isn't an issue in release
builds. It can be fixed by adding
outputFileNameMapping${module.artifactId}-${project.version}${dashClassifier?}.${module.extension}/outputFileNameMapping
to the binaries tag of the moduleSet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3641: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-3561 CapacityScheduler should be more conservative assigning off-switch requests --- Key: MAPREDUCE-3641 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, scheduler Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3641.patch In hadoop-1, the CS is very conservative handing out off-switch assignments, we need to do the same in YARN. We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3614) finalState UNDEFINED if AM is killed by hand
[
https://issues.apache.org/jira/browse/MAPREDUCE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182760#comment-13182760
]
Ravi Prakash commented on MAPREDUCE-3614:
-
The client would fail with this exception
{noformat}
RemoteTrace:
at Local Trace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1326140392720_0001
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:151)
at $Proxy8.getTaskAttemptCompletionEvents(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBClientImpl.java:172)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:328)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:372)
at
org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:438)
at org.apache.hadoop.mapreduce.Job$5.run(Job.java:656)
at org.apache.hadoop.mapreduce.Job$5.run(Job.java:653)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:653)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1295)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1235)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
{noformat}
finalState UNDEFINED if AM is killed by hand
Key: MAPREDUCE-3614
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3614
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Courtesy [~dcapwell]
{quote}
If the AM is running and you kill the process (sudo kill #pid), the State in
Yarn would be FINISHED and FinalStatus is UNDEFINED. The Tracking UI would
say History and point to the proxy url (which will redirect to the history
server).
The state should be more descriptive that the job failed and the tracker url
shouldn't point to the history server.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3312) Make MR AM not send a stopContainer w/o corresponding start container
[ https://issues.apache.org/jira/browse/MAPREDUCE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182764#comment-13182764 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3312: Looking at the patch now... Make MR AM not send a stopContainer w/o corresponding start container - Key: MAPREDUCE-3312 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3312 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.23.1, 0.24.0 Attachments: MR-3312.txt, MR-3312.txt, MR-3312.txt This is a follow on to MAPREDUCE-3274. It is possible, although rare, for the MR AM to send a stop container before it sends a start container. This needs to stop that from happening. If a stop is found first it should prevent the start from being sent. It tries to do this, but only if the stop is currently pending. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3614) finalState UNDEFINED if AM is killed by hand
[
https://issues.apache.org/jira/browse/MAPREDUCE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-3614:
Attachment: MAPREDUCE-3614.branch-0.23.patch
I identified the problem to RMCommunicator.java:unregister() . The call
{noformat}
scheduler.finishApplicationMaster(request);
{noformat}
is what is throwing the Exception. I'm attaching a patch which prevents the
exception from being thrown and the output on the UI and everywhere else seems
to be what is desired, but the approach to me seems to be dicey. I'm
essentially not making the call when finishState == JobState.FAILED. I doubt
that is the right approach. I'll dig deeper into code to see what the right
approach would be
finalState UNDEFINED if AM is killed by hand
Key: MAPREDUCE-3614
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3614
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Attachments: MAPREDUCE-3614.branch-0.23.patch
Courtesy [~dcapwell]
{quote}
If the AM is running and you kill the process (sudo kill #pid), the State in
Yarn would be FINISHED and FinalStatus is UNDEFINED. The Tracking UI would
say History and point to the proxy url (which will redirect to the history
server).
The state should be more descriptive that the job failed and the tracker url
shouldn't point to the history server.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182773#comment-13182773 ] Siddharth Seth commented on MAPREDUCE-3511: --- Also, for jobs which use custom counters (non-optimized) - it would be good to document that the AM heap may need to be adjusted. Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182780#comment-13182780
]
Siddharth Seth commented on MAPREDUCE-3596:
---
From another set of logs, sequence of events.
1. AM calls a startContainer
2. NM receives this, starts processing but takes about 1minutes20 seconds to
finish processing it.
3. Meanwhile, the AM times out the call after 1 minute - and sends a release
container to the RM
4. RM ends up removing references to the container
5. The NM sends a containerStarted event to the RM - which ends up causing the
NPE.
From a quick look at the code - if the AM release event had gone out after the
NM containerStarted, things would've been handled.
Sort benchmark got hang after completion of 99% map phase
-
Key: MAPREDUCE-3596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Attachments: logs.tar.bz2, logs.tar.bz2
Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after
completion 99% map phase. There are some map tasks failed. Also it's not
scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version:revision 1212681 by someone source checksum
on Fri Dec 9 16:52:07 PST 2011
Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
2011
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182791#comment-13182791 ] Hadoop QA commented on MAPREDUCE-3511: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509947/MAPREDUCE-3511-20120109.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1571//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1571//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1571//console This message is automatically generated. Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3625: -- Attachment: CapacityScheduler.png Based on discussions with Arun, we decided to rework the UI a bit. Attached is a sample screenshot of new capacity scheduler UI. CapacityScheduler web-ui display of queue's used capacity is broken --- Key: MAPREDUCE-3625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Jason Lowe Priority: Critical Attachments: CapacityScheduler.png The display of the queue's used capacity at runtime is broken because it display's 'used' relative to the queue's capacity and not the parent's capacity as shown in the above attachment. The display should be relative to parent's capacity and not leaf queues as everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-3625 started by Jason Lowe. CapacityScheduler web-ui display of queue's used capacity is broken --- Key: MAPREDUCE-3625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Jason Lowe Priority: Critical Attachments: CapacityScheduler.png The display of the queue's used capacity at runtime is broken because it display's 'used' relative to the queue's capacity and not the parent's capacity as shown in the above attachment. The display should be relative to parent's capacity and not leaf queues as everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182793#comment-13182793 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3511: bq. Also, for jobs which use custom counters (non-optimized) - it would be good to document that the AM heap may need to be adjusted. That's a good to have, I'll open a separate ticket to add a best-practices doc. I am pushing this in. Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3511: --- Resolution: Fixed Release Note: Removed a multitude of cloned/duplicate counters in the AM thereby reducing the AM heap size and preventing full GCs. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Just committed this to trunk and branch-0.23. Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3625: -- Attachment: MAPREDUCE-3625.patch Patch reworks capacity scheduler UI to fix the reported problem and make the UI a bit easier to read in general. Green now represents used capacity instead of spare capacity and always grows from left to right. The color of the bar changes to orange if the used capacity goes over 100%, and it doesn't streak off the right end of the page as it did previously. Also added a legend so it's clear what the colors indicate and added a used capacity column for quick reference. CapacityScheduler web-ui display of queue's used capacity is broken --- Key: MAPREDUCE-3625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Jason Lowe Priority: Critical Attachments: CapacityScheduler.png, MAPREDUCE-3625.patch The display of the queue's used capacity at runtime is broken because it display's 'used' relative to the queue's capacity and not the parent's capacity as shown in the above attachment. The display should be relative to parent's capacity and not leaf queues as everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3625: -- Target Version/s: 0.23.1, 0.24.0 (was: 0.23.1) Status: Patch Available (was: In Progress) CapacityScheduler web-ui display of queue's used capacity is broken --- Key: MAPREDUCE-3625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Jason Lowe Priority: Critical Attachments: CapacityScheduler.png, MAPREDUCE-3625.patch The display of the queue's used capacity at runtime is broken because it display's 'used' relative to the queue's capacity and not the parent's capacity as shown in the above attachment. The display should be relative to parent's capacity and not leaf queues as everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3639: -- Attachment: MR3639.txt TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Priority: Critical Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3639: -- Assignee: Siddharth Seth Status: Patch Available (was: Open) TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3639: -- Priority: Blocker (was: Critical) MR + viewFs will fail without this. Marking as blocker. TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3528: -- Attachment: MR3528.txt Trivial patch - introducing a new config param for the timeout check. Not including a unit test. The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3528: -- Assignee: Siddharth Seth Status: Patch Available (was: Open) The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3645) TestJobHistory fails
TestJobHistory fails Key: MAPREDUCE-3645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Thomas Graves Priority: Blocker Fix For: 0.23.0 TestJobHistory fails. org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile It looks like this was introduced by MAPREDUCE-3349 and the issue is that the test expects the hostname to be in the format rackname/hostname, but with 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3645) TestJobHistory fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182825#comment-13182825 ] Thomas Graves commented on MAPREDUCE-3645: -- One of the errors with added debug into to print hostname and expected hostname Host name : host1.foo.com of task attempt attempt_20120109210634113_0001_m_02_0 obtained from history file did not match the expected value /default-rack/host1.foo.comjunit.framework.AssertionFailedError: Host name : host1.foo.com of task attempt attempt_20120109210634113_0001_m_02_0 obtained from history file did not match the expected value /default-rack/host1.foo.comat org.apache.hadoop.mapred.TestJobHistory.validateTaskAttemptLevelKeyValues(TestJobHistory.java:527) at org.apache.hadoop.mapred.TestJobHistory.validateJobHistoryFileContent(TestJobHistory.java:595) at org.apache.hadoop.mapred.TestJobHistory.runDoneFolderTest(TestJobHistory.java:751) at org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS(TestJobHistory.java:621) TestJobHistory fails Key: MAPREDUCE-3645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Thomas Graves Priority: Blocker Fix For: 0.23.0 TestJobHistory fails. org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile It looks like this was introduced by MAPREDUCE-3349 and the issue is that the test expects the hostname to be in the format rackname/hostname, but with 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3511-20120107.1.txt, MAPREDUCE-3511-20120109.txt Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182840#comment-13182840 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3639: Trivial patch. +1. Waiting for Jenkins. TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182841#comment-13182841 ] Hadoop QA commented on MAPREDUCE-3528: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509964/MR3528.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1574//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1574//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1574//console This message is automatically generated. The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182844#comment-13182844 ] Hadoop QA commented on MAPREDUCE-3639: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509961/MR3639.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1572//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1572//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1572//console This message is automatically generated. TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182846#comment-13182846 ] Hadoop QA commented on MAPREDUCE-3625: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509960/MAPREDUCE-3625.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1573//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1573//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1573//console This message is automatically generated. CapacityScheduler web-ui display of queue's used capacity is broken --- Key: MAPREDUCE-3625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Jason Lowe Priority: Critical Attachments: CapacityScheduler.png, MAPREDUCE-3625.patch The display of the queue's used capacity at runtime is broken because it display's 'used' relative to the queue's capacity and not the parent's capacity as shown in the above attachment. The display should be relative to parent's capacity and not leaf queues as everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182853#comment-13182853 ] Thomas Graves commented on MAPREDUCE-3639: -- this fixes the following tests then? org.apache.hadoop.mapreduce.security.TestTokenCache.testGetTokensForHftpFS org.apache.hadoop.mapreduce.security.TestTokenCache.testGetTokensForViewFS TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182857#comment-13182857 ] Jason Lowe commented on MAPREDUCE-3625: --- Findbugs warnings are unrelated to this patch. CapacityScheduler web-ui display of queue's used capacity is broken --- Key: MAPREDUCE-3625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Jason Lowe Priority: Critical Attachments: CapacityScheduler.png, MAPREDUCE-3625.patch The display of the queue's used capacity at runtime is broken because it display's 'used' relative to the queue's capacity and not the parent's capacity as shown in the above attachment. The display should be relative to parent's capacity and not leaf queues as everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182859#comment-13182859 ] Daryn Sharp commented on MAPREDUCE-3639: Hang on with the commit. I don't think this patch is complete, need to cross check with 205. TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3616) Thread pool for launching containers in MR AM not expanding as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3616: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1. Committed to trunk and branch-0.23. Thanks Vinod. Thread pool for launching containers in MR AM not expanding as expected --- Key: MAPREDUCE-3616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3616 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, performance Affects Versions: 0.23.1 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3616-20120105.txt Found this while running some benchmarks on 350 nodes. The thread pool stays at 60 for a long time and only expands to 350 towards the fag end of the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3646) Remove redundant URL info from mapred job output
Remove redundant URL info from mapred job output
--
Key: MAPREDUCE-3646
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3646
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.1
Reporter: Ramya Sunil
Priority: Minor
Fix For: 0.23.1
The URL information to track the job is printed for all the mapred jobmrv2
commands. This information is redundant and has to be removed.
E.g:
{noformat}
-bash-3.2$ mapred job -list
Total jobs:3
JobId State StartTime UserNameQueue PriorityMaps
Reduces UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info
12/01/09 22:20:15 INFO mapred.ClientServiceDelegate: The url to track the job:
RM host:8088/proxy/application ID 1/
job ID 1 RUNNING 1326147596446 ramya default NORMAL 10 10 21
0 22528M 0M 22528M RM host:8088/proxy/application ID 1/
12/01/09 22:20:15 INFO mapred.ClientServiceDelegate: The url to track the job:
RM host:8088/proxy/application ID 2/
job ID 2 RUNNING 1326147603726 ramya default NORMAL 10 10 11
0 12288M 0M 12288M RM host:8088/proxy/application ID 2/
12/01/09 22:20:16 INFO mapred.ClientServiceDelegate: The url to track the job:
RM host:8088/proxy/application ID 3/
job ID 3 RUNNING 1326147520126 ramya default NORMAL 10 10 21
0 22528M 0M 22528M RM host:8088/proxy/application ID 3/
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3616) Thread pool for launching containers in MR AM not expanding as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182868#comment-13182868 ] Hudson commented on MAPREDUCE-3616: --- Integrated in Hadoop-Common-trunk-Commit #1515 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1515/]) MAPREDUCE-3616. Thread pool for launching containers in MR AM not expanding as expected. (Contributed by Vinod Kumar Vavilapalli) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229394 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java Thread pool for launching containers in MR AM not expanding as expected --- Key: MAPREDUCE-3616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3616 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, performance Affects Versions: 0.23.1 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3616-20120105.txt Found this while running some benchmarks on 350 nodes. The thread pool stays at 60 for a long time and only expands to 350 towards the fag end of the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3647) Pipes job fails with Illegal text protocol
Pipes job fails with Illegal text protocol Key: MAPREDUCE-3647 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3647 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 Pipes job fail with Hadoop Pipes Exception: Illegal text protocol command -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3645) TestJobHistory fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned MAPREDUCE-3645: Assignee: Thomas Graves TestJobHistory fails Key: MAPREDUCE-3645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3645.patch TestJobHistory fails. org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile It looks like this was introduced by MAPREDUCE-3349 and the issue is that the test expects the hostname to be in the format rackname/hostname, but with 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[
https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182871#comment-13182871
]
Daryn Sharp commented on MAPREDUCE-3639:
{{TokenCache#obtainTokensForNamenodesInternal}} needs the same null check on
{{getCanonicalServiceName}} as 205 does.
TokenCache likely broken for FileSystems which don't issue delegation tokens
Key: MAPREDUCE-3639
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Blocker
Attachments: MR3639.txt
Ref HADOOP-7963.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3616) Thread pool for launching containers in MR AM not expanding as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182873#comment-13182873 ] Hudson commented on MAPREDUCE-3616: --- Integrated in Hadoop-Hdfs-trunk-Commit #1588 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1588/]) MAPREDUCE-3616. Thread pool for launching containers in MR AM not expanding as expected. (Contributed by Vinod Kumar Vavilapalli) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229394 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java Thread pool for launching containers in MR AM not expanding as expected --- Key: MAPREDUCE-3616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3616 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, performance Affects Versions: 0.23.1 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3616-20120105.txt Found this while running some benchmarks on 350 nodes. The thread pool stays at 60 for a long time and only expands to 350 towards the fag end of the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3645) TestJobHistory fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3645: - Attachment: MAPREDUCE-3645.patch TestJobHistory fails Key: MAPREDUCE-3645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3645.patch TestJobHistory fails. org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile It looks like this was introduced by MAPREDUCE-3349 and the issue is that the test expects the hostname to be in the format rackname/hostname, but with 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182874#comment-13182874 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3528: Neither the findBugs warning, nor the test-failure is related to this patch. Trivial patch replacing the interval configuration. So, no tests. +1. Pushing this in. The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3616) Thread pool for launching containers in MR AM not expanding as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182878#comment-13182878 ] Hudson commented on MAPREDUCE-3616: --- Integrated in Hadoop-Hdfs-0.23-Commit #345 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/345/]) merge MAPREDUCE-3616 from trunk sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229395 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java Thread pool for launching containers in MR AM not expanding as expected --- Key: MAPREDUCE-3616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3616 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, performance Affects Versions: 0.23.1 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3616-20120105.txt Found this while running some benchmarks on 350 nodes. The thread pool stays at 60 for a long time and only expands to 350 towards the fag end of the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3616) Thread pool for launching containers in MR AM not expanding as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182877#comment-13182877 ] Hudson commented on MAPREDUCE-3616: --- Integrated in Hadoop-Common-0.23-Commit #355 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/355/]) merge MAPREDUCE-3616 from trunk sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229395 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java Thread pool for launching containers in MR AM not expanding as expected --- Key: MAPREDUCE-3616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3616 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, performance Affects Versions: 0.23.1 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3616-20120105.txt Found this while running some benchmarks on 350 nodes. The thread pool stays at 60 for a long time and only expands to 350 towards the fag end of the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3528: --- Resolution: Fixed Fix Version/s: 0.23.1 Release Note: Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to trunk and branch 0.23. Thanks Sid! The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.23.1 Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182888#comment-13182888 ] Hudson commented on MAPREDUCE-3528: --- Integrated in Hadoop-Hdfs-0.23-Commit #346 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/346/]) MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. Contributed by Siddharth Seth. svn merge --ignore-ancestry -c 1229403 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229405 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.23.1 Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3646) Remove redundant URL info from mapred job output
[
https://issues.apache.org/jira/browse/MAPREDUCE-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182887#comment-13182887
]
Jonathan Eagles commented on MAPREDUCE-3646:
This information is printed for both map tasks and reduces tasks per job.
path is
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
(displayJobList-getTaskReports)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
(getTaskReports-invoke-getProxy)
getProxy always print tracking url to screen.
Remove redundant URL info from mapred job output
--
Key: MAPREDUCE-3646
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3646
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.1
Reporter: Ramya Sunil
Priority: Minor
Fix For: 0.23.1
The URL information to track the job is printed for all the mapred jobmrv2
commands. This information is redundant and has to be removed.
E.g:
{noformat}
-bash-3.2$ mapred job -list
Total jobs:3
JobId State StartTime UserNameQueue PriorityMaps
Reduces UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM
info
12/01/09 22:20:15 INFO mapred.ClientServiceDelegate: The url to track the
job: RM host:8088/proxy/application ID 1/
job ID 1 RUNNING 1326147596446 ramya default NORMAL 10 10 21
0 22528M 0M 22528M RM host:8088/proxy/application ID 1/
12/01/09 22:20:15 INFO mapred.ClientServiceDelegate: The url to track the
job: RM host:8088/proxy/application ID 2/
job ID 2 RUNNING 1326147603726 ramya default NORMAL 10 10 11
0 12288M 0M 12288M RM host:8088/proxy/application ID 2/
12/01/09 22:20:16 INFO mapred.ClientServiceDelegate: The url to track the
job: RM host:8088/proxy/application ID 3/
job ID 3 RUNNING 1326147520126 ramya default NORMAL 10 10 21
0 22528M 0M 22528M RM host:8088/proxy/application ID 3/
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182890#comment-13182890 ] Hudson commented on MAPREDUCE-3528: --- Integrated in Hadoop-Common-trunk-Commit #1516 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1516/]) MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. (Siddharth Seth via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229403 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.23.1 Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182892#comment-13182892 ] Siddharth Seth commented on MAPREDUCE-3639: --- @Daryn, That check should not be present. getCanonicalServiceName will return null for filesystems like viewfs. With that additional check - TokenCache will not get tokens for the underlying filesystems. @Thomas - didn't run the ant tests earlier. Looks like testGetTokensForViewFS is fixed, but HftpFS is not. Do you know when these tests started failing ? TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3616) Thread pool for launching containers in MR AM not expanding as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182897#comment-13182897 ] Hudson commented on MAPREDUCE-3616: --- Integrated in Hadoop-Mapreduce-0.23-Commit #366 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/366/]) merge MAPREDUCE-3616 from trunk sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229395 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java Thread pool for launching containers in MR AM not expanding as expected --- Key: MAPREDUCE-3616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3616 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, performance Affects Versions: 0.23.1 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3616-20120105.txt Found this while running some benchmarks on 350 nodes. The thread pool stays at 60 for a long time and only expands to 350 towards the fag end of the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182894#comment-13182894 ] Hudson commented on MAPREDUCE-3528: --- Integrated in Hadoop-Hdfs-trunk-Commit #1589 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1589/]) MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. (Siddharth Seth via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229403 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.23.1 Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182896#comment-13182896 ] Hudson commented on MAPREDUCE-3528: --- Integrated in Hadoop-Common-0.23-Commit #356 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/356/]) MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. Contributed by Siddharth Seth. svn merge --ignore-ancestry -c 1229403 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229405 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.23.1 Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3644) Snapshot builds have confusing jar file names in share/hadoop/mapreduce in tarball
[
https://issues.apache.org/jira/browse/MAPREDUCE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated MAPREDUCE-3644:
---
Labels: bigtop (was: )
Snapshot builds have confusing jar file names in share/hadoop/mapreduce in
tarball
--
Key: MAPREDUCE-3644
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3644
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, mrv2
Affects Versions: 0.23.0
Reporter: Andrew Bayer
Labels: bigtop
If you build a Hadoop tarball with a non-release version, the moduleSet used
in hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
results in jar files going into share/hadoop/mapreduce with unique snapshot
versions - i.e., the timestamp they were built. This isn't an issue in
release builds. It can be fixed by adding
outputFileNameMapping${module.artifactId}-${project.version}${dashClassifier?}.${module.extension}/outputFileNameMapping
to the binaries tag of the moduleSet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182903#comment-13182903 ] Thomas Graves commented on MAPREDUCE-3639: -- Looks like the Jan 5th or 6th. I saw this and was hoping it fixed them. If you think the other is different issue we can file separate jira. TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MR3639.txt Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182904#comment-13182904
]
Siddharth Seth commented on MAPREDUCE-3596:
---
For the NM taking a long time to process a startContainer call - would be
interesting to see if changing RPC thread priorities will make a difference, or
if RPC traffic can be prioritized over shuffle. Also, trying to reduce what the
startContainer call does in the NM.
Sort benchmark got hang after completion of 99% map phase
-
Key: MAPREDUCE-3596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Attachments: logs.tar.bz2, logs.tar.bz2
Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after
completion 99% map phase. There are some map tasks failed. Also it's not
scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version:revision 1212681 by someone source checksum
on Fri Dec 9 16:52:07 PST 2011
Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
2011
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
[
https://issues.apache.org/jira/browse/MAPREDUCE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182905#comment-13182905
]
Siddharth Seth commented on MAPREDUCE-3639:
---
From running ant -Dtestcase=TestTokenCache. I would lean towards fixing that
via another jira.
{{nofromat}}
Testcase: testTokenCache took 12.438 sec
Testcase: testLocalJobTokenCache took 13.397 sec
Testcase: testGetTokensForNamenodes took 0.002 sec
Testcase: testGetTokensForHftpFS took 0.16 sec
Caused an ERROR
java.net.UnknownHostException: host
java.lang.IllegalArgumentException: java.net.UnknownHostException: host
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:425)
at
org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:308)
at
org.apache.hadoop.mapreduce.security.TestTokenCache.testGetTokensForHftpFS(TestTokenCache.java:324)
Caused by: java.net.UnknownHostException: host
Testcase: testGetJTPrincipal took 0.011 sec
Testcase: testGetTokensForViewFS took 0.01 sec
{{nofromat}}
TokenCache likely broken for FileSystems which don't issue delegation tokens
Key: MAPREDUCE-3639
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Blocker
Attachments: MR3639.txt
Ref HADOOP-7963.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3647) Pipes job fails with Illegal text protocol
[ https://issues.apache.org/jira/browse/MAPREDUCE-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3647: - Component/s: mrv2 Priority: Critical (was: Major) Pipes job fails with Illegal text protocol Key: MAPREDUCE-3647 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3647 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, pipes Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Critical Fix For: 0.23.1 Pipes job fail with Hadoop Pipes Exception: Illegal text protocol command -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182915#comment-13182915 ] Hudson commented on MAPREDUCE-3528: --- Integrated in Hadoop-Mapreduce-0.23-Commit #367 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/367/]) MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. Contributed by Siddharth Seth. svn merge --ignore-ancestry -c 1229403 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229405 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.23.1 Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3645) TestJobHistory fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182919#comment-13182919 ] Hadoop QA commented on MAPREDUCE-3645: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509969/MAPREDUCE-3645.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1575//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1575//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1575//console This message is automatically generated. TestJobHistory fails Key: MAPREDUCE-3645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3645.patch TestJobHistory fails. org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile It looks like this was introduced by MAPREDUCE-3349 and the issue is that the test expects the hostname to be in the format rackname/hostname, but with 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3528) The task timeout check interval should be configurable independent of mapreduce.task.timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182921#comment-13182921 ] Hudson commented on MAPREDUCE-3528: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1535 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1535/]) MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. (Siddharth Seth via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229403 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java The task timeout check interval should be configurable independent of mapreduce.task.timeout Key: MAPREDUCE-3528 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3528 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.23.1 Attachments: MR3528.txt TaskHeartbeatHandler sleeps for 'mapreduce.task.timeout' - between each check. If a task/NM goes bad immediately after starting a task - the timeout is detected in ~2x the configured timeout interval. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3616) Thread pool for launching containers in MR AM not expanding as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182922#comment-13182922 ] Hudson commented on MAPREDUCE-3616: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1535 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1535/]) MAPREDUCE-3616. Thread pool for launching containers in MR AM not expanding as expected. (Contributed by Vinod Kumar Vavilapalli) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229394 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java Thread pool for launching containers in MR AM not expanding as expected --- Key: MAPREDUCE-3616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3616 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, performance Affects Versions: 0.23.1 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3616-20120105.txt Found this while running some benchmarks on 350 nodes. The thread pool stays at 60 for a long time and only expands to 350 towards the fag end of the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3402) AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-3402. Resolution: Fixed Fixed after MAPREDUCE-3511. AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly --- Key: MAPREDUCE-3402 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3402 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 The world was rosier before October 19-25, [~karams] says. The 100K 1 second sleep job used to take around 800mins or 13-14 mins. It now runs till 45 mins and still manages to complete only about 45K tasks. One/more of the flurry of commits for 0.23.0 deserve(s) the blame. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
