[jira] [Created] (MAPREDUCE-3012) Change org.apache.hadoop.mapred.lib.NLineInputFormat and org.apache.hadoop.mapred.MapFileOutputFormat to use new api for hadoop 0.20
Change org.apache.hadoop.mapred.lib.NLineInputFormat and org.apache.hadoop.mapred.MapFileOutputFormat to use new api for hadoop 0.20 Key: MAPREDUCE-3012 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3012 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 0.20.203.0 Environment: all Reporter: Arsen Zahray Fix For: 0.21.0 This bug has been fixed for hadoop 0.21 api, but it still is open for hadoop 0.20. As 0.21 is hardly used anywhere, and 0.20 is the main version on all the clusters, I feel that the issue has to be reopened. https://issues.apache.org/jira/browse/MAPREDUCE-375 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1788) o.a.h.mapreduce.Job shouldn't make a copy of the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105142#comment-13105142 ] Hadoop QA commented on MAPREDUCE-1788: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494570/MAPREDUCE-1788.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/747//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/747//console This message is automatically generated. > o.a.h.mapreduce.Job shouldn't make a copy of the JobConf > > > Key: MAPREDUCE-1788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.23.0 > > Attachments: MAPREDUCE-1788.patch > > > Having o.a.h.mapreduce.Job make a copy of the passed in JobConf has several > issues: any modifications done by various pieces such as InputSplit etc. are > not reflected back and causes issues for frameworks built on top. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3002) Delink History Context from AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105143#comment-13105143 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3002: bq. This was used in MR JobHistory to be able to share the UI - all known jobs end up belonging to a single dummy App. okay, that is the point I missed completely. bq. getJob ofc can continue to use the Job interface. I was concerned that you were going to throw the Job interface away. Alright, +1 for delinking the HistoryContext from AppContext. Thanks for the clear explanation, Sid! > Delink History Context from AppContext > -- > > Key: MAPREDUCE-3002 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3002 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver, mrv2 >Affects Versions: 0.24.0 >Reporter: Robert Joseph Evans > > Currently the JobHistory Server has a HistoryContext that pretends to be a > Map Reduce ApplicationMaster's AppContext so that UI pages can be shared > between the two. This is not ideal because the UIs have already diverged a > lot, and we have to translate the native History Server's data into > implementations of Job to provide the same interface. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2925) job -status is giving continuously info message for completed jobs on the console
[
https://issues.apache.org/jira/browse/MAPREDUCE-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105144#comment-13105144
]
Arun C Murthy commented on MAPREDUCE-2925:
--
Seems like not having any retries if JobHistory throws an exception is ok?
> job -status is giving continuously info message for completed jobs
> on the console
> --
>
> Key: MAPREDUCE-2925
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2925
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2925.patch
>
>
> This below message is coming continuously on the console.
> {code:xml}
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Failed to contact AM for
> job job_1314955256658_0009 Will retry..
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server null
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Failed to contact AM for
> job job_1314955256658_0009 Will retry..
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server null
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1788) o.a.h.mapreduce.Job shouldn't make a copy of the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1788: - Fix Version/s: 0.23.0 Status: Patch Available (was: Open) > o.a.h.mapreduce.Job shouldn't make a copy of the JobConf > > > Key: MAPREDUCE-1788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.23.0 > > Attachments: MAPREDUCE-1788.patch > > > Having o.a.h.mapreduce.Job make a copy of the passed in JobConf has several > issues: any modifications done by various pieces such as InputSplit etc. are > not reflected back and causes issues for frameworks built on top. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1788) o.a.h.mapreduce.Job shouldn't make a copy of the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1788: - Attachment: MAPREDUCE-1788.patch Straight-fwd fix. It also illustrates the pain it causes and the *ugly* hack to which we resorted to get around this... > o.a.h.mapreduce.Job shouldn't make a copy of the JobConf > > > Key: MAPREDUCE-1788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: MAPREDUCE-1788.patch > > > Having o.a.h.mapreduce.Job make a copy of the passed in JobConf has several > issues: any modifications done by various pieces such as InputSplit etc. are > not reflected back and causes issues for frameworks built on top. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105132#comment-13105132 ] Ravi Gummadi commented on MAPREDUCE-2957: - +1 to avoid calling TaskTracker.initialize() if there are no good local dirs. Even with this patch, i.e. though State.DENIED is returned by offerService() when DiskCheckerException is seen, initialize() is getting called once more and is getting DiskCheckerException from TT.initialize() -> DiskChecker.checkLocalDirs(). This behavior is similar to what was happening without this patch. This needs to be improved somehow and we need to avoid the call to TT.initialize() in this case. > The TT should not re-init if it has no good local dirs > -- > > Key: MAPREDUCE-2957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.20.205.0 > > Attachments: mapreduce-2957.patch > > > The TT will currently try to re-init itself on disk failure even if it has no > good local dirs. It should shutdown instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3011) TT should remove bad local dirs from conf to prevent constant disk checking
[ https://issues.apache.org/jira/browse/MAPREDUCE-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105131#comment-13105131 ] Todd Lipcon commented on MAPREDUCE-3011: What's the method to re-trigger a check, eg if the disk is onlined again? Restart the TT, I guess? > TT should remove bad local dirs from conf to prevent constant disk checking > --- > > Key: MAPREDUCE-3011 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3011 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Eli Collins > Fix For: 0.20.205.0 > > > Per HADOOP-7551 the TT does not remove bad mapred.local.dirs from the conf so > after a single disk failure *every* call to get a local path for reading or > writing results in a disk check of *all* configured local dirs. After > detecting that a local dir is bad we should remove it from the conf so that > we don't repeatedly perform this expensive operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2928) MR-2413 improvements
[ https://issues.apache.org/jira/browse/MAPREDUCE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-2928: --- Attachment: mapreduce-2928-2.patch > MR-2413 improvements > > > Key: MAPREDUCE-2928 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2928 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.20.205.0 > > Attachments: mapreduce-2928-1.patch, mapreduce-2928-2.patch > > > Tracks improvements to MR-2413. See [this > comment|https://issues.apache.org/jira/browse/MAPREDUCE-2413?focusedCommentId=13095073&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13095073]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2928) MR-2413 improvements
[ https://issues.apache.org/jira/browse/MAPREDUCE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105127#comment-13105127 ] Eli Collins commented on MAPREDUCE-2928: Patch attached. > MR-2413 improvements > > > Key: MAPREDUCE-2928 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2928 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.20.205.0 > > Attachments: mapreduce-2928-1.patch, mapreduce-2928-2.patch > > > Tracks improvements to MR-2413. See [this > comment|https://issues.apache.org/jira/browse/MAPREDUCE-2413?focusedCommentId=13095073&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13095073]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2928) MR-2413 improvements
[ https://issues.apache.org/jira/browse/MAPREDUCE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105125#comment-13105125 ] Eli Collins commented on MAPREDUCE-2928: Owen, Ravi - thank you for the feedback. Will re-introduce getGoodLocalDirsString. Wrt #1 are we worried about this case? It doesn't seem like something that would happen in practice as disk failures aren't typically recoverable, and per MAPREDUCE-3011 I'm thinking once a local path is identified as bad we should remove it from the config. In the DN we maintain a count of total failures, how about we do that here and use it instead of the # of valid dirs? This value always increases and is therefore not susceptible to the bug you described. Wrt #2 - good point, will fix this. > MR-2413 improvements > > > Key: MAPREDUCE-2928 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2928 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.20.205.0 > > Attachments: mapreduce-2928-1.patch > > > Tracks improvements to MR-2413. See [this > comment|https://issues.apache.org/jira/browse/MAPREDUCE-2413?focusedCommentId=13095073&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13095073]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2124) Add job counters for measuring time spent in three different phases in reducers
[
https://issues.apache.org/jira/browse/MAPREDUCE-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105126#comment-13105126
]
Scott Chen commented on MAPREDUCE-2124:
---
Thanks for the comment. I will rebase this soon.
> Add job counters for measuring time spent in three different phases in
> reducers
> ---
>
> Key: MAPREDUCE-2124
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2124
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Scott Chen
>Assignee: Scott Chen
>Priority: Minor
> Attachments: MAPREDUCE-2124-v2.txt, MAPREDUCE-2124.3.txt,
> MAPREDUCE-2124.txt, test-result.txt
>
>
> We currently have SLOTS_MILLIS_REDUCES which measures the total slot time of
> reducer.
> It will be useful if we have
> {code}
> SLOTS_MILLIS_REDUCES_COPY
> SLOTS_MILLIS_REDUCES_SORT
> SLOTS_MILLIS_REDUCES_REDUCE
> {code}
> which measures three different phases of a reducer.
> This will help us identify the bottleneck of the reducers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105123#comment-13105123 ] Hadoop QA commented on MAPREDUCE-2965: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494564/MR2965_v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/746//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/746//console This message is automatically generated. > Streamline hashCode(), equals(), compareTo() and toString() for all IDs > --- > > Key: MAPREDUCE-2965 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2965_v1.patch, MR2965_v2.patch, MR2965_v3.patch, > MR2965_v4.patch > > > MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls > for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they > don't need to be tied to the implementation. > We should do the same for all IDs. In fact some of these are missing for IDs > like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen resolved MAPREDUCE-2198. --- Resolution: Won't Fix > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198-v2.txt, MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105119#comment-13105119 ] Scott Chen commented on MAPREDUCE-2198: --- Arun: Thanks for the comments. You are right. I guess this is not an issue since we have MRv2. Closing this now. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198-v2.txt, MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2108) Allow TaskScheduler manage number slots on TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen resolved MAPREDUCE-2108. --- Resolution: Won't Fix > Allow TaskScheduler manage number slots on TaskTrackers > --- > > Key: MAPREDUCE-2108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2108 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/capacity-sched, contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2108-v2.txt, MAPREDUCE-2108.txt > > > Currently the map slots and reduce slots are managed by TaskTracker > configuration. > To change the task tracker slots, we need to restart the TaskTrackers. > Also, for a non-uniform cluster, we have to deploy different sets of > configuration. > Now JobTracker holds the CPU and memory status of TaskTrackers > (MAPREDUCE-1218). > So it makes sense to just let JobTracker.taskScheduler decided the number of > slots on each node. > This way we can > 1. Change the number of slots dynamically without restarting TaskTracker > 2. Use different number of slots based on the resource of a TaskTracker > To achieve this, we need to change the logic that we use totalMapSlots and > totalReduceSlots in JobTracker. > I think they are used in WebUI and speculativeCap. > We will need to make JobTracker calculate these numbers from TaskScheduler > and TaskTrackerStatus. > TaskScheduler and TaskTracker can both hold their maximum slots. We pick the > smaller one. > Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2108) Allow TaskScheduler manage number slots on TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105118#comment-13105118 ] Scott Chen commented on MAPREDUCE-2108: --- Arun: Thanks for the comments. You are right. I guess this is not an issue since we have MRv2. Closing this now. > Allow TaskScheduler manage number slots on TaskTrackers > --- > > Key: MAPREDUCE-2108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2108 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/capacity-sched, contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2108-v2.txt, MAPREDUCE-2108.txt > > > Currently the map slots and reduce slots are managed by TaskTracker > configuration. > To change the task tracker slots, we need to restart the TaskTrackers. > Also, for a non-uniform cluster, we have to deploy different sets of > configuration. > Now JobTracker holds the CPU and memory status of TaskTrackers > (MAPREDUCE-1218). > So it makes sense to just let JobTracker.taskScheduler decided the number of > slots on each node. > This way we can > 1. Change the number of slots dynamically without restarting TaskTracker > 2. Use different number of slots based on the resource of a TaskTracker > To achieve this, we need to change the logic that we use totalMapSlots and > totalReduceSlots in JobTracker. > I think they are used in WebUI and speculativeCap. > We will need to make JobTracker calculate these numbers from TaskScheduler > and TaskTrackerStatus. > TaskScheduler and TaskTracker can both hold their maximum slots. We pick the > smaller one. > Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2994) Parse Error is coming for App ID when we click application link on the RM UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-2994:
-
Component/s: mrv2
Fix Version/s: 0.23.0
> Parse Error is coming for App ID when we click application link on the RM UI
>
>
> Key: MAPREDUCE-2994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2994
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
>Affects Versions: 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2994.patch
>
>
> {code:xml}
> Caused by: org.apache.hadoop.yarn.YarnException: Error parsing app ID:
> application_1315895242400_1
> at org.apache.hadoop.yarn.util.Apps.throwParseException(Apps.java:60)
> at org.apache.hadoop.yarn.util.Apps.toAppID(Apps.java:43)
> at org.apache.hadoop.yarn.util.Apps.toAppID(Apps.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RmController.app(RmController.java:74)
> ... 30 more
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2965: -- Attachment: MR2965_v4.patch Same patch without the changes to dev-support/test-patch.properties > Streamline hashCode(), equals(), compareTo() and toString() for all IDs > --- > > Key: MAPREDUCE-2965 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2965_v1.patch, MR2965_v2.patch, MR2965_v3.patch, > MR2965_v4.patch > > > MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls > for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they > don't need to be tied to the implementation. > We should do the same for all IDs. In fact some of these are missing for IDs > like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105111#comment-13105111 ] Hadoop QA commented on MAPREDUCE-2672: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494562/MR-2672.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/744//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/744//console This message is automatically generated. > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > MR-2672.patch, MR-2672.patch, Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3011) TT should remove bad local dirs from conf to prevent constant disk checking
TT should remove bad local dirs from conf to prevent constant disk checking --- Key: MAPREDUCE-3011 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3011 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Eli Collins Fix For: 0.20.205.0 Per HADOOP-7551 the TT does not remove bad mapred.local.dirs from the conf so after a single disk failure *every* call to get a local path for reading or writing results in a disk check of *all* configured local dirs. After detecting that a local dir is bad we should remove it from the conf so that we don't repeatedly perform this expensive operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105109#comment-13105109 ] Hadoop QA commented on MAPREDUCE-2965: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494560/MR2965_v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/745//console This message is automatically generated. > Streamline hashCode(), equals(), compareTo() and toString() for all IDs > --- > > Key: MAPREDUCE-2965 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2965_v1.patch, MR2965_v2.patch, MR2965_v3.patch > > > MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls > for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they > don't need to be tied to the implementation. > We should do the same for all IDs. In fact some of these are missing for IDs > like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2672: -- Status: Patch Available (was: Open) > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > MR-2672.patch, MR-2672.patch, Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2672: -- Attachment: MR-2672.patch Fixed the history tests and some AM webapp tests to construct the actual page instead of an error page. > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > MR-2672.patch, MR-2672.patch, Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105106#comment-13105106 ] Hadoop QA commented on MAPREDUCE-2965: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494560/MR2965_v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/743//console This message is automatically generated. > Streamline hashCode(), equals(), compareTo() and toString() for all IDs > --- > > Key: MAPREDUCE-2965 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2965_v1.patch, MR2965_v2.patch, MR2965_v3.patch > > > MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls > for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they > don't need to be tied to the implementation. > We should do the same for all IDs. In fact some of these are missing for IDs > like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2965: -- Status: Patch Available (was: Open) > Streamline hashCode(), equals(), compareTo() and toString() for all IDs > --- > > Key: MAPREDUCE-2965 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2965_v1.patch, MR2965_v2.patch, MR2965_v3.patch > > > MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls > for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they > don't need to be tied to the implementation. > We should do the same for all IDs. In fact some of these are missing for IDs > like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2965: -- Attachment: MR2965_v3.patch Updated patch. Re-using number format. Updated allowed releaseaudit and findbugs warnings to 0 bq. Several needed null-checks are missing in toString() and comparesTo() methods in all the IDs. Have hesitantly added some null checks to toString() (not compareTo). I don't think we should be adding these null checks for toString, hashCode and comapreTo. They'll just end up masking actual errors. Are there any cases for the ids to be used without all parameters set ? bq. Thought more about this, and realized I am wrong about this. Because getProto() itself is synchronized, and we only use the getters which are also synchronized, we are good. Yep, we should be fine - and without deadlocks like 2954. > Streamline hashCode(), equals(), compareTo() and toString() for all IDs > --- > > Key: MAPREDUCE-2965 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2965_v1.patch, MR2965_v2.patch, MR2965_v3.patch > > > MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls > for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they > don't need to be tied to the implementation. > We should do the same for all IDs. In fact some of these are missing for IDs > like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2989) JobHistory should link to task logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105094#comment-13105094 ] Siddharth Seth commented on MAPREDUCE-2989: --- Agreed. Jira was just to track that JH eventually gets a working log link. > JobHistory should link to task logs > --- > > Key: MAPREDUCE-2989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2989 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Siddharth Seth > > The log link on the task attempt page is currently broken - since it relies > on a ContainerId. We should either pass the containerId via a history event - > or some kind of field with information about the log location. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3002) Delink History Context from AppContext
[
https://issues.apache.org/jira/browse/MAPREDUCE-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105092#comment-13105092
]
Siddharth Seth commented on MAPREDUCE-3002:
---
{{AppContext}} is meant for a single Application which may have multiple jobs.
This was used in MR JobHistory to be able to share the UI - all known jobs end
up belonging to a single dummy App.
Most of the {{AppContext}} interface isn't useful for JobHistory -
{{getAppAttemptId()}}, {{getStartTime()}} etc. Functionality required by
JobHistory is getAllJobsForListingPage(params?), getJob(JobId) and at some
point it may support Apps.
The current equivalent of getAllJobsForListingPage - returns a partial
implementation of the {{Job}} interface. This could be replaced by whatever it
is we require on the listing page. getJob ofc can continue to use the {{Job}}
interface.
> Delink History Context from AppContext
> --
>
> Key: MAPREDUCE-3002
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3002
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2
>Affects Versions: 0.24.0
>Reporter: Robert Joseph Evans
>
> Currently the JobHistory Server has a HistoryContext that pretends to be a
> Map Reduce ApplicationMaster's AppContext so that UI pages can be shared
> between the two. This is not ideal because the UIs have already diverged a
> lot, and we have to translate the native History Server's data into
> implementations of Job to provide the same interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3004) sort example fails in shuffle/reduce stage as it assumes a local job by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105083#comment-13105083 ] Hitesh Shah commented on MAPREDUCE-3004: Ran the sort example manually wihout specifying a job tracker address in the mapred config. Not sure if there is a simple way to enable a automated test for this case. > sort example fails in shuffle/reduce stage as it assumes a local job by > default > > > Key: MAPREDUCE-3004 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3004 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.0 > > Attachments: mapreduce-3004-branch-0.23.patch > > > Log trace when running sort on a single node setup: > 11/09/13 17:01:06 INFO mapreduce.Job: map 100% reduce 0% > 11/09/13 17:01:10 INFO mapreduce.Job: Task Id : > attempt_1315949787252_0009_r_00_0, Status : FAILED > java.lang.UnsupportedOperationException: Incompatible with LocalRunner > at > org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200) > at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:183) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:365) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3007) JobClient cannot talk to JobHistory server in secure mode
[
https://issues.apache.org/jira/browse/MAPREDUCE-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105079#comment-13105079
]
Mahadev konar commented on MAPREDUCE-3007:
--
Vinod,
The conf property setting is not needed any longer:
{code}
myConf.setClass(YarnConfiguration.YARN_SECURITY_INFO,
-ClientRMSecurityInfo.class, SecurityInfo.class);
-YarnRPC rpc = YarnRPC.create(myConf);
+ClientHSSecurityInfo.class, SecurityInfo.class);
{code}
Setting the conf property is redundant since SecurityUtil picks it up via going
through the providers. The only issue with HS is that both the AM and HS
implement the same interface (MRClientProtocol) one of which is token based and
the other kerberos info based (the fact that getKerberos in
MRClientSecurityInfo returns null saves the day).
I tried out the patch on a cluster. Had to remove the sleep in AppMaster to
reproduce the issue. The patch worked for me, though I started getting some
other error on getTaskCompletions(). I think thats more related to the
handshake for history file transfer not getting completed becasue of immediate
AppMaster shutdown.
In all, the patch looks good with minor changes to removing the conf settings
mentioned above.
> JobClient cannot talk to JobHistory server in secure mode
> -
>
> Key: MAPREDUCE-3007
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3007
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3007-20110914.2.txt,
> MAPREDUCE-3007-20110914.txt
>
>
> In secure mode, Jobclient cannot connect to HistoryServer. Thanks to
> [~karams] for finding this out.
> {code}
> 11/09/14 09:57:51 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server
> 11/09/14 09:57:51 INFO security.ApplicationTokenSelector: Looking for a token
> with service :10020
> 11/09/14 09:57:51 INFO security.ApplicationTokenSelector: Token kind is
> YARN_APPLICATION_TOKEN and the token's service name is :46257
> 11/09/14 09:57:51 INFO security.UserGroupInformation: Initiating logout for
>
> 11/09/14 09:57:51 INFO security.UserGroupInformation: Initiating re-login for
>
> 11/09/14 09:57:55 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:57:56 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:00 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:05 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:05 WARN ipc.Client: Couldn't setup connection for
> to null
> 11/09/14 09:58:05 INFO mapred.ClientServiceDelegate: Failed to contact
> AM/History for job job_1315993268700_0001 Will retry..
> {code}
> Am surprised no one working with YARN+MR ever ran into this!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105066#comment-13105066 ] Eli Collins edited comment on MAPREDUCE-2957 at 9/15/11 2:06 AM: - The TT trys to shut down due to the DiskErrorException thrown from checkLocalDirs, however because offerService returns STALE in this case run tries to reinitialize instead of shutdown. was (Author: eli): It's buggy, the TT shuts down due to the DiskErrorException thrown from checkLocalDirs, however because we return STALE from offerService run tries to reinitialized instead of shutdown. > The TT should not re-init if it has no good local dirs > -- > > Key: MAPREDUCE-2957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.20.205.0 > > Attachments: mapreduce-2957.patch > > > The TT will currently try to re-init itself on disk failure even if it has no > good local dirs. It should shutdown instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105066#comment-13105066 ] Eli Collins commented on MAPREDUCE-2957: It's buggy, the TT shuts down due to the DiskErrorException thrown from checkLocalDirs, however because we return STALE from offerService run tries to reinitialized instead of shutdown. > The TT should not re-init if it has no good local dirs > -- > > Key: MAPREDUCE-2957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Eli Collins > Fix For: 0.20.205.0 > > Attachments: mapreduce-2957.patch > > > The TT will currently try to re-init itself on disk failure even if it has no > good local dirs. It should shutdown instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-2957: --- Attachment: mapreduce-2957.patch Patch attached. > The TT should not re-init if it has no good local dirs > -- > > Key: MAPREDUCE-2957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Eli Collins > Fix For: 0.20.205.0 > > Attachments: mapreduce-2957.patch > > > The TT will currently try to re-init itself on disk failure even if it has no > good local dirs. It should shutdown instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned MAPREDUCE-2957: -- Assignee: Eli Collins > The TT should not re-init if it has no good local dirs > -- > > Key: MAPREDUCE-2957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.20.205.0 > > Attachments: mapreduce-2957.patch > > > The TT will currently try to re-init itself on disk failure even if it has no > good local dirs. It should shutdown instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2991) queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105064#comment-13105064 ] Hudson commented on MAPREDUCE-2991: --- Integrated in Hadoop-Mapreduce-trunk-Commit #902 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/902/]) MAPREDUCE-2991. queueinfo.jsp fails if queue names contain special symbols. Contributed by Priyo Mustafi. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170917 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp > queueinfo.jsp fails to show queue status if any Capacity scheduler queue name > has dash/hiphen in it. > > > Key: MAPREDUCE-2991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2991 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 0.22.0 >Reporter: Priyo Mustafi >Assignee: Priyo Mustafi > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: Job Queue Information page.html, Job Queue Information > page.pdf, MAPREDUCE_2991_022.patch, MAPREDUCE_2991_trunk.patch, > mapred-queues.xml > > > If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any > queue information. This is happening because the queue name is used to > create javascript variables and javascript doesn't allow dash in variable > names. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2991) queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-2991: --- Fix Version/s: 0.23.0 > queueinfo.jsp fails to show queue status if any Capacity scheduler queue name > has dash/hiphen in it. > > > Key: MAPREDUCE-2991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2991 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 0.22.0 >Reporter: Priyo Mustafi >Assignee: Priyo Mustafi > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: Job Queue Information page.html, Job Queue Information > page.pdf, MAPREDUCE_2991_022.patch, MAPREDUCE_2991_trunk.patch, > mapred-queues.xml > > > If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any > queue information. This is happening because the queue name is used to > create javascript variables and javascript doesn't allow dash in variable > names. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3004) sort example fails in shuffle/reduce stage as it assumes a local job by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105061#comment-13105061 ] Hadoop QA commented on MAPREDUCE-3004: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494551/mapreduce-3004-branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/742//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/742//console This message is automatically generated. > sort example fails in shuffle/reduce stage as it assumes a local job by > default > > > Key: MAPREDUCE-3004 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3004 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.0 > > Attachments: mapreduce-3004-branch-0.23.patch > > > Log trace when running sort on a single node setup: > 11/09/13 17:01:06 INFO mapreduce.Job: map 100% reduce 0% > 11/09/13 17:01:10 INFO mapreduce.Job: Task Id : > attempt_1315949787252_0009_r_00_0, Status : FAILED > java.lang.UnsupportedOperationException: Incompatible with LocalRunner > at > org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200) > at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:183) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:365) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2991) queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105054#comment-13105054 ] Hudson commented on MAPREDUCE-2991: --- Integrated in Hadoop-Common-trunk-Commit #890 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/890/]) MAPREDUCE-2991. queueinfo.jsp fails if queue names contain special symbols. Contributed by Priyo Mustafi. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170917 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp > queueinfo.jsp fails to show queue status if any Capacity scheduler queue name > has dash/hiphen in it. > > > Key: MAPREDUCE-2991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2991 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 0.22.0 >Reporter: Priyo Mustafi >Assignee: Priyo Mustafi > Fix For: 0.22.0, 0.24.0 > > Attachments: Job Queue Information page.html, Job Queue Information > page.pdf, MAPREDUCE_2991_022.patch, MAPREDUCE_2991_trunk.patch, > mapred-queues.xml > > > If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any > queue information. This is happening because the queue name is used to > create javascript variables and javascript doesn't allow dash in variable > names. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2991) queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105056#comment-13105056 ] Hudson commented on MAPREDUCE-2991: --- Integrated in Hadoop-Hdfs-trunk-Commit #967 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/967/]) MAPREDUCE-2991. queueinfo.jsp fails if queue names contain special symbols. Contributed by Priyo Mustafi. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170917 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp > queueinfo.jsp fails to show queue status if any Capacity scheduler queue name > has dash/hiphen in it. > > > Key: MAPREDUCE-2991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2991 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 0.22.0 >Reporter: Priyo Mustafi >Assignee: Priyo Mustafi > Fix For: 0.22.0, 0.24.0 > > Attachments: Job Queue Information page.html, Job Queue Information > page.pdf, MAPREDUCE_2991_022.patch, MAPREDUCE_2991_trunk.patch, > mapred-queues.xml > > > If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any > queue information. This is happening because the queue name is used to > create javascript variables and javascript doesn't allow dash in variable > names. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2991) queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-2991: --- Resolution: Fixed Fix Version/s: 0.24.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. > queueinfo.jsp fails to show queue status if any Capacity scheduler queue name > has dash/hiphen in it. > > > Key: MAPREDUCE-2991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2991 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 0.22.0 >Reporter: Priyo Mustafi >Assignee: Priyo Mustafi > Fix For: 0.22.0, 0.24.0 > > Attachments: Job Queue Information page.html, Job Queue Information > page.pdf, MAPREDUCE_2991_022.patch, MAPREDUCE_2991_trunk.patch, > mapred-queues.xml > > > If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any > queue information. This is happening because the queue name is used to > create javascript variables and javascript doesn't allow dash in variable > names. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2991) queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105052#comment-13105052 ] Konstantin Shvachko commented on MAPREDUCE-2991: +1 lgtm > queueinfo.jsp fails to show queue status if any Capacity scheduler queue name > has dash/hiphen in it. > > > Key: MAPREDUCE-2991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2991 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 0.22.0 >Reporter: Priyo Mustafi >Assignee: Priyo Mustafi > Fix For: 0.22.0 > > Attachments: Job Queue Information page.html, Job Queue Information > page.pdf, MAPREDUCE_2991_022.patch, MAPREDUCE_2991_trunk.patch, > mapred-queues.xml > > > If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any > queue information. This is happening because the queue name is used to > create javascript variables and javascript doesn't allow dash in variable > names. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3004) sort example fails in shuffle/reduce stage as it assumes a local job by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3004: --- Attachment: mapreduce-3004-branch-0.23.patch > sort example fails in shuffle/reduce stage as it assumes a local job by > default > > > Key: MAPREDUCE-3004 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3004 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.0 > > Attachments: mapreduce-3004-branch-0.23.patch > > > Log trace when running sort on a single node setup: > 11/09/13 17:01:06 INFO mapreduce.Job: map 100% reduce 0% > 11/09/13 17:01:10 INFO mapreduce.Job: Task Id : > attempt_1315949787252_0009_r_00_0, Status : FAILED > java.lang.UnsupportedOperationException: Incompatible with LocalRunner > at > org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200) > at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:183) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:365) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3004) sort example fails in shuffle/reduce stage as it assumes a local job by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3004: --- Status: Patch Available (was: Open) Changed ReduceTask to check both framework name and master address to determine if local > sort example fails in shuffle/reduce stage as it assumes a local job by > default > > > Key: MAPREDUCE-3004 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3004 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.0 > > Attachments: mapreduce-3004-branch-0.23.patch > > > Log trace when running sort on a single node setup: > 11/09/13 17:01:06 INFO mapreduce.Job: map 100% reduce 0% > 11/09/13 17:01:10 INFO mapreduce.Job: Task Id : > attempt_1315949787252_0009_r_00_0, Status : FAILED > java.lang.UnsupportedOperationException: Incompatible with LocalRunner > at > org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200) > at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:183) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:365) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3004) sort example fails in shuffle/reduce stage as it assumes a local job by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3004: --- Fix Version/s: 0.23.0 > sort example fails in shuffle/reduce stage as it assumes a local job by > default > > > Key: MAPREDUCE-3004 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3004 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.0 > > > Log trace when running sort on a single node setup: > 11/09/13 17:01:06 INFO mapreduce.Job: map 100% reduce 0% > 11/09/13 17:01:10 INFO mapreduce.Job: Task Id : > attempt_1315949787252_0009_r_00_0, Status : FAILED > java.lang.UnsupportedOperationException: Incompatible with LocalRunner > at > org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200) > at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:183) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:365) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2672: - Status: Open (was: Patch Available) Bobby/Sid, can you please look at the test failure? Tx > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > MR-2672.patch, Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105032#comment-13105032 ] Hadoop QA commented on MAPREDUCE-2672: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494545/MR-2672.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/741//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/741//console This message is automatically generated. > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > MR-2672.patch, Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in java.library.path
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105023#comment-13105023 ] Arun C Murthy commented on MAPREDUCE-2801: -- Bobby, the more I think about this approach the less I like it. Essentially relying on NM to supply critical env for the task is a broken concept - this ensures that you can't run the same task with different env. Here 'env' can be jars, shared-objects etc. I'm thinking we shud just bite the bullet, go the whole 9 yards and fix the classpath/ld_library_path etc. in the *correct* way i.e. throw out our reliance on bin/mapred and do everything via the DistributedCache i.e. MAPREDUCE-2880. Thoughts? Apologies for dragging this. IAC, this won't fix the original issue - the NM doesn't have the right java.library.path for setting up compression libs correctly. Arguably we can fix this, but I'd rather do it the *right* way i.e. MAPREDUCE-2880. > Include the native libs in java.library.path > - > > Key: MAPREDUCE-2801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2801.patch, MR-2801.patch > > > For the child tasks in mrv2, java.library.path is set to just $PWD and the > native libs are not included. Whereas in 0.20.x, java.library.path for child > tasks was set to :$PWD -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2880) Fix classpath construction for MRv2
[ https://issues.apache.org/jira/browse/MAPREDUCE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2880: - Assignee: Arun C Murthy I'll take this up. We need it to fix this asap, particularly for issues like MAPREDUCE-2801. > Fix classpath construction for MRv2 > --- > > Key: MAPREDUCE-2880 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2880 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Luke Lu >Assignee: Arun C Murthy > Fix For: 0.23.0 > > > MRConstants.java refers a hard-coded version of MR AM jar. The build config > works around with a symlink. The deployment currently needs symlink > workaround as well. We need to fix this so that we can actually launch > arbitrary versions of AMs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2672: -- Status: Patch Available (was: Open) Bobby, could you please take a look at the patch. Thx > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > MR-2672.patch, Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in java.library.path
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2801: - Status: Open (was: Patch Available) > Include the native libs in java.library.path > - > > Key: MAPREDUCE-2801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2801.patch, MR-2801.patch > > > For the child tasks in mrv2, java.library.path is set to just $PWD and the > native libs are not included. Whereas in 0.20.x, java.library.path for child > tasks was set to :$PWD -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2672: -- Attachment: MR-2672.patch Adds a sort/merge complete field to the task pages. Also, unrelated to this jira, a JH UI fix for elapse time using finishTime as currentTime if it isn't set. > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > MR-2672.patch, Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2836) Provide option to fail jobs when submitted to non-existent pools.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104984#comment-13104984
]
Todd Lipcon commented on MAPREDUCE-2836:
{code}
+ public final static String ALLOW_UNDECLARED_POOLS_KEY =
+"mapred.fairscheduler.allow.undeclared.pools"; // A configuration
property
+// that controls the ability of submitting jobs to pools not declared in
the
+// scheduler allocation file.
+private boolean allowUndeclaredPools = false;
{code}
Please convert the comment describing this variable to be a javadoc comment
that proceeds the declaration.
Also, indentation is messed up on {{allowUndeclaredPools}}
{code}
+ private Boolean submitJob(String pool) throws IOException {
{code}
The result of this function is never used, so may as well make it {{void}}. Or,
perhaps it should return {{boolean}} (unboxed) and then the callers should
assert that the job was successful in testValidPoolName
{code}
+if (pool != null)
+ conf.set(POOL_PROPERTY, pool);
{code}
Style nit: add braces
{code}
+ } catch (IOException ioe){
+LOG.error("Problem in job submission: " + ioe);
+throw ioe;
{code}
Would be nice if the log message here included the job ID.
{code}
+ * Subclasses can override to provide any scheduler-specific checking
+ * mechanism for job submission.
+ * @param job
+ * @return
+ * @throws IOException
{code}
No need for empty @return in the javadoc, since it's void.
{code}
+public class TestFairSchedulerPoolNames extends TestCase {
{code}
Should use JUnit 4 style test case - ie don't extend anything, then annotate
the test methods with @Before, @After, and @Test.
Should also update src/docs/src/documentation/content/xdocs/fair_scheduler.xml
to include the new parameter in the documentation, perhaps under the "Scheduler
parameters in mapred-site.xml" section
> Provide option to fail jobs when submitted to non-existent pools.
> -
>
> Key: MAPREDUCE-2836
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2836
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
>Reporter: Jeff Bean
>Assignee: Ahmed Radwan
>Priority: Minor
> Attachments: MAPREDUCE-2836.patch,
> MAPREDUCE-2836_0.20_security.patch, MAPREDUCE-2836_rev2.patch,
> MAPREDUCE-2836_rev3.patch, MAPREDUCE-2836_rev4.patch,
> MAPREDUCE-2836_rev5.patch, MAPREDUCE-2836_trunk.patch,
> MAPREDUCE-2836_trunk_rev2.patch
>
>
> In some environments, it might be desirable to explicitly specify the fair
> scheduler pools and to explicitly fail jobs that are not submitted to any of
> the pools.
> Current behavior of the fair scheduler is to submit jobs to a default pool if
> a pool name isn't specified or to create a pool with the new name if the pool
> name doesn't already exist. There should be a configuration option for the
> fair scheduler that causes it to noisily fail the job if it's submitted to a
> pool that isn't pre-specified or if the specified pool doesn't exist.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3005) MR app hangs because of a NPE in ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104981#comment-13104981
]
Hudson commented on MAPREDUCE-3005:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #901 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/901/])
MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly
enforce locality constraints.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170879
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
> MR app hangs because of a NPE in ResourceManager
>
>
> Key: MAPREDUCE-3005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3005-20110914.txt, MAPREDUCE-3005.patch,
> MAPREDUCE-3005.patch
>
>
> The app hangs and it turns out to be a NPE in ResourceManager. This happened
> two of five times on [~karams]'s sort runs on a big cluster.
> {code}
> 2011-09-12 15:02:33,715 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:230)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:933)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:725)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:579)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:620)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:266)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3005) MR app hangs because of a NPE in ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104973#comment-13104973
]
Hudson commented on MAPREDUCE-3005:
---
Integrated in Hadoop-Hdfs-trunk-Commit #966 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/966/])
MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly
enforce locality constraints.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170879
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
> MR app hangs because of a NPE in ResourceManager
>
>
> Key: MAPREDUCE-3005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3005-20110914.txt, MAPREDUCE-3005.patch,
> MAPREDUCE-3005.patch
>
>
> The app hangs and it turns out to be a NPE in ResourceManager. This happened
> two of five times on [~karams]'s sort runs on a big cluster.
> {code}
> 2011-09-12 15:02:33,715 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:230)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:933)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:725)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:579)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:620)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:266)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3005) MR app hangs because of a NPE in ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104972#comment-13104972
]
Hudson commented on MAPREDUCE-3005:
---
Integrated in Hadoop-Common-trunk-Commit #889 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/889/])
MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly
enforce locality constraints.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170879
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
> MR app hangs because of a NPE in ResourceManager
>
>
> Key: MAPREDUCE-3005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3005-20110914.txt, MAPREDUCE-3005.patch,
> MAPREDUCE-3005.patch
>
>
> The app hangs and it turns out to be a NPE in ResourceManager. This happened
> two of five times on [~karams]'s sort runs on a big cluster.
> {code}
> 2011-09-12 15:02:33,715 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:230)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:933)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:725)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:579)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:620)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:266)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3005) MR app hangs because of a NPE in ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3005:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> MR app hangs because of a NPE in ResourceManager
>
>
> Key: MAPREDUCE-3005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3005-20110914.txt, MAPREDUCE-3005.patch,
> MAPREDUCE-3005.patch
>
>
> The app hangs and it turns out to be a NPE in ResourceManager. This happened
> two of five times on [~karams]'s sort runs on a big cluster.
> {code}
> 2011-09-12 15:02:33,715 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:230)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:933)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:725)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:579)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:620)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:266)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3005) MR app hangs because of a NPE in ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104958#comment-13104958
]
Mahadev konar commented on MAPREDUCE-3005:
--
+1 looks good to me!
> MR app hangs because of a NPE in ResourceManager
>
>
> Key: MAPREDUCE-3005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3005-20110914.txt, MAPREDUCE-3005.patch,
> MAPREDUCE-3005.patch
>
>
> The app hangs and it turns out to be a NPE in ResourceManager. This happened
> two of five times on [~karams]'s sort runs on a big cluster.
> {code}
> 2011-09-12 15:02:33,715 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:230)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:933)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:725)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:579)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:620)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:266)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3010) ant mvn-install doesn't work on hadoop-mapreduce-project
[
https://issues.apache.org/jira/browse/MAPREDUCE-3010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104953#comment-13104953
]
Ravi Prakash commented on MAPREDUCE-3010:
-
Apart from a few warnings I see these errors
{noformat}
-compile-fault-inject:
[echo] Start weaving aspects in place
[iajc]
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/src/test/system/aop/org/apache/hadoop/mapred/JobTrackerAspect.aj:193
[error] The method setUser(String) is undefined for the type JobTracker
[iajc] tracker.setUser(ugi.getShortUserName());
[iajc]
[iajc]
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/src/test/system/aop/org/apache/hadoop/mapred/JobTrackerAspect.aj:198
[error] The method setReady(boolean) is undefined for the type JobTracker
[iajc] tracker.setReady(true);
[iajc] ^^^
{noformat}
> ant mvn-install doesn't work on hadoop-mapreduce-project
>
>
> Key: MAPREDUCE-3010
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3010
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ravi Prakash
>
> Even though ant jar works, ant mvn-install fails in the compile-fault-inject
> step
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3010) ant mvn-install doesn't work on hadoop-mapreduce-project
ant mvn-install doesn't work on hadoop-mapreduce-project Key: MAPREDUCE-3010 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3010 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Prakash Even though ant jar works, ant mvn-install fails in the compile-fault-inject step -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3005) MR app hangs because of a NPE in ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104913#comment-13104913
]
Hadoop QA commented on MAPREDUCE-3005:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12494527/MAPREDUCE-3005.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/740//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/740//console
This message is automatically generated.
> MR app hangs because of a NPE in ResourceManager
>
>
> Key: MAPREDUCE-3005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3005-20110914.txt, MAPREDUCE-3005.patch,
> MAPREDUCE-3005.patch
>
>
> The app hangs and it turns out to be a NPE in ResourceManager. This happened
> two of five times on [~karams]'s sort runs on a big cluster.
> {code}
> 2011-09-12 15:02:33,715 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:230)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:933)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:725)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:579)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:620)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:266)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3005) MR app hangs because of a NPE in ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3005:
-
Status: Patch Available (was: Open)
> MR app hangs because of a NPE in ResourceManager
>
>
> Key: MAPREDUCE-3005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3005-20110914.txt, MAPREDUCE-3005.patch,
> MAPREDUCE-3005.patch
>
>
> The app hangs and it turns out to be a NPE in ResourceManager. This happened
> two of five times on [~karams]'s sort runs on a big cluster.
> {code}
> 2011-09-12 15:02:33,715 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:230)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:933)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:725)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:579)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:620)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:266)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3005) MR app hangs because of a NPE in ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3005:
-
Attachment: MAPREDUCE-3005.patch
Same fix, after testing. Also added a unit test.
> MR app hangs because of a NPE in ResourceManager
>
>
> Key: MAPREDUCE-3005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3005-20110914.txt, MAPREDUCE-3005.patch,
> MAPREDUCE-3005.patch
>
>
> The app hangs and it turns out to be a NPE in ResourceManager. This happened
> two of five times on [~karams]'s sort runs on a big cluster.
> {code}
> 2011-09-12 15:02:33,715 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:230)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:933)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:725)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:579)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:620)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:266)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2899) Replace major parts of ApplicationSubmissionContext with a ContainerLaunchContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104890#comment-13104890 ] Hudson commented on MAPREDUCE-2899: --- Integrated in Hadoop-Mapreduce-0.23-Build #2 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/2/]) MAPREDUCE-2899. Replace major parts of ApplicationSubmissionContext with a ContainerLaunchContext (Arun Murthy via mahadev) - Merging r1170459 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170460 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/capacity-scheduler.xml.template * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerLaunchContext.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerLaunchContextPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/h
[jira] [Commented] (MAPREDUCE-2676) MR-279: JobHistory Job page needs reformatted
[ https://issues.apache.org/jira/browse/MAPREDUCE-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104887#comment-13104887 ] Hudson commented on MAPREDUCE-2676: --- Integrated in Hadoop-Mapreduce-0.23-Build #2 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/2/]) MAPREDUCE-2676. MR-279: JobHistory Job page needs reformatted. (Robert Evans via mahadev) - Merging r1170379 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170380 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/capacity-scheduler.xml.template * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsAttemptsPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsConfPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsController.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/m
[jira] [Commented] (MAPREDUCE-2949) NodeManager in a inconsistent state if a service startup fails.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104888#comment-13104888 ] Hudson commented on MAPREDUCE-2949: --- Integrated in Hadoop-Mapreduce-0.23-Build #2 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/2/]) MAPREDUCE-2949. svn merge -c r1170689 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170690 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java > NodeManager in a inconsistent state if a service startup fails. > --- > > Key: MAPREDUCE-2949 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2949 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.24.0 >Reporter: Ravi Teja Ch N V >Assignee: Ravi Teja Ch N V > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2949.1.patch, MAPREDUCE-2949.patch, > Threaddump.txt > > > When a service startup fails at the Nodemanager, the Nodemanager JVM doesnot > exit as the following threads are still running. > Daemon Thread [Timer for 'NodeManager' metrics system] (Running) > Thread [pool-1-thread-1] (Running) > Thread [Thread-11] (Running) > Thread [DestroyJavaVM] (Running). > As a result, the NodeManager keeps running even though no services are > started. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2966) Add ShutDown hooks for MRV2 processes
[ https://issues.apache.org/jira/browse/MAPREDUCE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104889#comment-13104889 ] Hudson commented on MAPREDUCE-2966: --- Integrated in Hadoop-Mapreduce-0.23-Build #2 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/2/]) MAPREDUCE-2966. svn merge -c r1170746 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170751 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/CompositeService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestCompositeService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > Add ShutDown hooks for MRV2 processes > - > > Key: MAPREDUCE-2966 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2966 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, jobhistoryserver, nodemanager, > resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Abhijit Suresh Shingate >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2966.patch, MAPREDUCE-2966.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > NodeManager registers a shudown hook in case of JVM exit. > Similar way, all other processes RM, HistoryServer, MRAppMaster should also > handle the shutdown gracefully in case of JVM exit. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2836) Provide option to fail jobs when submitted to non-existent pools.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104873#comment-13104873 ] Hadoop QA commented on MAPREDUCE-2836: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494511/MAPREDUCE-2836_0.20_security.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/739//console This message is automatically generated. > Provide option to fail jobs when submitted to non-existent pools. > - > > Key: MAPREDUCE-2836 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2836 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Jeff Bean >Assignee: Ahmed Radwan >Priority: Minor > Attachments: MAPREDUCE-2836.patch, > MAPREDUCE-2836_0.20_security.patch, MAPREDUCE-2836_rev2.patch, > MAPREDUCE-2836_rev3.patch, MAPREDUCE-2836_rev4.patch, > MAPREDUCE-2836_rev5.patch, MAPREDUCE-2836_trunk.patch, > MAPREDUCE-2836_trunk_rev2.patch > > > In some environments, it might be desirable to explicitly specify the fair > scheduler pools and to explicitly fail jobs that are not submitted to any of > the pools. > Current behavior of the fair scheduler is to submit jobs to a default pool if > a pool name isn't specified or to create a pool with the new name if the pool > name doesn't already exist. There should be a configuration option for the > fair scheduler that causes it to noisily fail the job if it's submitted to a > pool that isn't pre-specified or if the specified pool doesn't exist. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2836) Provide option to fail jobs when submitted to non-existent pools.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-2836: Attachment: MAPREDUCE-2836_0.20_security.patch Since the backport patch for MR-2981 is committed, I am attaching the patch for the "branch-0.20-security" branch. > Provide option to fail jobs when submitted to non-existent pools. > - > > Key: MAPREDUCE-2836 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2836 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Jeff Bean >Assignee: Ahmed Radwan >Priority: Minor > Attachments: MAPREDUCE-2836.patch, > MAPREDUCE-2836_0.20_security.patch, MAPREDUCE-2836_rev2.patch, > MAPREDUCE-2836_rev3.patch, MAPREDUCE-2836_rev4.patch, > MAPREDUCE-2836_rev5.patch, MAPREDUCE-2836_trunk.patch, > MAPREDUCE-2836_trunk_rev2.patch > > > In some environments, it might be desirable to explicitly specify the fair > scheduler pools and to explicitly fail jobs that are not submitted to any of > the pools. > Current behavior of the fair scheduler is to submit jobs to a default pool if > a pool name isn't specified or to create a pool with the new name if the pool > name doesn't already exist. There should be a configuration option for the > fair scheduler that causes it to noisily fail the job if it's submitted to a > pool that isn't pre-specified or if the specified pool doesn't exist. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2991) queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104852#comment-13104852 ] Hadoop QA commented on MAPREDUCE-2991: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494484/Job+Queue+Information+page.pdf against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/738//console This message is automatically generated. > queueinfo.jsp fails to show queue status if any Capacity scheduler queue name > has dash/hiphen in it. > > > Key: MAPREDUCE-2991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2991 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 0.22.0 >Reporter: Priyo Mustafi >Assignee: Priyo Mustafi > Fix For: 0.22.0 > > Attachments: Job Queue Information page.html, Job Queue Information > page.pdf, MAPREDUCE_2991_022.patch, MAPREDUCE_2991_trunk.patch, > mapred-queues.xml > > > If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any > queue information. This is happening because the queue name is used to > create javascript variables and javascript doesn't allow dash in variable > names. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2925) job -status is giving continuously info message for completed jobs on the console
[
https://issues.apache.org/jira/browse/MAPREDUCE-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104836#comment-13104836
]
Devaraj K commented on MAPREDUCE-2925:
--
Thanks Arun for reviewing and suggestion.
There are few problems around this.
1. If the RM doesn't give application report it is giving NullPointerException.
This can be handled by redirecting to history server as it still may
aware of the application.
2. After redirecting to History Server, if the history server doesn't have
information about it(or it failed to give because of some other reason), it is
going to infinite loop and keep on printing the message.
I have faced the similar problem. RM is giving the application report
with status as success and then it is redirecting to History server. History
server is not able to find the application info, it throwing the exception.
That is converting to InvocationTargetException and it is retrying infinitely.
3. If it throws other than 'YarnRemoteException' and
'InvocationTargetException' also it goes to infinite times. This needs to break
at some point.
Here we need to differentiate remote end exceptions and connection failures to
RM/AM/HS, if it is remote end exception then it can be reported directly. If it
is connection failure then retry can happen in the RPC and after retries it can
be reported.
Please provide your suggestions.
> job -status is giving continuously info message for completed jobs
> on the console
> --
>
> Key: MAPREDUCE-2925
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2925
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2925.patch
>
>
> This below message is coming continuously on the console.
> {code:xml}
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Failed to contact AM for
> job job_1314955256658_0009 Will retry..
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server null
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Failed to contact AM for
> job job_1314955256658_0009 Will retry..
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server null
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2738) Missing cluster level stats on the RM UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned MAPREDUCE-2738: -- Assignee: Robert Joseph Evans (was: John George) > Missing cluster level stats on the RM UI > > > Key: MAPREDUCE-2738 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > > Cluster usage information such as the following are currently not available > in the RM UI. > - Total number of apps submitted so far > - Total number of containers running/total memory usage > - Total capacity of the cluster (in terms of memory) > - Reserved memory > - Total number of NMs - sorting based on Node IDs is an option but when there > are lost NMs or restarted NMs, the node ids does not correspond to the actual > value > - Blacklisted NMs - sorting based on health-status and counting manually is > not very straight forward > - Excluded NMs > - Handle to the jobhistory server -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on the RM UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104809#comment-13104809 ] Robert Joseph Evans commented on MAPREDUCE-2738: Stealing this from John :). > Missing cluster level stats on the RM UI > > > Key: MAPREDUCE-2738 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > > Cluster usage information such as the following are currently not available > in the RM UI. > - Total number of apps submitted so far > - Total number of containers running/total memory usage > - Total capacity of the cluster (in terms of memory) > - Reserved memory > - Total number of NMs - sorting based on Node IDs is an option but when there > are lost NMs or restarted NMs, the node ids does not correspond to the actual > value > - Blacklisted NMs - sorting based on health-status and counting manually is > not very straight forward > - Excluded NMs > - Handle to the jobhistory server -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[
https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104793#comment-13104793
]
Alejandro Abdelnur commented on MAPREDUCE-3003:
---
Arun, no worries.
I'm not sure how Luke got that result. Pelase try the following:
Run:
{code}
$ mkdir /tmp/repo
$ mvn deploy -P-cbuild -DskipTests
-DaltDeploymentRepository=foo::default::file:///tmp/repo
{code}
And then check any of the MR poms under /tmp/repo, you'll see that the
variables are not resolved, for example:
{code}
hadoop-mapreduce-client
org.apache.hadoop
${hadoop-mapreduce.version}
4.0.0
org.apache.hadoop
hadoop-mapreduce-client-core
hadoop-mapreduce-client-core
{code}
> Publish Yarn and MapReduce artifacts to Maven snapshot repository
> -
>
> Key: MAPREDUCE-3003
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Reporter: Tom White
>Assignee: Tom White
> Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch,
> MAPREDUCE-3003.patch
>
>
> Currently this is failing since no distribution management section is defined
> in the POM.
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in java.library.path
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104777#comment-13104777 ] Hadoop QA commented on MAPREDUCE-2801: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494478/MR-2801.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/737//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/737//console This message is automatically generated. > Include the native libs in java.library.path > - > > Key: MAPREDUCE-2801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2801.patch, MR-2801.patch > > > For the child tasks in mrv2, java.library.path is set to just $PWD and the > native libs are not included. Whereas in 0.20.x, java.library.path for child > tasks was set to :$PWD -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2966) Add ShutDown hooks for MRV2 processes
[ https://issues.apache.org/jira/browse/MAPREDUCE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104778#comment-13104778 ] Hudson commented on MAPREDUCE-2966: --- Integrated in Hadoop-Common-trunk-Commit #888 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/888/]) MAPREDUCE-2966. Added ShutDown hooks for MRV2 processes so that they can gracefully exit. Contributed by Abhijit Suresh Shingate. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170746 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/CompositeService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestCompositeService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > Add ShutDown hooks for MRV2 processes > - > > Key: MAPREDUCE-2966 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2966 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, jobhistoryserver, nodemanager, > resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Abhijit Suresh Shingate >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2966.patch, MAPREDUCE-2966.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > NodeManager registers a shudown hook in case of JVM exit. > Similar way, all other processes RM, HistoryServer, MRAppMaster should also > handle the shutdown gracefully in case of JVM exit. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2966) Add ShutDown hooks for MRV2 processes
[ https://issues.apache.org/jira/browse/MAPREDUCE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104775#comment-13104775 ] Hudson commented on MAPREDUCE-2966: --- Integrated in Hadoop-Hdfs-trunk-Commit #965 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/965/]) MAPREDUCE-2966. Added ShutDown hooks for MRV2 processes so that they can gracefully exit. Contributed by Abhijit Suresh Shingate. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170746 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/CompositeService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestCompositeService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > Add ShutDown hooks for MRV2 processes > - > > Key: MAPREDUCE-2966 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2966 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, jobhistoryserver, nodemanager, > resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Abhijit Suresh Shingate >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2966.patch, MAPREDUCE-2966.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > NodeManager registers a shudown hook in case of JVM exit. > Similar way, all other processes RM, HistoryServer, MRAppMaster should also > handle the shutdown gracefully in case of JVM exit. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2991) queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyo Mustafi updated MAPREDUCE-2991: - Attachment: Job Queue Information page.pdf Job Queue Information page.html mapred-queues.xml Used the attached mapred-queues.xml file to test. The queueinfo.jsp showed up as expected (attached) > queueinfo.jsp fails to show queue status if any Capacity scheduler queue name > has dash/hiphen in it. > > > Key: MAPREDUCE-2991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2991 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 0.22.0 >Reporter: Priyo Mustafi >Assignee: Priyo Mustafi > Fix For: 0.22.0 > > Attachments: Job Queue Information page.html, Job Queue Information > page.pdf, MAPREDUCE_2991_022.patch, MAPREDUCE_2991_trunk.patch, > mapred-queues.xml > > > If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any > queue information. This is happening because the queue name is used to > create javascript variables and javascript doesn't allow dash in variable > names. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104764#comment-13104764 ] Arun C Murthy commented on MAPREDUCE-3003: -- Alejandro, please excuse my maven n00biness. Here is the original comment from Luke: https://issues.apache.org/jira/browse/MAPREDUCE-2879?focusedCommentId=13091212&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13091212 > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in java.library.path
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2801: --- Attachment: MR-2801.patch Upmerged to latest trunk, to resolve some merge conflicts. > Include the native libs in java.library.path > - > > Key: MAPREDUCE-2801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2801.patch, MR-2801.patch > > > For the child tasks in mrv2, java.library.path is set to just $PWD and the > native libs are not included. Whereas in 0.20.x, java.library.path for child > tasks was set to :$PWD -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in java.library.path
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2801: --- Status: Patch Available (was: Open) > Include the native libs in java.library.path > - > > Key: MAPREDUCE-2801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2801.patch, MR-2801.patch > > > For the child tasks in mrv2, java.library.path is set to just $PWD and the > native libs are not included. Whereas in 0.20.x, java.library.path for child > tasks was set to :$PWD -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104751#comment-13104751 ] Robert Joseph Evans commented on MAPREDUCE-2672: Yes I am fine with you adding it in. Happy to have all the help I can get. > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2925) job -status is giving continuously info message for completed jobs on the console
[
https://issues.apache.org/jira/browse/MAPREDUCE-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2925:
-
Status: Open (was: Patch Available)
Devaraj, sorry to come in late.
I'm concerned about adding application level retries... a better solution is to
use the 'rpcTimeout' in getProxy to limit #retries. Can you please use that?
Thanks.
> job -status is giving continuously info message for completed jobs
> on the console
> --
>
> Key: MAPREDUCE-2925
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2925
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2925.patch
>
>
> This below message is coming continuously on the console.
> {code:xml}
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Failed to contact AM for
> job job_1314955256658_0009 Will retry..
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server null
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Failed to contact AM for
> job job_1314955256658_0009 Will retry..
> 11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server null
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2966) Add ShutDown hooks for MRV2 processes
[ https://issues.apache.org/jira/browse/MAPREDUCE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104742#comment-13104742 ] Hudson commented on MAPREDUCE-2966: --- Integrated in Hadoop-Mapreduce-trunk-Commit #900 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/900/]) MAPREDUCE-2966. Added ShutDown hooks for MRV2 processes so that they can gracefully exit. Contributed by Abhijit Suresh Shingate. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170746 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/CompositeService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestCompositeService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > Add ShutDown hooks for MRV2 processes > - > > Key: MAPREDUCE-2966 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2966 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, jobhistoryserver, nodemanager, > resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Abhijit Suresh Shingate >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2966.patch, MAPREDUCE-2966.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > NodeManager registers a shudown hook in case of JVM exit. > Similar way, all other processes RM, HistoryServer, MRAppMaster should also > handle the shutdown gracefully in case of JVM exit. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2672: -- Status: Open (was: Patch Available) Updated patch looks good with elapsed times fixed and the additional fields added to the ReducesTasksPage. Cancelling it though to add sort/merge finish time. They're not the same - a sort phase may run after shuffle completes. Bobby, this should be a trivial change on top of your patch. Will post an update shortly, if you're ok with that. > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104734#comment-13104734 ] Alejandro Abdelnur commented on MAPREDUCE-3003: --- @Arun, I don't know what you mean by 'sanitized poms'; but the current poms are unusable when consumed from a Maven repo. The reason is that they done have a hardcoded version, then the parent version cannot be resolved, thus the fetching of the parent fails. Another thing, as best practices, the version of the current pom and the parent pom should be specified. The version plugin takes care of updating both when you want to rev. > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2966) Add ShutDown hooks for MRV2 processes
[ https://issues.apache.org/jira/browse/MAPREDUCE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2966: --- Resolution: Fixed Status: Resolved (was: Patch Available) You took so long, Jenkins! I just committed this to trunk and branch 0.23. Thanks Abhijit! > Add ShutDown hooks for MRV2 processes > - > > Key: MAPREDUCE-2966 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2966 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, jobhistoryserver, nodemanager, > resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Abhijit Suresh Shingate >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2966.patch, MAPREDUCE-2966.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > NodeManager registers a shudown hook in case of JVM exit. > Similar way, all other processes RM, HistoryServer, MRAppMaster should also > handle the shutdown gracefully in case of JVM exit. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2965: --- Status: Open (was: Patch Available) Cancelling the patch to incorporate comments. > Streamline hashCode(), equals(), compareTo() and toString() for all IDs > --- > > Key: MAPREDUCE-2965 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2965_v1.patch, MR2965_v2.patch > > > MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls > for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they > don't need to be tied to the implementation. > We should do the same for all IDs. In fact some of these are missing for IDs > like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104723#comment-13104723 ] Arun C Murthy commented on MAPREDUCE-3003: -- I'm no expert - but AFAIK the 'sanitized poms' were the ones with the substituted versions? Why not keep the parameterized in the normal pom.xml? > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3007) JobClient cannot talk to JobHistory server in secure mode
[
https://issues.apache.org/jira/browse/MAPREDUCE-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3007:
---
Attachment: MAPREDUCE-3007-20110914.2.txt
Correct patch AFAIU.
> JobClient cannot talk to JobHistory server in secure mode
> -
>
> Key: MAPREDUCE-3007
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3007
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3007-20110914.2.txt,
> MAPREDUCE-3007-20110914.txt
>
>
> In secure mode, Jobclient cannot connect to HistoryServer. Thanks to
> [~karams] for finding this out.
> {code}
> 11/09/14 09:57:51 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server
> 11/09/14 09:57:51 INFO security.ApplicationTokenSelector: Looking for a token
> with service :10020
> 11/09/14 09:57:51 INFO security.ApplicationTokenSelector: Token kind is
> YARN_APPLICATION_TOKEN and the token's service name is :46257
> 11/09/14 09:57:51 INFO security.UserGroupInformation: Initiating logout for
>
> 11/09/14 09:57:51 INFO security.UserGroupInformation: Initiating re-login for
>
> 11/09/14 09:57:55 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:57:56 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:00 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:05 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:05 WARN ipc.Client: Couldn't setup connection for
> to null
> 11/09/14 09:58:05 INFO mapred.ClientServiceDelegate: Failed to contact
> AM/History for job job_1315993268700_0001 Will retry..
> {code}
> Am surprised no one working with YARN+MR ever ran into this!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAPREDUCE-3000) Move /mapred to /user/mapred for Hadoop 0.20.205
[
https://issues.apache.org/jira/browse/MAPREDUCE-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104686#comment-13104686
]
Eli Collins edited comment on MAPREDUCE-3000 at 9/14/11 5:24 PM:
-
The default has been ${hadoop.tmp.dir}/mapred/system (not /user/mapred) for a
long time. Per Allen, why does it make sense to put the mapred system dir in a
home directory? That doesn't make sense to me either.
was (Author: eli):
The default has been ${hadoop.tmp.dir}/mapred/system (/user/mapred) for a
long time. Per Allen, why does it make sense to put the mapred system dir in a
home directory? That doesn't make sense to me either.
> Move /mapred to /user/mapred for Hadoop 0.20.205
>
>
> Key: MAPREDUCE-3000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Affects Versions: 0.20.205.0
> Environment: Java 6, RHEL 5.6
>Reporter: Eric Yang
>Assignee: Eric Yang
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-3000.patch
>
>
> /mapred directory should be default to /user/mapred. Owen request this to be
> changed for RPM deployment in 0.20.204. This is a regression in 0.20.205
> that /user/mapred is configured back to /mapred.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3000) Move /mapred to /user/mapred for Hadoop 0.20.205
[
https://issues.apache.org/jira/browse/MAPREDUCE-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104686#comment-13104686
]
Eli Collins commented on MAPREDUCE-3000:
The default has been ${hadoop.tmp.dir}/mapred/system (/user/mapred) for a long
time. Per Allen, why does it make sense to put the mapred system dir in a home
directory? That doesn't make sense to me either.
> Move /mapred to /user/mapred for Hadoop 0.20.205
>
>
> Key: MAPREDUCE-3000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Affects Versions: 0.20.205.0
> Environment: Java 6, RHEL 5.6
>Reporter: Eric Yang
>Assignee: Eric Yang
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-3000.patch
>
>
> /mapred directory should be default to /user/mapred. Owen request this to be
> changed for RPM deployment in 0.20.204. This is a regression in 0.20.205
> that /user/mapred is configured back to /mapred.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2672: --- Status: Patch Available (was: Open) > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2672) MR-279: JobHistory Server needs Analysis this job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2672: --- Attachment: MR-2672.patch Added in more columns to task page and tasks page. Fixed some sorting and filtering issues. Fixed some issues with the task page where it could not get the proper task type. All known issues should be fixed with this patch and all pages should have similar data. > MR-279: JobHistory Server needs Analysis this job > - > > Key: MAPREDUCE-2672 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2672 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2672-trunk-v2.patch, MR-2672-trunk-v3.patch, > MR-2672-trunk-v4.patch, MR-2672-v1.patch, MR-2672.patch, MR-2672.patch, > Updated-Column-Layout.jpg > > > The JobHistory Server needs to implement the Analysis this job functionality > from the previous server. > This should include the following info > Hadoop Job ID > User : > JobName : > JobConf : > Submitted At : > Launched At : (including duration) > Finished At : (including duration) > Status : > Time taken by best performing Map task : > Average time taken by Map tasks: > Worse performing map tasks: (including task links and duration) > The last Map task finished at (relative to the Job launch time): > (including duration) > Time taken by best performing shuffle : > Average time taken by shuffle: > Worse performing Shuffles: (including task links and duration) > The last Shuffle finished at (relative to the Job launch time): > (including duration) > Time taken by best performing Reduce task : > Average time taken by Reduce tasks: > Worse performing reduce tasks: (including task links and duration) > The last Reduce task finished at (relative to the Job launch > time): (including duration) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104661#comment-13104661 ] Alejandro Abdelnur commented on MAPREDUCE-3003: --- +1 tested deploying to an alternate (local) repo and all artifacts are there. > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2966) Add ShutDown hooks for MRV2 processes
[ https://issues.apache.org/jira/browse/MAPREDUCE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104643#comment-13104643 ] Hadoop QA commented on MAPREDUCE-2966: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494454/MAPREDUCE-2966.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/736//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/736//console This message is automatically generated. > Add ShutDown hooks for MRV2 processes > - > > Key: MAPREDUCE-2966 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2966 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, jobhistoryserver, nodemanager, > resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Abhijit Suresh Shingate >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2966.patch, MAPREDUCE-2966.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > NodeManager registers a shudown hook in case of JVM exit. > Similar way, all other processes RM, HistoryServer, MRAppMaster should also > handle the shutdown gracefully in case of JVM exit. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2994) Parse Error is coming for App ID when we click application link on the RM UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104629#comment-13104629
]
Eric Payne commented on MAPREDUCE-2994:
---
+1 (non-binding)
Mae Culpa. I introduced this problem in MAPREDUCE-2874. Sorry about that! I was
attempting to consolidate code.
> Parse Error is coming for App ID when we click application link on the RM UI
>
>
> Key: MAPREDUCE-2994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2994
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-2994.patch
>
>
> {code:xml}
> Caused by: org.apache.hadoop.yarn.YarnException: Error parsing app ID:
> application_1315895242400_1
> at org.apache.hadoop.yarn.util.Apps.throwParseException(Apps.java:60)
> at org.apache.hadoop.yarn.util.Apps.toAppID(Apps.java:43)
> at org.apache.hadoop.yarn.util.Apps.toAppID(Apps.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RmController.app(RmController.java:74)
> ... 30 more
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2949) NodeManager in a inconsistent state if a service startup fails.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104625#comment-13104625 ] Hudson commented on MAPREDUCE-2949: --- Integrated in Hadoop-Hdfs-trunk-Commit #964 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/964/]) MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service startup fails. Contributed by Ravi Teja Ch N V. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170689 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java > NodeManager in a inconsistent state if a service startup fails. > --- > > Key: MAPREDUCE-2949 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2949 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.24.0 >Reporter: Ravi Teja Ch N V >Assignee: Ravi Teja Ch N V > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2949.1.patch, MAPREDUCE-2949.patch, > Threaddump.txt > > > When a service startup fails at the Nodemanager, the Nodemanager JVM doesnot > exit as the following threads are still running. > Daemon Thread [Timer for 'NodeManager' metrics system] (Running) > Thread [pool-1-thread-1] (Running) > Thread [Thread-11] (Running) > Thread [DestroyJavaVM] (Running). > As a result, the NodeManager keeps running even though no services are > started. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2949) NodeManager in a inconsistent state if a service startup fails.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104620#comment-13104620 ] Hudson commented on MAPREDUCE-2949: --- Integrated in Hadoop-Common-trunk-Commit #887 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/887/]) MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service startup fails. Contributed by Ravi Teja Ch N V. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170689 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java > NodeManager in a inconsistent state if a service startup fails. > --- > > Key: MAPREDUCE-2949 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2949 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.24.0 >Reporter: Ravi Teja Ch N V >Assignee: Ravi Teja Ch N V > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2949.1.patch, MAPREDUCE-2949.patch, > Threaddump.txt > > > When a service startup fails at the Nodemanager, the Nodemanager JVM doesnot > exit as the following threads are still running. > Daemon Thread [Timer for 'NodeManager' metrics system] (Running) > Thread [pool-1-thread-1] (Running) > Thread [Thread-11] (Running) > Thread [DestroyJavaVM] (Running). > As a result, the NodeManager keeps running even though no services are > started. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3007) JobClient cannot talk to JobHistory server in secure mode
[
https://issues.apache.org/jira/browse/MAPREDUCE-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3007:
---
Status: Open (was: Patch Available)
This patch is buggy and doesn't work, [~karams] reports.
> JobClient cannot talk to JobHistory server in secure mode
> -
>
> Key: MAPREDUCE-3007
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3007
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3007-20110914.txt
>
>
> In secure mode, Jobclient cannot connect to HistoryServer. Thanks to
> [~karams] for finding this out.
> {code}
> 11/09/14 09:57:51 INFO mapred.ClientServiceDelegate: Application state is
> completed. Redirecting to job history server
> 11/09/14 09:57:51 INFO security.ApplicationTokenSelector: Looking for a token
> with service :10020
> 11/09/14 09:57:51 INFO security.ApplicationTokenSelector: Token kind is
> YARN_APPLICATION_TOKEN and the token's service name is :46257
> 11/09/14 09:57:51 INFO security.UserGroupInformation: Initiating logout for
>
> 11/09/14 09:57:51 INFO security.UserGroupInformation: Initiating re-login for
>
> 11/09/14 09:57:55 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:57:56 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:00 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:05 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 11/09/14 09:58:05 WARN ipc.Client: Couldn't setup connection for
> to null
> 11/09/14 09:58:05 INFO mapred.ClientServiceDelegate: Failed to contact
> AM/History for job job_1315993268700_0001 Will retry..
> {code}
> Am surprised no one working with YARN+MR ever ran into this!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
