[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-3999:
Attachment: MAPREDUCE-3999.patch
Removing a stray logging statement =D
> Tracking link gives an error if the AppMaster hasn't started yet
>
>
> Key: MAPREDUCE-3999
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, webapps
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3999.patch, MAPREDUCE-3999.patch
>
>
> Courtesy [~sseth]
> {quote}
> "The MRAppMaster died before writing anything."
> Steps to generate the error:
> 1. Setup a queue with 1 max active application per user
> 2. Submit a long running job to this queue.
> 3. Submit another job to the queue as the same user. Access the tracking URL
> for job 2 directly or via Oozie (not via the RM link - which is rewritten once
> the app starts).
> This would exist in situations where the queue doesn't have enough capacity -
> or for the small period of time between app submission and AM start.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-3999:
Status: Open (was: Patch Available)
> Tracking link gives an error if the AppMaster hasn't started yet
>
>
> Key: MAPREDUCE-3999
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, webapps
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3999.patch
>
>
> Courtesy [~sseth]
> {quote}
> "The MRAppMaster died before writing anything."
> Steps to generate the error:
> 1. Setup a queue with 1 max active application per user
> 2. Submit a long running job to this queue.
> 3. Submit another job to the queue as the same user. Access the tracking URL
> for job 2 directly or via Oozie (not via the RM link - which is rewritten once
> the app starts).
> This would exist in situations where the queue doesn't have enough capacity -
> or for the small period of time between app submission and AM start.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response
[ https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-3992: --- Target Version/s: 0.24.0, 1.0.2, 0.23.3 (was: 1.0.2) Affects Version/s: 0.24.0 0.23.1 Status: Patch Available (was: Open) > Reduce fetcher doesn't verify HTTP status code of response > -- > > Key: MAPREDUCE-3992 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.0.1, 0.23.1, 0.24.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: mr-3992.txt > > > Currently, the reduce fetch code doesn't check the HTTP status code of the > response. This can lead to the following situation: > - the map output servlet gets an IOException after setting the headers but > before the first call to flush() > - this causes it to send a response with a non-OK result code, including the > exception text as the response body (response.sendError() does this if the > response isn't committed) > - it will still include the response headers indicating it's a valid response > In the case of a merge-to-memory, the compression codec might then try to > interpret the HTML response as compressed data, resulting in either a huge > allocation (OOME) or some other nasty error. This bug seems to be present in > MR1, but haven't checked trunk/MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response
[ https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-3992: --- Attachment: mr-3992.txt Simple patch against trunk. I haven't tried this on an MR2 cluster, but I tested the equivalent 0.20-based patch. Unfortunately it's difficult to engineer a unit test case for this. I'll upload a patch for branch-1 as well if this is accepted. > Reduce fetcher doesn't verify HTTP status code of response > -- > > Key: MAPREDUCE-3992 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 0.23.1, 0.24.0, 1.0.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: mr-3992.txt > > > Currently, the reduce fetch code doesn't check the HTTP status code of the > response. This can lead to the following situation: > - the map output servlet gets an IOException after setting the headers but > before the first call to flush() > - this causes it to send a response with a non-OK result code, including the > exception text as the response body (response.sendError() does this if the > response isn't committed) > - it will still include the response headers indicating it's a valid response > In the case of a merge-to-memory, the compression codec might then try to > interpret the HTML response as compressed data, resulting in either a huge > allocation (OOME) or some other nasty error. This bug seems to be present in > MR1, but haven't checked trunk/MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-3999:
Status: Patch Available (was: Open)
> Tracking link gives an error if the AppMaster hasn't started yet
>
>
> Key: MAPREDUCE-3999
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, webapps
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3999.patch
>
>
> Courtesy [~sseth]
> {quote}
> "The MRAppMaster died before writing anything."
> Steps to generate the error:
> 1. Setup a queue with 1 max active application per user
> 2. Submit a long running job to this queue.
> 3. Submit another job to the queue as the same user. Access the tracking URL
> for job 2 directly or via Oozie (not via the RM link - which is rewritten once
> the app starts).
> This would exist in situations where the queue doesn't have enough capacity -
> or for the small period of time between app submission and AM start.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-3999:
Attachment: MAPREDUCE-3999.patch
This fixes the issue. Now I am checking the FinalApplicationStatus too to
figure out if the AM has completed / hasn't started
> Tracking link gives an error if the AppMaster hasn't started yet
>
>
> Key: MAPREDUCE-3999
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, webapps
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3999.patch
>
>
> Courtesy [~sseth]
> {quote}
> "The MRAppMaster died before writing anything."
> Steps to generate the error:
> 1. Setup a queue with 1 max active application per user
> 2. Submit a long running job to this queue.
> 3. Submit another job to the queue as the same user. Access the tracking URL
> for job 2 directly or via Oozie (not via the RM link - which is rewritten once
> the app starts).
> This would exist in situations where the queue doesn't have enough capacity -
> or for the small period of time between app submission and AM start.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3974: - Resolution: Fixed Fix Version/s: (was: 0.24.0) 0.23.3 Status: Resolved (was: Patch Available) I committed this to branch-0.23 > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3974.patch > > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3682) Tracker URL says AM tasks run on localhost
[ https://issues.apache.org/jira/browse/MAPREDUCE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3682: Attachment: MAPREDUCE-3682.patch Using NetUtils.getHostname() which returns "/", and then splitting using / to get "" Simple enough change. Not including a test because its simple enough. > Tracker URL says AM tasks run on localhost > -- > > Key: MAPREDUCE-3682 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3682 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: David Capwell >Assignee: Hitesh Shah > Attachments: MAPREDUCE-3682.patch > > > If you look at the task page, it will show you the node the task ran on. For > jobs that run in UberAM they point to http://localhost: and logs points > to http://localhost:/node/containerlogs/$container_id/ > This was run on a multi node cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3971: --- Summary: Job History web services need to have limits on the number of items they can return. (was: Job History web services need to have limits on the number of itmes they can return.) > Job History web services need to have limits on the number of items they can > return. > > > Key: MAPREDUCE-3971 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > > The Job History web services canput a very large load on the job history > server. We should put in a limit on the number of entries that can be > returned by the web service, and also add in the ability to modify the > starting location in the list, so that all entries can still be downlaoded. > Just not all at once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3996) zookeeper artifact is missing from the hadoop-dist assembly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-3996: Status: Patch Available (was: Open) > zookeeper artifact is missing from the hadoop-dist assembly > --- > > Key: MAPREDUCE-3996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3996 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3996.patch.txt > > > According to maven, zookeeper happens to be a dependency of > hadoop-yarn-server-common. Yet it is missing from the final distribution > assembly (and hence from the binary tarball) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3996) zookeeper artifact is missing from the hadoop-dist assembly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-3996: Attachment: MAPREDUCE-3996.patch.txt The alternative is to depend on hadoop-yarn-server-common which seems to be an overkill > zookeeper artifact is missing from the hadoop-dist assembly > --- > > Key: MAPREDUCE-3996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3996 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3996.patch.txt > > > According to maven, zookeeper happens to be a dependency of > hadoop-yarn-server-common. Yet it is missing from the final distribution > assembly (and hence from the binary tarball) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError
[
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3869:
-
Status: Patch Available (was: Open)
Thanks Hitesh and Vinod for looking into the patch.
I'm sorry for the late response.
I have updated the patch with the suggested change.
> Distributed shell application fails with NoClassDefFoundError
> -
>
> Key: MAPREDUCE-3869
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869.patch
>
>
> Distributed shell application always fails to start the application master
> with the following error.
> \\
> {code:xml}
> 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running
> ApplicationMaster
> java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.(ApplicationMaster.java:252)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.ipc.YarnRPC
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
> ... 2 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError
[
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3869:
-
Attachment: MAPREDUCE-3869-1.patch
> Distributed shell application fails with NoClassDefFoundError
> -
>
> Key: MAPREDUCE-3869
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869.patch
>
>
> Distributed shell application always fails to start the application master
> with the following error.
> \\
> {code:xml}
> 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running
> ApplicationMaster
> java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.(ApplicationMaster.java:252)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.ipc.YarnRPC
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
> ... 2 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given
[ https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3829: Release Note: Makes Gridmix emit out correct error message when the input data directory already exists and -generate option is used. Makes Gridmix exit with proper exit codes when Gridmix fails in args-processing, startup/setup. (was: Makes Gridmix emit out correct error message when the input data directory already exists and -generate option is used. Makes Gridmix exit with proper exit codes when Gridmix fails in startup/setup.) Hadoop Flags: Reviewed Status: Patch Available (was: Open) > [Gridmix] Gridmix should give better error message when input-data directory > already exists and -generate option is given > - > > Key: MAPREDUCE-3829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, > 3829.v2.patch > > > Instead of throwing exception messages on to the console, Gridmix should give > better error message when input-data directory already exists and -generate > option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given
[ https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3829: Attachment: 3829.v2.patch Attaching new patch with minor code changes and more testcases added. > [Gridmix] Gridmix should give better error message when input-data directory > already exists and -generate option is given > - > > Key: MAPREDUCE-3829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, > 3829.v2.patch > > > Instead of throwing exception messages on to the console, Gridmix should give > better error message when input-data directory already exists and -generate > option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given
[ https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3829: Status: Open (was: Patch Available) > [Gridmix] Gridmix should give better error message when input-data directory > already exists and -generate option is given > - > > Key: MAPREDUCE-3829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch > > > Instead of throwing exception messages on to the console, Gridmix should give > better error message when input-data directory already exists and -generate > option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Status: Patch Available (was: Open)
Patch with test. I'm submitting to see what Jenkins says, but I've realised
that adding the listeners to {{AbstractService}} isn't the right place to add a
list of static listeners.
It would be better to move the code to the {{ServiceOperations}} class of
MAPREDUCE-3970, and then invoke it from {{AbstractService}}. Any implementation
of {{Service}} that did not extend that base class would still have the ability
to invoke the event list.
Other enhancements
# clone the list before making the notifications to make the iterator robust
against concurrency problems.
# add a package-scoped {{resetGlobalListeners()}} method for testing, to ensure
the list is empty after each test run.
Such a patch will have to wait for MAPREDUCE-3970, so I'll mark this as a
dependency.
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch,
> MAPREDUCE-3995.patch, MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Status: Open (was: Patch Available)
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch,
> MAPREDUCE-3995.patch, MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Attachment: MAPREDUCE-3995.patch
latest diff
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch,
> MAPREDUCE-3995.patch, MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1109) ConcurrentModificationException in jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1109: --- Attachment: MAPREDUCE-1109.patch A very small subset of the patch at MAPREDUCE-679, which only changes the JSP's job list calls to their synchronized counterparts. Issue is already fixed in 0.22. > ConcurrentModificationException in jobtracker.jsp > - > > Key: MAPREDUCE-1109 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1109 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 1.0.0 > > Attachments: MAPREDUCE-1109.patch > > > The jobtracker.jsp invoked methods tracker.runningJobs(), > tracker.completedJobs() and tracker.failedJobs() but these methods are not > synchronized and can cause ConcurrentModificationException if the JT is > concurrently changing the contents of these datastructures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1109) ConcurrentModificationException in jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1109: --- Target Version/s: 1.1.0 Fix Version/s: (was: 1.0.0) > ConcurrentModificationException in jobtracker.jsp > - > > Key: MAPREDUCE-1109 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1109 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: MAPREDUCE-1109.patch > > > The jobtracker.jsp invoked methods tracker.runningJobs(), > tracker.completedJobs() and tracker.failedJobs() but these methods are not > synchronized and can cause ConcurrentModificationException if the JT is > concurrently changing the contents of these datastructures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1109) ConcurrentModificationException in jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1109: --- Status: Patch Available (was: Reopened) > ConcurrentModificationException in jobtracker.jsp > - > > Key: MAPREDUCE-1109 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1109 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: MAPREDUCE-1109.patch > > > The jobtracker.jsp invoked methods tracker.runningJobs(), > tracker.completedJobs() and tracker.failedJobs() but these methods are not > synchronized and can cause ConcurrentModificationException if the JT is > concurrently changing the contents of these datastructures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Status: Patch Available (was: Open)
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch,
> MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Attachment: MAPREDUCE-3995.patch
# unregistering a global listener returns true iff the listener was registered.
Needed for testing.
# adds one useful listener that logs the events via commons logging.
# adds a listener designed to count the number of events received, cache the
last one, and can be set to fail on any specific state change event.
# add tests to verify listener register/unregister, event callback, and explore
the behaviour of listeners when listeners themselves fail.
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch,
> MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Status: Open (was: Patch Available)
patch off while the next stage goes in:
# useful listener -one that logs.
# test listener -one that caches event history and can be set to fail on a
specific state event; put this in test/ .
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Attachment: MAPREDUCE-3921-branch-0.23.patch > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-branch-0.23.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-764) TypedBytesInput's readRaw() does not preserve custom type codes
[
https://issues.apache.org/jira/browse/MAPREDUCE-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-764:
Affects Version/s: 1.0.2
Fix Version/s: 1.0.2
Thanks Matt. I've committed this to branch-1 after running all tests.
> TypedBytesInput's readRaw() does not preserve custom type codes
> ---
>
> Key: MAPREDUCE-764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-764
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
>Affects Versions: 0.21.0, 1.0.2
>Reporter: Klaas Bosteels
>Assignee: Klaas Bosteels
>Priority: Blocker
> Fix For: 0.21.0, 1.0.2
>
> Attachments: MAPREDUCE-764.patch, MAPREDUCE-764.patch
>
>
> The typed bytes format supports byte sequences of the form {{ code> }}. When reading such a sequence via
> {{TypedBytesInput}}'s {{readRaw()}} method, however, the returned sequence
> currently is {{0 }} (0 is the type code for a bytes array),
> which leads to bugs such as the one described
> [here|http://dumbo.assembla.com/spaces/dumbo/tickets/54].
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3773: - Resolution: Fixed Fix Version/s: 1.0.2 0.23.2 Target Version/s: (was: 1.1.0, 0.23.1) Status: Resolved (was: Patch Available) I just committed this. Thanks Owen! > Add queue metrics with buckets for job run times > > > Key: MAPREDUCE-3773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.23.2, 1.0.2 > > Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch > > > It would be nice to have queue metrics that reflect the number of jobs in > each queue that have been running for different ranges of time. > Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but > they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3650) testGetTokensForHftpFS() fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3650: Target Version/s: 0.23.0, 0.23.2 (was: 0.23.0) Can someone please review and commit to 0.23.2? > testGetTokensForHftpFS() fails > -- > > Key: MAPREDUCE-3650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3650 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Ravi Prakash >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3650.patch > > > org.apache.hadoop.mapreduce.security.TestTokenCache.testGetTokensForHftpFS > fails. > Looks like it may have been introduced with HADOOP-7808 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3991: --- Resolution: Fixed Fix Version/s: 0.23.3 0.24.0 Target Version/s: (was: 0.23.3, 0.24.0) Status: Resolved (was: Patch Available) Committed to branch-0.23 and trunk. > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Fix For: 0.24.0, 0.23.3 > > Attachments: MAPREDUCE-3991.patch, MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3982: --- Resolution: Fixed Fix Version/s: 0.23.2 Release Note: Fixed FileOutputCommitter to not err out for an 'empty-job' whose tasks don't write any outputs. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to trunk, branch-0.23 and branch-0.23.2. Thanks Robert! > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3982.txt, MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3991: --- Attachment: MAPREDUCE-3991.patch Bad filename for earlier attachment. > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-3991.patch, MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3991: --- Attachment: (was: MAPREDUCE-3391.patch) > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-3991.patch, MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3970) Add ServiceOperations class to aid working with Services
[ https://issues.apache.org/jira/browse/MAPREDUCE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3970: -- Attachment: MAPREDUCE-3970.patch javadocs corrected. > Add ServiceOperations class to aid working with Services > > > Key: MAPREDUCE-3970 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3970 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: MAPREDUCE-3970.patch, MAPREDUCE-3970.patch > > > Add Helper methods to move things through lifecycles. init->start is common, > stop-if-service!=null another. Some static methods can execute these, and > even call stop() if init() raises an exception. These could go into a class > ServiceOps in the same package. These can be used by those services that wrap > other services, and help manage more robust shutdowns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3970) Add ServiceOperations class to aid working with Services
[ https://issues.apache.org/jira/browse/MAPREDUCE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3970: -- Status: Patch Available (was: Open) > Add ServiceOperations class to aid working with Services > > > Key: MAPREDUCE-3970 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3970 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: MAPREDUCE-3970.patch, MAPREDUCE-3970.patch > > > Add Helper methods to move things through lifecycles. init->start is common, > stop-if-service!=null another. Some static methods can execute these, and > even call stop() if init() raises an exception. These could go into a class > ServiceOps in the same package. These can be used by those services that wrap > other services, and help manage more robust shutdowns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3970) Add ServiceOperations class to aid working with Services
[ https://issues.apache.org/jira/browse/MAPREDUCE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3970: -- Status: Open (was: Patch Available) > Add ServiceOperations class to aid working with Services > > > Key: MAPREDUCE-3970 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3970 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: MAPREDUCE-3970.patch, MAPREDUCE-3970.patch > > > Add Helper methods to move things through lifecycles. init->start is common, > stop-if-service!=null another. Some static methods can execute these, and > even call stop() if init() raises an exception. These could go into a class > ServiceOps in the same package. These can be used by those services that wrap > other services, and help manage more robust shutdowns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3991: --- Attachment: MAPREDUCE-3391.patch Made the doc a little less ambiguous based on the suggestions from a user. > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-3391.patch, MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Status: Patch Available (was: Open)
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Attachment: MAPREDUCE-3995.patch
proper diff
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Status: Open (was: Patch Available)
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3650) testGetTokensForHftpFS() fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3650: Attachment: MAPREDUCE-3650.patch Embarassingly small patch. The test was tripping on a hostname lookup. So I just exchanged the hostname with an IP address =D > testGetTokensForHftpFS() fails > -- > > Key: MAPREDUCE-3650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3650 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Ravi Prakash >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3650.patch > > > org.apache.hadoop.mapreduce.security.TestTokenCache.testGetTokensForHftpFS > fails. > Looks like it may have been introduced with HADOOP-7808 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3650) testGetTokensForHftpFS() fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3650: Status: Patch Available (was: Open) > testGetTokensForHftpFS() fails > -- > > Key: MAPREDUCE-3650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3650 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Ravi Prakash >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3650.patch > > > org.apache.hadoop.mapreduce.security.TestTokenCache.testGetTokensForHftpFS > fails. > Looks like it may have been introduced with HADOOP-7808 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3348) mapred job -status fails to give info even if the job is present in History
[
https://issues.apache.org/jira/browse/MAPREDUCE-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3348:
---
Resolution: Fixed
Release Note: Fixed a bug in MR client to redirect to JobHistoryServer
correctly when RM forgets the app.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Just committed this to branch-0.23 and trunk, Thanks Devaraj K!
Will merge into 0.23.2 later.
> mapred job -status fails to give info even if the job is present in History
> ---
>
> Key: MAPREDUCE-3348
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3348
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3348-1.patch, MAPREDUCE-3348.patch
>
>
> It is trying to get the app report from the RM for the job, RM throws
> exception when it doesn't find and then it is giving the same exception
> without trying from History Server.
> {code}
> 11/11/03 08:47:27 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.mapred
>uce.v2.api.MRClientProtocol
> 11/11/03 08:47:28 WARN mapred.ClientServiceDelegate: Exception thrown by
> remote end.
> RemoteTrace:
> at LocalTrace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> Trying to get information for an absent applicat
> ion
> application_1320278804241_0002
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:142)
> at $Proxy6.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClie
>
> ntImpl.java:111)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:321)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:137)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:273)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:353)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:429)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:240)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1106)
> Exception in thread "main" RemoteTrace:
> at Local Trace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> Trying to get information for an absent applicat
> ion
> application_1320278804241_0002
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:142)
> at $Proxy6.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClie
>
> ntImpl.java:111)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:321)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:137)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:273)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:353)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:429)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:240)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83
[jira] [Updated] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-3989: --- Target Version/s: 0.23.3 > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Attachment: MAPREDUCE-3995.patch
First pass for review -no tests.
This is designed to be as simple as possible -the same service listener is used
for the "global" listeners, and the notifications are blocking on state
changes. I started to write something asynchronous but it was over-complex,
hard to test and could easily leak threads.
The global listener list is synchronised on the list itself, both for changes
and for the list run through itself. Listeners must not attempt to unregister
themselves during a callback or the iterator will fail.
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3729) Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, TestLocalModeWithNewApis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3729: --- Priority: Critical (was: Blocker) Dropping priority because Giri fixed the current test failures. Still looking into code changes so the failures do not come back again. > Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, > TestLocalModeWithNewApis > -- > > Key: MAPREDUCE-3729 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3729 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.1 > > > See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1671//testReport/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .
[
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3995:
--
Status: Patch Available (was: Open)
> Add support for static service lifecycle listeners .
> -
>
> Key: MAPREDUCE-3995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-3995.patch
>
>
> Add support to {{AbstractService}} that allow callers to register listeners
> for all instances. The existing listener interface could be used. This allows
> management tools to hook into the events.
> The static listeners would be invoked for all state changes except creation
> (base class shouldn't be handing out references to itself at this point).
> These static events could all be async, pushed through a shared
> {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the
> listeners invoked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3578) starting nodemanager as 'root' gives "Unknown -jvm option"
[
https://issues.apache.org/jira/browse/MAPREDUCE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated MAPREDUCE-3578:
-
Resolution: Fixed
Fix Version/s: 0.23.3
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this.
> starting nodemanager as 'root' gives "Unknown -jvm option"
> --
>
> Key: MAPREDUCE-3578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3578
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.0
> Environment: jdk 1.6.0.26, ubuntu 11.04
>Reporter: Gilad Wolff
>Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: MAPREDUCE-3578.patch
>
>
> running "sudo HADOOP_ROOT/bin/yarn-daemon.sh start nodemanager" I get
> "unknown -jvm option" (jdk version 1.6.0.26). The problem seems to be with
> line 204 in yarn:
> elif [ "$COMMAND" = "nodemanager" ] ; then
> CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
> CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
> if [[ $EUID -eq 0 ]]; then
> YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS"
> else
> YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
> fi
> using -server seems to solve the problem for me.
> I tested using build 929 from
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3970) Add ServiceOperations class to aid working with Services
[
https://issues.apache.org/jira/browse/MAPREDUCE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-3970:
--
Attachment: MAPREDUCE-3970.patch
* {{ServiceOperations}} class to start/stop services with preflight checking of
initial state. This is how callers should start and stop services -including
{{Service}} implementations that encapsulate other services.
* Tests, including some more in {{TestServiceLifecycle}}. That test formalises
the current behaviour of service operations (all subclass operations called
first).
> Add ServiceOperations class to aid working with Services
>
>
> Key: MAPREDUCE-3970
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3970
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: MAPREDUCE-3970.patch
>
>
> Add Helper methods to move things through lifecycles. init->start is common,
> stop-if-service!=null another. Some static methods can execute these, and
> even call stop() if init() raises an exception. These could go into a class
> ServiceOps in the same package. These can be used by those services that wrap
> other services, and help manage more robust shutdowns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3970) Add ServiceOperations class to aid working with Services
[ https://issues.apache.org/jira/browse/MAPREDUCE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3970: -- Status: Patch Available (was: In Progress) > Add ServiceOperations class to aid working with Services > > > Key: MAPREDUCE-3970 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3970 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: MAPREDUCE-3970.patch > > > Add Helper methods to move things through lifecycles. init->start is common, > stop-if-service!=null another. Some static methods can execute these, and > even call stop() if init() raises an exception. These could go into a class > ServiceOps in the same package. These can be used by those services that wrap > other services, and help manage more robust shutdowns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3982: --- Status: Patch Available (was: Open) > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt, MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3982: --- Attachment: MR-3982.txt This patch adds in a unit test that verifies that it still works even without any output. I started to port TestEmptyJob to the MiniMrYarnCluster, but it uses a lot of methods that are specific to the JT implementation to do some sort of synchronization test. I really don't understand all of the reasons for that test and why it is trying to force the scheduler to schedule something. I will file a separate JIRA to port it over because it is probably going to take a while for me to fully understand the test. > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt, MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3348) mapred job -status fails to give info even if the job is present in History
[
https://issues.apache.org/jira/browse/MAPREDUCE-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3348:
-
Status: Patch Available (was: Open)
Thanks Vinod for taking a look into the patch. I have addressed the comments in
the updated patch.
I'm sorry for the late response.
> mapred job -status fails to give info even if the job is present in History
> ---
>
> Key: MAPREDUCE-3348
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3348
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3348-1.patch, MAPREDUCE-3348.patch
>
>
> It is trying to get the app report from the RM for the job, RM throws
> exception when it doesn't find and then it is giving the same exception
> without trying from History Server.
> {code}
> 11/11/03 08:47:27 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.mapred
>uce.v2.api.MRClientProtocol
> 11/11/03 08:47:28 WARN mapred.ClientServiceDelegate: Exception thrown by
> remote end.
> RemoteTrace:
> at LocalTrace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> Trying to get information for an absent applicat
> ion
> application_1320278804241_0002
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:142)
> at $Proxy6.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClie
>
> ntImpl.java:111)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:321)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:137)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:273)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:353)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:429)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:240)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1106)
> Exception in thread "main" RemoteTrace:
> at Local Trace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> Trying to get information for an absent applicat
> ion
> application_1320278804241_0002
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:142)
> at $Proxy6.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClie
>
> ntImpl.java:111)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:321)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:137)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:273)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:353)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:429)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:240)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1106)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3348) mapred job -status fails to give info even if the job is present in History
[
https://issues.apache.org/jira/browse/MAPREDUCE-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3348:
-
Attachment: MAPREDUCE-3348-1.patch
> mapred job -status fails to give info even if the job is present in History
> ---
>
> Key: MAPREDUCE-3348
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3348
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3348-1.patch, MAPREDUCE-3348.patch
>
>
> It is trying to get the app report from the RM for the job, RM throws
> exception when it doesn't find and then it is giving the same exception
> without trying from History Server.
> {code}
> 11/11/03 08:47:27 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.mapred
>uce.v2.api.MRClientProtocol
> 11/11/03 08:47:28 WARN mapred.ClientServiceDelegate: Exception thrown by
> remote end.
> RemoteTrace:
> at LocalTrace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> Trying to get information for an absent applicat
> ion
> application_1320278804241_0002
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:142)
> at $Proxy6.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClie
>
> ntImpl.java:111)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:321)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:137)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:273)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:353)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:429)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:240)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1106)
> Exception in thread "main" RemoteTrace:
> at Local Trace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> Trying to get information for an absent applicat
> ion
> application_1320278804241_0002
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:142)
> at $Proxy6.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClie
>
> ntImpl.java:111)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:321)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:137)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:273)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:353)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:429)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:240)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1106)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3991: --- Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Patch Available (was: Open) > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3991: --- Attachment: MAPREDUCE-3991.patch > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3990) MRBench allows Long-sized input-lines value but parses CLI argument as an Integer
[
https://issues.apache.org/jira/browse/MAPREDUCE-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3990:
---
Target Version/s: 0.24.0, 0.23.3 (was: 0.24.0)
Status: Patch Available (was: Open)
> MRBench allows Long-sized input-lines value but parses CLI argument as an
> Integer
> -
>
> Key: MAPREDUCE-3990
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3990
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.23.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: MAPREDUCE-3990.patch
>
>
> MRBench has the following method:
> {code}
> public void generateTextFile(FileSystem fs, Path inputFile, long numLines,
> Order sortOrder) { ... }
> {code}
> The method is already set to accept a long datatype for numLines, for
> generating very large amount of data.
> However, in {{MRBench#run(...)}}, the inputLines CLI parameter is parsed via
> an Integer.parseInt, causing numbers passed > Integer.MAX_VALUE to throw
> NumberFormatExceptions as a result.
> The parsing should be Long.parseLong and the inputLines datatype should be
> switched to the same type as passed to the method (long).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3990) MRBench allows Long-sized input-lines value but parses CLI argument as an Integer
[
https://issues.apache.org/jira/browse/MAPREDUCE-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3990:
---
Attachment: MAPREDUCE-3990.patch
> MRBench allows Long-sized input-lines value but parses CLI argument as an
> Integer
> -
>
> Key: MAPREDUCE-3990
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3990
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.23.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: MAPREDUCE-3990.patch
>
>
> MRBench has the following method:
> {code}
> public void generateTextFile(FileSystem fs, Path inputFile, long numLines,
> Order sortOrder) { ... }
> {code}
> The method is already set to accept a long datatype for numLines, for
> generating very large amount of data.
> However, in {{MRBench#run(...)}}, the inputLines CLI parameter is parsed via
> an Integer.parseInt, causing numbers passed > Integer.MAX_VALUE to throw
> NumberFormatExceptions as a result.
> The parsing should be Long.parseLong and the inputLines datatype should be
> switched to the same type as passed to the method (long).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated MAPREDUCE-3989: Status: Patch Available (was: Open) > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated MAPREDUCE-3989: Attachment: MAPREDUCE-3989.patch Updated patch with mr only in this patch. I re-verified by running from tarball with DEBUG turned on and the maxfilesize set to just 10k. > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-1740: Attachment: MAPREDUCE-1740_branch-1.0.patch Thanks Steve and Todd for the review! I am attaching an updated patch for branch-1.0 incorporating Steve's style comments. The patch mainly is to avoid the NPE that we are seeing with some users when the compared nodes are at different depth. > NPE in getMatchingLevelForNodes when node locations are variable depth > -- > > Key: MAPREDUCE-1740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.3 >Reporter: Todd Lipcon > Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, > mapreduce-1740.txt > > > In getMatchingLevelForNodes, we assume that both nodes have the same "depth" > (ie number of path components). If the user provides a topology script that > assigns one node a path like /foo/bar/baz and another node a path like > /foo/blah, this function will throw an NPE. > I'm not sure if there are other places where we assume that all node > locations have a constant number of paths. If so we should check the output > of the topology script aggressively to be sure this is the case. Otherwise I > think we simply need to add && n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3940: --- Status: Patch Available (was: Open) > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3940: --- Attachment: MAPREDUCE-3940-20120308.txt Attaching patch that adds expiry time to container-token-ids and makes NM reject containers with expired tokens. Added a unit test to exactly validate the above. > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Status: Open (was: Patch Available) > Scheduler.normalizeRequest does not account for allocation requests that > exceed maximumAllocation limits > - > > Key: MAPREDUCE-3796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, > MR-3796.wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Attachment: MR-3796.3.patch Made min/max settings global at the RM level instead of the per-scheduler settings. > Scheduler.normalizeRequest does not account for allocation requests that > exceed maximumAllocation limits > - > > Key: MAPREDUCE-3796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, > MR-3796.wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Status: Patch Available (was: Open) > Scheduler.normalizeRequest does not account for allocation requests that > exceed maximumAllocation limits > - > > Key: MAPREDUCE-3796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, > MR-3796.wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Hadoop Flags: Incompatible change > Scheduler.normalizeRequest does not account for allocation requests that > exceed maximumAllocation limits > - > > Key: MAPREDUCE-3796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, > MR-3796.wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3982: --- Status: Open (was: Patch Available) Patch looks good. Can you please port the test TestEmptyJob to maven build and possibly MiniMRYarnCluster? Tx. > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3942: --- Status: Patch Available (was: Open) > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3942: --- Attachment: MAPREDUCE-3942-20120307.1.txt Fixing test failure. > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3942: --- Status: Open (was: Patch Available) > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3982: --- Priority: Critical (was: Major) > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3982: --- Attachment: MR-3982.txt This fixes the issue by createing the directory earlier instead of relying on the mapper or reducer to do it. > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3982: --- Status: Patch Available (was: Open) > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3942: --- Attachment: MAPREDUCE-3942-20120307.txt Attached patch implements what is discussed in the description, nothing more nothing less. Added a test to validate that token are invalidated after app-finish and that tokens continue to work even after the master-key roll-over. > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3942: --- Status: Patch Available (was: Open) > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3975: --- Resolution: Fixed Fix Version/s: 0.23.2 Status: Resolved (was: Patch Available) Thanks Eric, I just put this in trunk 0.23 and 0.23.2. > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3977) LogAggregationService leaks log aggregator objects
[
https://issues.apache.org/jira/browse/MAPREDUCE-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3977:
---
Resolution: Fixed
Fix Version/s: 0.23.2
Target Version/s: 0.24.0, 0.23.2, 0.23.3 (was: 0.23.3, 0.23.2, 0.24.0)
Status: Resolved (was: Patch Available)
Thanks Jason, I just put this into trunk, branch-0.23 and branch-0.23.2
> LogAggregationService leaks log aggregator objects
> --
>
> Key: MAPREDUCE-3977
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3977
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
>Affects Versions: 0.23.1
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3977.patch
>
>
> LogAggregationService adds log aggregator objects to the
> {{appLogAggregators}} map but never removes them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3961: --- Resolution: Fixed Fix Version/s: 0.23.2 Status: Resolved (was: Patch Available) Thanks Sid, I just put this into trunk, 0.23 and 0.23.2 > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 0.23.2 > > Attachments: MR3961.txt, MR3961v2.txt, MR3961v3.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[
https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated MAPREDUCE-3975:
--
Attachment: MAPREDUCE-3975-1.txt
@Arun,
I modified YarnChild.configureLocalDirs(...) to mimic what
JobLocalizer.createWorkDir(...) did in 20.205. Namely, to use the
LocalDirAllocator to get a local path within one of the
"mapreduce.cluster.local.dir" locations and create a scratch directory under it.
TESTING:
On a 10-node secure cluster, I ran manual tests to make sure that the
"mapreduce.job.local.dir" was being set and that the directory was being
created.
My manual tests printed the value of "job.local.dir" which showed up in the
task logs. I also made sure that the scratch directories existed and were
writeable by the user.
The 20.205 value for "job.local.dir" looked something like this:
${mapred.local.dir[x]}/taskTracker/$user/jobcache/$jobid/work
e.g.: /cluster/0/tmp/mapred-local/taskTracker/joe/jobcache/job_1234_0001/work
The 23 value for "mapreduce.job.local.dir" looks something like this:
${mapreduce.cluster.local.dir[x]}/usercache/$user/appcache/$appid/work
e.g.:
/cluster/2/tmp/mapred-local/usercache/joe/appcache/application_5678_0002/work
CONCERN:
This solution works as far as it goes. However, I do have one concern.
In 20.205, it appears that the "job.local.dir" has the same parent dir (e.g.
/cluster/0) for all of the task attempts that are run on a specific node.
However, this is not the case in 23. That is, on 23, even if two task attempts
are run on the same node for application_5678_0002, they could have different
root directories (e.g., one task attempt would have /cluster/2 for its root dir
and the other would have /cluster/4).
Is this expected behavior, or is YarnChild the wrong place to set this value?
> Default value not set for Configuration parameter mapreduce.job.local.dir
> -
>
> Key: MAPREDUCE-3975
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Blocker
> Attachments: MAPREDUCE-3975-1.txt
>
>
> mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by
> default. This is a regression from 0.20.205.
> In 0.20.205, JobLocalizer.createWorkDir() constructs the
> "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on
> $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's
> JobConf.
> So far, I haven't found where this is done in 0.23. It could be that this is
> what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am
> still investigating.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3977) LogAggregationService leaks log aggregator objects
[
https://issues.apache.org/jira/browse/MAPREDUCE-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-3977:
--
Target Version/s: 0.24.0, 0.23.2, 0.23.3
Affects Version/s: 0.23.1
Status: Patch Available (was: Open)
> LogAggregationService leaks log aggregator objects
> --
>
> Key: MAPREDUCE-3977
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3977
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
>Affects Versions: 0.23.1
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: MAPREDUCE-3977.patch
>
>
> LogAggregationService adds log aggregator objects to the
> {{appLogAggregators}} map but never removes them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3975: -- Status: Patch Available (was: Open) > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3961: -- Attachment: MR3961v3.txt > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt, MR3961v2.txt, MR3961v3.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3977) LogAggregationService leaks log aggregator objects
[
https://issues.apache.org/jira/browse/MAPREDUCE-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-3977:
--
Attachment: MAPREDUCE-3977.patch
Patch to wrap the log aggregator run() method so we remove the entry from the
map when the method completes.
> LogAggregationService leaks log aggregator objects
> --
>
> Key: MAPREDUCE-3977
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3977
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: MAPREDUCE-3977.patch
>
>
> LogAggregationService adds log aggregator objects to the
> {{appLogAggregators}} map but never removes them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3961: -- Status: Patch Available (was: Open) > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt, MR3961v2.txt, MR3961v3.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3961: --- Status: Open (was: Patch Available) The patch is stale, and needs to be upmerged. > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt, MR3961v2.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3978) getTotalMaps, getTotalReduces is incorrect for JobHistory PartialJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3978: -- Component/s: (was: mrv1) (was: job submission) mrv2 jobhistoryserver Affects Version/s: (was: 0.23.1) 0.23.0 > getTotalMaps, getTotalReduces is incorrect for JobHistory PartialJob > > > Key: MAPREDUCE-3978 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3978 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth > > These are currently returning the number of completed map / reduce tasks - > instead of the total. The information doesn't make it over to the history > server via the job history file name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-3944:
--
Resolution: Fixed
Fix Version/s: 0.23.2
Target Version/s: 0.24.0, 0.23.2 (was: 0.23.2, 0.24.0)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk, branch-0.23 and branch-0.23.2
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-3974: -- Fix Version/s: 0.24.0 Offline Arun informed me that I do indeed have commit privs for MR. So, I've just committed this to trunk. It doesn't make sense to commit this to branch-0.23 until HDFS-1623 and friends are back-ported to branch-0.23, so leaving this JIRA open. > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > Fix For: 0.24.0 > > Attachments: MAPREDUCE-3974.patch > > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Status: Patch Available (was: Open) > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Status: Open (was: Patch Available) > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3944:
---
Target Version/s: 0.24.0, 0.23.2 (was: 0.23.2, 0.24.0)
Status: Patch Available (was: Open)
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3944:
---
Attachment: MR-3944.txt
Added in the new file
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3944:
---
Target Version/s: 0.24.0, 0.23.2 (was: 0.23.2, 0.24.0)
Status: Open (was: Patch Available)
You are correct that I missed it. Sorry about that.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3976) TestRMContainerAllocator failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3976: --- Resolution: Fixed Fix Version/s: 0.23.2 Target Version/s: 0.24.0, 0.23.2, 0.23.3 (was: 0.23.3, 0.23.2, 0.24.0) Status: Resolved (was: Patch Available) Thanks Arun for finding this bug. Thaks Jason for the quick turn around on the patch. I just put this into trunk branch-0.23 and branch-0.23.2 > TestRMContainerAllocator failing > > > Key: MAPREDUCE-3976 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3976 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Bikas Saha >Assignee: Jason Lowe > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3976.patch > > > The following stack trace is being generated > > org.apache.hadoop.metrics2.MetricsException: Metrics source JvmMetrics > already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:126) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:107) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:216) > at > org.apache.hadoop.metrics2.source.JvmMetrics.create(JvmMetrics.java:80) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:58) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:54) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:186) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:170) > at org.apache.hadoop.mapreduce.v2.app.MRApp.(MRApp.java:160) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator$1.(TestRMContainerAllocator.java:372) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgress(TestRMContainerAllocator.java:371) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > > This test fails when git trunk is reset to - commit > 6689d99b38c7c562e8cae484207ad30ad7b56eb5 > but passes when git trunk is reset to - commit > f429fdaf78a02211c4faee54b1ee92822edc5741 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3976) TestRMContainerAllocator failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3976: -- Attachment: MAPREDUCE-3976.patch Patch to call JvmMetrics.initSingleton() instead of JvmMetrics.create() to prevent multiple instantiations. No additional unit tests since this is fixing existing tests. > TestRMContainerAllocator failing > > > Key: MAPREDUCE-3976 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3976 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Bikas Saha >Assignee: Jason Lowe > Attachments: MAPREDUCE-3976.patch > > > The following stack trace is being generated > > org.apache.hadoop.metrics2.MetricsException: Metrics source JvmMetrics > already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:126) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:107) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:216) > at > org.apache.hadoop.metrics2.source.JvmMetrics.create(JvmMetrics.java:80) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:58) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:54) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:186) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:170) > at org.apache.hadoop.mapreduce.v2.app.MRApp.(MRApp.java:160) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator$1.(TestRMContainerAllocator.java:372) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgress(TestRMContainerAllocator.java:371) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > > This test fails when git trunk is reset to - commit > 6689d99b38c7c562e8cae484207ad30ad7b56eb5 > but passes when git trunk is reset to - commit > f429fdaf78a02211c4faee54b1ee92822edc5741 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3976) TestRMContainerAllocator failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3976: -- Target Version/s: 0.24.0, 0.23.2, 0.23.3 (was: 0.23.2) Status: Patch Available (was: Open) > TestRMContainerAllocator failing > > > Key: MAPREDUCE-3976 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3976 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Bikas Saha >Assignee: Jason Lowe > Attachments: MAPREDUCE-3976.patch > > > The following stack trace is being generated > > org.apache.hadoop.metrics2.MetricsException: Metrics source JvmMetrics > already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:126) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:107) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:216) > at > org.apache.hadoop.metrics2.source.JvmMetrics.create(JvmMetrics.java:80) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:58) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:54) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:186) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:170) > at org.apache.hadoop.mapreduce.v2.app.MRApp.(MRApp.java:160) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator$1.(TestRMContainerAllocator.java:372) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgress(TestRMContainerAllocator.java:371) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > > This test fails when git trunk is reset to - commit > 6689d99b38c7c562e8cae484207ad30ad7b56eb5 > but passes when git trunk is reset to - commit > f429fdaf78a02211c4faee54b1ee92822edc5741 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-1740: Status: Patch Available (was: Open) > NPE in getMatchingLevelForNodes when node locations are variable depth > -- > > Key: MAPREDUCE-1740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Todd Lipcon > Attachments: MAPREDUCE-1740.patch, mapreduce-1740.txt > > > In getMatchingLevelForNodes, we assume that both nodes have the same "depth" > (ie number of path components). If the user provides a topology script that > assigns one node a path like /foo/bar/baz and another node a path like > /foo/blah, this function will throw an NPE. > I'm not sure if there are other places where we assume that all node > locations have a constant number of paths. If so we should check the output > of the topology script aggressively to be sure this is the case. Otherwise I > think we simply need to add && n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3976) TestRMContainerAllocator failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3976: - Target Version/s: 0.23.2 Affects Version/s: (was: 0.23.0) 0.23.2 Fix Version/s: (was: 0.23.0) Jason - can you please take a look at this asap? Thanks. > TestRMContainerAllocator failing > > > Key: MAPREDUCE-3976 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3976 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Bikas Saha >Assignee: Jason Lowe > > The following stack trace is being generated > > org.apache.hadoop.metrics2.MetricsException: Metrics source JvmMetrics > already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:126) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:107) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:216) > at > org.apache.hadoop.metrics2.source.JvmMetrics.create(JvmMetrics.java:80) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:58) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:54) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:186) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:170) > at org.apache.hadoop.mapreduce.v2.app.MRApp.(MRApp.java:160) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator$1.(TestRMContainerAllocator.java:372) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgress(TestRMContainerAllocator.java:371) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > > This test fails when git trunk is reset to - commit > 6689d99b38c7c562e8cae484207ad30ad7b56eb5 > but passes when git trunk is reset to - commit > f429fdaf78a02211c4faee54b1ee92822edc5741 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-1740: Attachment: MAPREDUCE-1740.patch An updated patch for branch-1.0. Please review. > NPE in getMatchingLevelForNodes when node locations are variable depth > -- > > Key: MAPREDUCE-1740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.3 >Reporter: Todd Lipcon > Attachments: MAPREDUCE-1740.patch, mapreduce-1740.txt > > > In getMatchingLevelForNodes, we assume that both nodes have the same "depth" > (ie number of path components). If the user provides a topology script that > assigns one node a path like /foo/bar/baz and another node a path like > /foo/blah, this function will throw an NPE. > I'm not sure if there are other places where we assume that all node > locations have a constant number of paths. If so we should check the output > of the topology script aggressively to be sure this is the case. Otherwise I > think we simply need to add && n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
