[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated MAPREDUCE-3451: --- Attachment: MAPREDUCE-3451.v2.patch.txt This patch should apply cleanly to trunk, thanks. Port Fair Scheduler to MR2 -- Key: MAPREDUCE-3451 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2, scheduler Reporter: Patrick Wendell Assignee: Patrick Wendell Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3927) Shuffle hang when set map.failures.percent
[ https://issues.apache.org/jira/browse/MAPREDUCE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239261#comment-13239261 ] Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-3927: --- Shouldn't we add notifyAll() when the remaining maps is zero? Shuffle hang when set map.failures.percent -- Key: MAPREDUCE-3927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3927 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.21.0, 0.23.0 Reporter: MengWang Labels: patch Fix For: 0.24.0 Attachments: MAPREDUCE-3927.patch, MAPREDUCE-3927.patch When set mapred.max.map.failures.percent and there does have some failed maps, then shuffle will hang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4013) Reduce task gets stuck when a M/R job is configured to tolerate failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239260#comment-13239260 ] Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4013: --- Thanks Ravi for taking a look into the patch bq.What about the progress of map tasks when there are failed-maps ? Is it getting updated to 100% ? I see copySucceded() is updating the progress of map-tasks. So what happens when the last few maps fail ? Suppose say a user has configured *mapreduce.map.failures.maxpercent* as 2, so job can tolerate upto 2% of map tasks failures. As progress of map tasks indicates percentage of the sucessful completion of map tasks, I *think* showing the actual *progress* may be more useful than showing 100%. i.e. if progress of map tasks indicates 99%, by this, atleast it gives an idea that 1% of map tasks have been failed and consequently may take action on that failed map tasks. OTOH, if progress of map tasks should indicate the overall progress of the map phase, then patch needs to be updated to reflect the same. As this has been duplicated, we can have our further discussion at [MAPREDUCE-3927|https://issues.apache.org/jira/browse/MAPREDUCE-3927] Reduce task gets stuck when a M/R job is configured to tolerate failures Key: MAPREDUCE-4013 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4013 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Amar Kamat Priority: Blocker Labels: shuffle Fix For: 0.24.0 Attachments: MAPREDUCE-4013.patch When a M/R job is configured to run with some tolerance to task failures (via mapreduce.map.failures.maxpercent), then the reduce task of that job gets stuck in the shuffle phase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3927) Shuffle hang when set map.failures.percent
[ https://issues.apache.org/jira/browse/MAPREDUCE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239273#comment-13239273 ] Ravi Gummadi commented on MAPREDUCE-3927: - copySucceeded() is updating the progress of map-tasks, but tipFailed() is not updating. My understanding is that overall progress of map tasks represents the completed maps' work. So it may be better to update it for the failed-map-tasks case also. Shuffle hang when set map.failures.percent -- Key: MAPREDUCE-3927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3927 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.21.0, 0.23.0 Reporter: MengWang Labels: patch Fix For: 0.24.0 Attachments: MAPREDUCE-3927.patch, MAPREDUCE-3927.patch When set mapred.max.map.failures.percent and there does have some failed maps, then shuffle will hang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3947) yarn.app.mapreduce.am.resource.mb not documented
[
https://issues.apache.org/jira/browse/MAPREDUCE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239304#comment-13239304
]
Devaraj K commented on MAPREDUCE-3947:
--
{quote}
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{quote}
No tests needed for this since it is a documentation issue.
yarn.app.mapreduce.am.resource.mb not documented
Key: MAPREDUCE-3947
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3947
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.24.0, 0.23.3
Reporter: Todd Lipcon
Priority: Minor
Labels: mrv2
Attachments: MAPREDUCE-3947.patch
This configuration is useful but doesn't appear to be documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.1, 0.23.2, 0.23.3
Reporter: Bhallamudi Venkata Siva Kamesh
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
[
https://issues.apache.org/jira/browse/MAPREDUCE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239322#comment-13239322
]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4071:
---
MRAppMaster received shutdown signal just before executing start method of
ContainerAllocatorRouter service. So while executing shutdown hook
of MRAppMaster
{code:title=MRAppMaster.java|borderStyle=solid}
if(appMaster.containerAllocator instanceof ContainerAllocatorRouter) {
((ContainerAllocatorRouter) appMaster.containerAllocator)
.setSignalled(true);
}
{code}
above code tries set the flag of containerAllocator object in
ContainerAllocatorRouter#setSignalled(). But containerAllocator object has been
instantiated while executing start method. So containerAllocator is null and
hence NPE.
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.1, 0.23.2, 0.23.3
Reporter: Bhallamudi Venkata Siva Kamesh
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4064) Job History Link in RM UI is redirecting to the URL which contains Job Id twice
[
https://issues.apache.org/jira/browse/MAPREDUCE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4064:
-
Attachment: MAPREDUCE-4064-1.patch
Job History Link in RM UI is redirecting to the URL which contains Job Id
twice
---
Key: MAPREDUCE-4064
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4064
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4064-1.patch, MAPREDUCE-4064.patch
{code:xml}
http://xx.x.x.x:19888/jobhistory/job/job_1332435449546_0001/jobhistory/job/job_1332435449546_0001
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4064) Job History Link in RM UI is redirecting to the URL which contains Job Id twice
[
https://issues.apache.org/jira/browse/MAPREDUCE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4064:
-
Status: Patch Available (was: Open)
Thank you Robert for looking into the patch.
Before posting I tested the patch including all the links in the AM/History UI
but I couldn't find out the cosmetic problems(caused by previous patch) in the
AM UI due to the issue MAPREDUCE-3173.
Now I have updated the patch with the change applies to only redirected history
URL. Can you please review this patch?
Job History Link in RM UI is redirecting to the URL which contains Job Id
twice
---
Key: MAPREDUCE-4064
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4064
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4064-1.patch, MAPREDUCE-4064.patch
{code:xml}
http://xx.x.x.x:19888/jobhistory/job/job_1332435449546_0001/jobhistory/job/job_1332435449546_0001
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3757) Rumen Folder is not adjusting the shuffleFinished and sortFinished times of reduce task attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3757: Release Note: Fixed the sortFinishTime and shuffleFinishTime adjustments in Rumen Folder. Status: Patch Available (was: Open) Rumen Folder is not adjusting the shuffleFinished and sortFinished times of reduce task attempts Key: MAPREDUCE-3757 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3757 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3757.v0.patch, 3757.v1.1.patch, 3757.v1.patch Rumen Folder is not adjusting the shuffleFinished and sortFinished times of reduce task attempts when it is adjusting the attempt-start-time and attempt-finish-time. This is leading to wrong values which are greater than the attempt-finish-time in trace file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3757) Rumen Folder is not adjusting the shuffleFinished and sortFinished times of reduce task attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3757: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Rumen Folder is not adjusting the shuffleFinished and sortFinished times of reduce task attempts Key: MAPREDUCE-3757 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3757 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3757.v0.patch, 3757.v1.1.patch, 3757.v1.patch Rumen Folder is not adjusting the shuffleFinished and sortFinished times of reduce task attempts when it is adjusting the attempt-start-time and attempt-finish-time. This is leading to wrong values which are greater than the attempt-finish-time in trace file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4064) Job History Link in RM UI is redirecting to the URL which contains Job Id twice
[
https://issues.apache.org/jira/browse/MAPREDUCE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239388#comment-13239388
]
Hadoop QA commented on MAPREDUCE-4064:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520094/MAPREDUCE-4064-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2107//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2107//console
This message is automatically generated.
Job History Link in RM UI is redirecting to the URL which contains Job Id
twice
---
Key: MAPREDUCE-4064
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4064
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4064-1.patch, MAPREDUCE-4064.patch
{code:xml}
http://xx.x.x.x:19888/jobhistory/job/job_1332435449546_0001/jobhistory/job/job_1332435449546_0001
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4052) Windows eclpise can not submit the job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xieguiming updated MAPREDUCE-4052: -- Attachment: MAPREDUCE-4052.patch Windows eclpise can not submit the job -- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Attachments: MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4052) Windows eclpise can not submit the job
[
https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xieguiming updated MAPREDUCE-4052:
--
Status: Patch Available (was: Open)
We will replace the windows path separator. and we can submit the job using
windows eclipse client.
Do like this:
1,modify
{code:title=ContainerLaunch.java|borderStyle=solid}
// Make a copy of env to iterate do variable expansion
for (EntryString, String entry : environment.entrySet()){
String value = entry.getValue();
entry.setValue(
value.replace(
ApplicationConstants.LOG_DIR_EXPANSION_VAR,
containerLogDir.toUri().getPath())
);
}
{code}
{code:title=ContainerLaunch.java|borderStyle=solid}
// Make a copy of env to iterate do variable expansion
for (EntryString, String entry : environment.entrySet()){
String value = entry.getValue().replace(
ApplicationConstants.LOG_DIR_EXPANSION_VAR,
containerLogDir.toUri().getPath()).replace(
ApplicationConstants.WINDOWS_PATH_SEPERATOR,
System.getProperty(path.separator));
entry.setValue(value);
}
{code}
2,add one constant
{code:title=ApplicationConstants.java|borderStyle=solid}
public static final String WINDOWS_PATH_SEPERATOR = ;;
{code}
Windows eclpise can not submit the job
--
Key: MAPREDUCE-4052
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Affects Versions: 0.23.1
Environment: client on the Windows, the the cluster on the suse
Reporter: xieguiming
Attachments: MAPREDUCE-4052.patch
when I use the eclipse on the windows to submit the job. and the
applicationmaster throw the exception:
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/v2/app/MRAppMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
The reasion is :
class Apps addToEnvironment function, use the
private static final String SYSTEM_PATH_SEPARATOR =
System.getProperty(path.separator);
and will result the MRApplicationMaster classpath use the ; separator.
I suggest that nodemanger do the replace.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher thread
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239427#comment-13239427 ] Hudson commented on MAPREDUCE-4061: --- Integrated in Hadoop-Hdfs-trunk #997 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/997/]) MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305607) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305607 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java RM only has 1 AM launcher thread Key: MAPREDUCE-4061 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4061.patch The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread. If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4066) To get yarn.app.mapreduce.am.staging-dir value, should set the default value
[
https://issues.apache.org/jira/browse/MAPREDUCE-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239429#comment-13239429
]
Hudson commented on MAPREDUCE-4066:
---
Integrated in Hadoop-Hdfs-trunk #997 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/997/])
MAPREDUCE-4066. Use default value when fetching MR_AM_STAGING_DIR.
Contributed by xieguiming. (Revision 1305343)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305343
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
To get yarn.app.mapreduce.am.staging-dir value, should set the default value
--
Key: MAPREDUCE-4066
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4066
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission, mrv2
Affects Versions: 0.23.1
Environment: client is windows eclipse, server is suse
Reporter: xieguiming
Priority: Minor
Fix For: 0.23.3
Attachments: MAPREDUCE-4066.patch, MAPREDUCE-4066.patch
when submit the job use the windows eclipse, and the
yarn.app.mapreduce.am.staging-dir value is null.
{code:title=MRApps.java|borderStyle=solid}
public static Path getStagingAreaDir(Configuration conf, String user) {
return new Path(
conf.get(MRJobConfig.MR_AM_STAGING_DIR) +
Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT);
}
{code}
should modify to:
{code:title=MRApps.java|borderStyle=solid}
public static Path getStagingAreaDir(Configuration conf, String user) {
return new Path(
conf.get(MRJobConfig.MR_AM_STAGING_DIR,/tmp/hadoop-yarn/staging) +
Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3353) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239428#comment-13239428 ] Hudson commented on MAPREDUCE-3353: --- Integrated in Hadoop-Hdfs-trunk #997 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/997/]) MAPREDUCE-3353. Fixed commit msg to point to right jira. (Revision 1305457) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305457 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes - Key: MAPREDUCE-3353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.3 Attachments: MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch When a node gets lost or turns faulty, AM needs to know about that event so that it can take some action like for e.g. re-executing map tasks whose intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4052) Windows eclpise can not submit the job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239440#comment-13239440 ] Hadoop QA commented on MAPREDUCE-4052: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520107/MAPREDUCE-4052.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2108//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2108//console This message is automatically generated. Windows eclpise can not submit the job -- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Attachments: MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4066) To get yarn.app.mapreduce.am.staging-dir value, should set the default value
[
https://issues.apache.org/jira/browse/MAPREDUCE-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239476#comment-13239476
]
Hudson commented on MAPREDUCE-4066:
---
Integrated in Hadoop-Mapreduce-trunk #1032 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1032/])
MAPREDUCE-4066. Use default value when fetching MR_AM_STAGING_DIR.
Contributed by xieguiming. (Revision 1305343)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305343
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
To get yarn.app.mapreduce.am.staging-dir value, should set the default value
--
Key: MAPREDUCE-4066
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4066
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission, mrv2
Affects Versions: 0.23.1
Environment: client is windows eclipse, server is suse
Reporter: xieguiming
Priority: Minor
Fix For: 0.23.3
Attachments: MAPREDUCE-4066.patch, MAPREDUCE-4066.patch
when submit the job use the windows eclipse, and the
yarn.app.mapreduce.am.staging-dir value is null.
{code:title=MRApps.java|borderStyle=solid}
public static Path getStagingAreaDir(Configuration conf, String user) {
return new Path(
conf.get(MRJobConfig.MR_AM_STAGING_DIR) +
Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT);
}
{code}
should modify to:
{code:title=MRApps.java|borderStyle=solid}
public static Path getStagingAreaDir(Configuration conf, String user) {
return new Path(
conf.get(MRJobConfig.MR_AM_STAGING_DIR,/tmp/hadoop-yarn/staging) +
Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher thread
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239474#comment-13239474 ] Hudson commented on MAPREDUCE-4061: --- Integrated in Hadoop-Mapreduce-trunk #1032 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1032/]) MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305607) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305607 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java RM only has 1 AM launcher thread Key: MAPREDUCE-4061 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4061.patch The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread. If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3353) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239475#comment-13239475 ] Hudson commented on MAPREDUCE-3353: --- Integrated in Hadoop-Mapreduce-trunk #1032 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1032/]) MAPREDUCE-3353. Fixed commit msg to point to right jira. (Revision 1305457) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305457 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes - Key: MAPREDUCE-3353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.3 Attachments: MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch When a node gets lost or turns faulty, AM needs to know about that event so that it can take some action like for e.g. re-executing map tasks whose intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4064) Job History Link in RM UI is redirecting to the URL which contains Job Id twice
[
https://issues.apache.org/jira/browse/MAPREDUCE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4064:
---
Status: Open (was: Patch Available)
Job History Link in RM UI is redirecting to the URL which contains Job Id
twice
---
Key: MAPREDUCE-4064
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4064
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4064-1.patch, MAPREDUCE-4064.patch
{code:xml}
http://xx.x.x.x:19888/jobhistory/job/job_1332435449546_0001/jobhistory/job/job_1332435449546_0001
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4064) Job History Link in RM UI is redirecting to the URL which contains Job Id twice
[
https://issues.apache.org/jira/browse/MAPREDUCE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239488#comment-13239488
]
Robert Joseph Evans commented on MAPREDUCE-4064:
Having the changes only apply when doing a redirect will fix the problem for
Map Reduce, but I think the underlying cause of the issue is not with the proxy
itself. I believe that the proxy is working as designed, the problem is with
the code that is producing the URI.
I believe that the issue is with ProxyUriUtils.getProxyUri(). It is adding on
the extra path part that should be removed. It should always act as if the
trackingUri parameter is null. And if that is the case that parameter should
just be removed, which will simplify things in RMAppAttemptImpl.java because we
will not have to ever regenerate the proxiedTrackingUrl.
I am fairly sure that I put this bug in so thank you for finding it and trying
to fix it.
Job History Link in RM UI is redirecting to the URL which contains Job Id
twice
---
Key: MAPREDUCE-4064
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4064
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4064-1.patch, MAPREDUCE-4064.patch
{code:xml}
http://xx.x.x.x:19888/jobhistory/job/job_1332435449546_0001/jobhistory/job/job_1332435449546_0001
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
User set java.library.path seems to overwrite default creating problems native lib loading -- Key: MAPREDUCE-4072 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth This was found by Peeyush Bishnoi. While running a distributed cache example with Hadoop-0.23, tasks are failing as follows: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user is user Same Pig script and command work successfully on 0.20 See this in the stderr: Exception in thread main java.lang.ExceptionInInitializerError at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264) at org.apache.hadoop.security.Groups.(Groups.java:54) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178) at org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223) at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75) Caused by: java.lang.RuntimeException: Bailing out since native library couldn't be loaded at org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48) ... 12 more Pig command: $ pig -Dmapred.job.queue.name=queue -Dmapred.cache.archives=archives -Dmapred.child.java.opts=-Djava.library.path=./ygeo/lib -Dip2geo.preLoadLibraries=some other libs -Djava.io.tmpdir=/grid/0/tmp -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4073) CS assigns multiple off-switch containers when using multi-level-queues
CS assigns multiple off-switch containers when using multi-level-queues --- Key: MAPREDUCE-4073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4073 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.1 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical CS is supposed to be allocating a single off-switch container per node heartbeat (MAPREDUCE-3641). This works for queues directly under root, but not in the case of multi-level queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4073) CS assigns multiple off-switch containers when using multi-level-queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4073: -- Attachment: MR4073.txt The type of allocation was getting reset to node-local in case of multi level queues. Simple patch to set the correct type. CS assigns multiple off-switch containers when using multi-level-queues --- Key: MAPREDUCE-4073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4073 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.1 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR4073.txt CS is supposed to be allocating a single off-switch container per node heartbeat (MAPREDUCE-3641). This works for queues directly under root, but not in the case of multi-level queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4073) CS assigns multiple off-switch containers when using multi-level-queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4073: -- Status: Patch Available (was: Open) CS assigns multiple off-switch containers when using multi-level-queues --- Key: MAPREDUCE-4073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4073 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.1 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR4073.txt CS is supposed to be allocating a single off-switch container per node heartbeat (MAPREDUCE-3641). This works for queues directly under root, but not in the case of multi-level queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
[ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239882#comment-13239882 ] Kihwal Lee commented on MAPREDUCE-4072: --- It look like MAPREDUCE-2880 got rid of merging of java.library.path and introduced setEnvFromInputString(). However this does not take care of the cases for java.library.path. When launching a container, -Djava.library.path= should contain both the system jni lib path (i.e. the location of libhadoop.so) and whatever user specified. Please note that setting LD_LIBRARY_PATH alone is not enough to load JNI libraries. A custom LD_LIBRARY_PATH needs to be used when the user supplies custom native libraries, which the JNI library depends on, but the base system does not provide. setEnvFromInputString() does path munging on LD_LIBRARY_PATH. User set java.library.path seems to overwrite default creating problems native lib loading -- Key: MAPREDUCE-4072 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth This was found by Peeyush Bishnoi. While running a distributed cache example with Hadoop-0.23, tasks are failing as follows: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user is user Same Pig script and command work successfully on 0.20 See this in the stderr: Exception in thread main java.lang.ExceptionInInitializerError at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264) at org.apache.hadoop.security.Groups.(Groups.java:54) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178) at org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223) at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75) Caused by: java.lang.RuntimeException: Bailing out since native library couldn't be loaded at org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48) ... 12 more Pig command: $ pig -Dmapred.job.queue.name=queue -Dmapred.cache.archives=archives -Dmapred.child.java.opts=-Djava.library.path=./ygeo/lib -Dip2geo.preLoadLibraries=some other libs -Djava.io.tmpdir=/grid/0/tmp -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4073) CS assigns multiple off-switch containers when using multi-level-queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239927#comment-13239927 ] Hadoop QA commented on MAPREDUCE-4073: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520180/MR4073.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2109//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2109//console This message is automatically generated. CS assigns multiple off-switch containers when using multi-level-queues --- Key: MAPREDUCE-4073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4073 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.1 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Attachments: MR4073.txt CS is supposed to be allocating a single off-switch container per node heartbeat (MAPREDUCE-3641). This works for queues directly under root, but not in the case of multi-level queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4045) RM UI - Applications - Application Master Link - Job Link - New Maps/Reduces leads to circular redirect error
[
https://issues.apache.org/jira/browse/MAPREDUCE-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K reassigned MAPREDUCE-4045:
Assignee: (was: Devaraj K)
RM UI - Applications - Application Master Link - Job Link - New
Maps/Reduces leads to circular redirect error
-
Key: MAPREDUCE-4045
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4045
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.2
Reporter: Devaraj K
{code:xml}
HTTP ERROR 500
Problem accessing
/proxy/application_1332261815858_0002/mapreduce/attempts/job_1332261815858_2_2/m/NEW.
Reason:
Circular redirect to
'http://HOST-192-168-47-207:41992/mapreduce/attempts/job_1332261815858_2_2/m/NEW'
Caused by:
org.apache.commons.httpclient.CircularRedirectException: Circular redirect to
'http://HOST-192-168-47-207:41992/mapreduce/attempts/job_1332261815858_2_2/m/NEW'
at
org.apache.commons.httpclient.HttpMethodDirector.processRedirectResponse(HttpMethodDirector.java:638)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:179)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:148)
at
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:269)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:940)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Powered by Jetty://
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3364) Job executed through ftp file system is failing with java.io.IOException: Seek not supported
[ https://issues.apache.org/jira/browse/MAPREDUCE-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-3364: Assignee: (was: Devaraj K) Job executed through ftp file system is failing with java.io.IOException: Seek not supported -- Key: MAPREDUCE-3364 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3364 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Ramgopal N Instead of hdfs file as input to the job,i have given local file through ftp as input and executed a job.The job is failing with ERROR Error: java.io.IOException: Seek not supported at org.apache.hadoop.fs.ftp.FTPInputStream.seek(FTPInputStream.java:60) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:47) at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:117) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:484) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:710) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) The same job is successfully getting executed in V1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4046) Task Log stdout and stderr don't honor the property mapred.userlog.limit.kb
[ https://issues.apache.org/jira/browse/MAPREDUCE-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-4046: Assignee: (was: Devaraj K) Task Log stdout and stderr don't honor the property mapred.userlog.limit.kb - Key: MAPREDUCE-4046 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4046 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.0.1 Reporter: Devaraj K -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
