[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751008#comment-13751008 ] Siddharth Seth commented on MAPREDUCE-5329: --- bq. I agree with you. I think that for this we'll need to clarify which aux-services belong to the MapReduce AM. Probably a new conf will be needed. Do you want me to open a ditinct MapReduce JIRA for that? I want to keep this one focused on APPLICATION_INIT. If the MR AM knows how to handle multiple services, and informs the NM about all of these, APPLICATION_INIT should just go out to all of them. Sounds like this jira can be used to fix this ? APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler --- Key: MAPREDUCE-5329 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.0.4-alpha Reporter: Avner BenHanoch APPLICATION_INIT is never sent to AuxServices other than the built-in ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be able to function, because APPLICATION_INIT enables the AuxiliaryService to map jobId-userId. This is needed for properly finding the MOFs of a job per reducers' requests. NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to hard-coded expression in hadoop code. The current TaskAttemptImpl.java code explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID, ...) and ignores any additional AuxiliaryService. As a result, only the built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party AuxillaryService will never get APPLICATION_INIT events. I think a solution can be in one of two ways: 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register each of them, by calling serviceData.put (…) in loop. 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668 APPLICATION_STOP is never sent to AuxServices. This means that in case the 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux Services regardless of the value in event.getServiceID(). I prefer the 2nd solution. I am welcoming any ideas. I can provide the needed patch for any option that people like. See [Pluggable Shuffle in Hadoop documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
reduce job hung in pending state: No room for reduce task
Hello All, We're running into the following 2 bugs again: https://issues.apache.org/jira/browse/HADOOP-5241 https://issues.apache.org/jira/browse/MAPREDUCE-2324 Both of them a listed as closed fixed. (I was actually the one that got cloudera to submit MAPREDUCE-2324) Does anyone know is anyone else seeing these in later releases? We're running the following on various version of Cent OS with Java 1.6: hadoop-2.0.0+1357-1.cdh4.3.0.p0.21.el5 hadoop-0.20-mapreduce-jobtracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5 hadoop-0.20-mapreduce-2.0.0+1357-1.cdh4.3.0.p0.21.el5 hadoop-0.20-mapreduce-tasktracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5 hadoop-hdfs-namenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5 hadoop-hdfs-secondarynamenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5 hadoop-hdfs-2.0.0+1357-1.cdh4.3.0.p0.21.el5 hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0.p0.21.el5 Just for a quick summary, basically a reduce job get hung in pending while trying to find room on a task tracker, it keeps trying over and over and never fails. So you end up with a whole bunch of these in the logs: 2013-08-27 00:48:01,412 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_104.sm.tld:127.0.0.1/127.0.0.1:43723 has 250176954368 bytes free; but we expect reduce input to take 283580756533 Thanks in advance for any help on the issue.. JC
[jira] [Commented] (MAPREDUCE-5402) DynamicInputFormat should allow overriding of MAX_CHUNKS_TOLERABLE
[ https://issues.apache.org/jira/browse/MAPREDUCE-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751376#comment-13751376 ] David Rosenstrauch commented on MAPREDUCE-5402: --- Hi. Just wondering if there's been any progress on getting this fix released. We're still running into issues in production with the long tail of distcp jobs taking hours to complete. I'd love to deploy a fix to our system soon to solve that, if possible. DynamicInputFormat should allow overriding of MAX_CHUNKS_TOLERABLE -- Key: MAPREDUCE-5402 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5402 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp, mrv2 Reporter: David Rosenstrauch Assignee: Tsuyoshi OZAWA Attachments: MAPREDUCE-5402.1.patch, MAPREDUCE-5402.2.patch, MAPREDUCE-5402.3.patch In MAPREDUCE-2765, which provided the design spec for DistCpV2, the author describes the implementation of DynamicInputFormat, with one of the main motivations cited being to reduce the chance of long-tails where a few leftover mappers run much longer than the rest. However, I today ran into a situation where I experienced exactly such a long tail using DistCpV2 and DynamicInputFormat. And when I tried to alleviate the problem by overriding the number of mappers and the split ratio used by the DynamicInputFormat, I was prevented from doing so by the hard-coded limit set in the code by the MAX_CHUNKS_TOLERABLE constant. (Currently set to 400.) This constant is actually set quite low for production use. (See a description of my use case below.) And although MAPREDUCE-2765 states that this is an overridable maximum, when reading through the code there does not actually appear to be any mechanism available to override it. This should be changed. It should be possible to expand the maximum # of chunks beyond this arbitrary limit. For example, here is the situation I ran into today: I ran a distcpv2 job on a cluster with 8 machines containing 128 map slots. The job consisted of copying ~2800 files from HDFS to Amazon S3. I overrode the number of mappers for the job from the default of 20 to 128, so as to more properly parallelize the copy across the cluster. The number of chunk files created was calculated as 241, and mapred.num.entries.per.chunk was calculated as 12. As the job ran on, it reached a point where there were only 4 remaining map tasks, which had each been running for over 2 hours. The reason for this was that each of the 12 files that those mappers were copying were quite large (several hundred megabytes in size) and took ~20 minutes each. However, during this time, all the other 124 mappers sat idle. In theory I should be able to alleviate this problem with DynamicInputFormat. If I were able to, say, quadruple the number of chunk files created, that would have made each chunk contain only 3 files, and these large files would have gotten distributed better around the cluster and copied in parallel. However, when I tried to do that - by overriding mapred.listing.split.ratio to, say, 10 - DynamicInputFormat responded with an exception (Too many chunks created with splitRatio:10, numMaps:128. Reduce numMaps or decrease split-ratio to proceed.) - presumably because I exceeded the MAX_CHUNKS_TOLERABLE value of 400. Is there any particular logic behind this MAX_CHUNKS_TOLERABLE limit? I can't personally see any. If this limit has no particular logic behind it, then it should be overridable - or even better: removed altogether. After all, I'm not sure I see any need for it. Even if numMaps * splitRatio resulted in an extraordinarily large number, if the code were modified so that the number of chunks got calculated as Math.min( numMaps * splitRatio, numFiles), then there would be no need for MAX_CHUNKS_TOLERABLE. In this worst-case scenario where the product of numMaps and splitRatio is large, capping the number of chunks at the number of files (numberOfChunks = numberOfFiles) would result in 1 file per chunk - the maximum parallelization possible. That may not be the best-tuned solution for some users, but I would think that it should be left up to the user to deal with the potential consequence of not having tuned their job properly. Certainly that would be better than having an arbitrary hard-coded limit that *prevents* proper parallelization when dealing with large files and/or large numbers of mappers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5362) clean up POM dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751399#comment-13751399 ] Roman Shaposhnik commented on MAPREDUCE-5362: - I'm cleaning up a few build issues (Bigtop 0.8.0 related) and I was wondering whether you'd think it would be a good idea for me to tackle this one as well. Please assign it to me if you think it is. clean up POM dependencies - Key: MAPREDUCE-5362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5362 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5480) TestJSHSecurity.testDelegationToken is breaking after YARN-1085
[
https://issues.apache.org/jira/browse/MAPREDUCE-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5480:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Fixed via YARN-1085
TestJSHSecurity.testDelegationToken is breaking after YARN-1085
---
Key: MAPREDUCE-5480
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5480
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Omkar Vinit Joshi
Priority: Blocker
Attachments: MAPREDUCE-5480.20130824.1.patch
See https://builds.apache.org/job/PreCommit-YARN-Build/1755//testReport/.
{code}
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:251)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.initializeWebApp(HistoryClientService.java:152)
---
Caused by: javax.servlet.ServletException: javax.servlet.ServletException:
Principal not defined in configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:203)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
---
Caused by: javax.servlet.ServletException: Principal not defined in
configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:164)
... 53 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5362) clean up POM dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned MAPREDUCE-5362: - Assignee: Roman Shaposhnik all yours, thx clean up POM dependencies - Key: MAPREDUCE-5362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5362 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Roman Shaposhnik Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5483) revert MAPREDUCE-5357
Alejandro Abdelnur created MAPREDUCE-5483: - Summary: revert MAPREDUCE-5357 Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Fix For: 2.1.1-beta MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5357) Job staging directory owner checking could fail on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751547#comment-13751547
]
Alejandro Abdelnur commented on MAPREDUCE-5357:
---
FYI, opened MAPREDUCE-5483 to revert this JIRA.
Job staging directory owner checking could fail on Windows
--
Key: MAPREDUCE-5357
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5357
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5357-trunk.patch
In {{JobSubmissionFiles.getStagingDir()}}, we have following code that will
throw exception if the directory owner is not the current user.
{code:java}
String owner = fsStatus.getOwner();
if (!(owner.equals(currentUser) || owner.equals(realUser))) {
throw new IOException(The ownership on the staging directory +
stagingArea + is not as expected. +
It is owned by + owner + . The directory must +
be owned by the submitter + currentUser + or +
by + realUser);
}
{code}
This check will fail on Windows when the underlying file system is
LocalFileSystem. Because on Windows, the default file or directory owner
could be Administrators group if the user belongs to Administrators group.
Quite a few MR unit tests that runs MR mini cluster with localFs as
underlying file system fail because of this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter reassigned MAPREDUCE-5483: Assignee: Robert Kanter revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751558#comment-13751558 ] Alejandro Abdelnur commented on MAPREDUCE-5483: --- I guess we could do a check if the platform is windows to do the chown() but the fix was because testcases failing on windows when running them as admin. it seems fishy to me that Windows will fail silently chown(). Regardless, either we guard this code to run only on Windows or we revert it. I'd prefer reverting it. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-5483: - Attachment: MAPREDUCE-5483.patch Patch reverts the change. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-5483: - Status: Patch Available (was: Open) revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751570#comment-13751570 ] Chuan Liu commented on MAPREDUCE-5483: -- Who is the owner of the staging area directory? My original thought was that it is always the user who submits jobs because we will enforce the ownership explicitly in another code path (the if clause) of the same method. Thus chown() will always be a NOP; and the Windows Administrators user case is the only exception. It seems my assumption was wrong from the Ozzie test. However, this seems suggesting Ozzie test may also fail if you re-run the test without deleting the staging directory. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751574#comment-13751574 ] Chuan Liu commented on MAPREDUCE-5483: -- I want to understand the scenario a little bit more. After reverting this change, it means: # If the directory already exists, we will enforce it belongs to the current user. # Otherwise, we create a new directory. Ownership does not matter in this case. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751578#comment-13751578 ] Alejandro Abdelnur commented on MAPREDUCE-5483: --- if you run builds in the same directory as different users you'll run into permission issues deleting files from previous run unless the user running the second time is a superuser. That seems a wrong thing to do. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751580#comment-13751580 ] Alejandro Abdelnur commented on MAPREDUCE-5483: --- if we revert this patch you don't do a chown() in a dir you created. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[
https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751594#comment-13751594
]
Hadoop QA commented on MAPREDUCE-5483:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600219/MAPREDUCE-5483.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3963//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3963//console
This message is automatically generated.
revert MAPREDUCE-5357
-
Key: MAPREDUCE-5483
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 2.1.0-beta
Reporter: Alejandro Abdelnur
Assignee: Robert Kanter
Fix For: 2.1.1-beta
Attachments: MAPREDUCE-5483.patch
MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid
unless you are superuser. if you a chown() to yourself is a NOP, that is why
has not been detected in Hadoop testcases where user is running as itself.
However, in distcp testcases run by Oozie which use test users/groups from
UGI for minicluster it is failing because of this chown() either because the
test user does not exist of because the current use does not have privileges
to do a chown().
We should revert MAPREDUCE-5357. Windows should handle this with some
conditional logic used only when running in Windows.
Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in
2.1.0-beta.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751603#comment-13751603 ] Alejandro Abdelnur commented on MAPREDUCE-5483: --- +1 from my side. [~chuanliu], are you OK with the revert? revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751667#comment-13751667 ] Chuan Liu commented on MAPREDUCE-5483: -- Who is the directory owner in your test case? We are setting owner to a newly created directory. To my understanding, the creator should be the owner under POSIX mode including HDFS. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751676#comment-13751676 ] Robert Kanter commented on MAPREDUCE-5483: -- In the testcase, the user is named 'test'. The user doesn't exist, so it complains about an Invalid argument. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751681#comment-13751681 ] Alejandro Abdelnur commented on MAPREDUCE-5483: --- UGI and minicluster have support for adding test users which do not map to OS users. when using such test users things blow up in the local file system. Before MAPREDUCE-5357 (without the chown) thing were working fine in such scenarios. MAPREDUCE-5357 introduced a regression. I'm planning to commit the current tomorrow. If you want to do a special handling for Windows (which I would not recommend) please upload a patch. The patch should have the effect of a 'revert' for non Windows platforms. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5483) revert MAPREDUCE-5357
[ https://issues.apache.org/jira/browse/MAPREDUCE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751696#comment-13751696 ] Chuan Liu commented on MAPREDUCE-5483: -- OK. This makes sense. I am also +1. We can work on a separate Windows fix in another JIRA. revert MAPREDUCE-5357 - Key: MAPREDUCE-5483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5483 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.1.1-beta Attachments: MAPREDUCE-5483.patch MAPREDUCE-5357 does a fileystem chown() operation. chown() is not valid unless you are superuser. if you a chown() to yourself is a NOP, that is why has not been detected in Hadoop testcases where user is running as itself. However, in distcp testcases run by Oozie which use test users/groups from UGI for minicluster it is failing because of this chown() either because the test user does not exist of because the current use does not have privileges to do a chown(). We should revert MAPREDUCE-5357. Windows should handle this with some conditional logic used only when running in Windows. Opening a new JIRA and not reverting directly because MAPREDUCE-5357 went in 2.1.0-beta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1729) Distributed cache should provide an option to fail the job or not, if cache file gets modified on the fly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751722#comment-13751722 ] Akira AJISAKA commented on MAPREDUCE-1729: -- [~yamashitasni], thanks for your comment. I'll try to implement your 2nd option. Distributed cache should provide an option to fail the job or not, if cache file gets modified on the fly. -- Key: MAPREDUCE-1729 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1729 Project: Hadoop Map/Reduce Issue Type: New Feature Components: distributed-cache Reporter: Amareshwari Sriramadasu Assignee: Akira AJISAKA Currently, distributed cache fails the job if the cache file gets modified on the fly. But there should be an option to fail a job or not. See discussions in MAPREDUCE-1288. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5439) mapred-default.xml has missing properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5439: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 mapred-default.xml has missing properties - Key: MAPREDUCE-5439 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5439 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.1.0-beta Reporter: Siddharth Wagle Fix For: 2.3.0 Properties that need to be added: mapreduce.map.memory.mb mapreduce.map.java.opts mapreduce.reduce.memory.mb mapreduce.reduce.java.opts Properties that need to be fixed: mapred.child.java.opts should not be in mapred-default. yarn.app.mapreduce.am.command-opts description needs fixing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4868) Allow multiple iteration for map
[ https://issues.apache.org/jira/browse/MAPREDUCE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4868: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Allow multiple iteration for map Key: MAPREDUCE-4868 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4868 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Jerry Chen Fix For: 3.0.0, 2.3.0 Original Estimate: 168h Remaining Estimate: 168h Currently, the Mapper class allows advanced users to override public void run(Context context) method for more control over the execution of the mapper, while Context interface limit the operations over the data which is the foundation of more control. One of use cases is that when I am considering a hive optimziation problem, I want to go two passes over the input data instead of using a another job or task ( which may slower the whole process). Each pass do the same thing but with a different parameters. This is a new paradigm of Map Reduce usage and can be archived easily by extend Context interface a little with the more control over the data such as reset the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5036: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036-2.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5028) Maps fail when io.sort.mb is set to high value
[
https://issues.apache.org/jira/browse/MAPREDUCE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5028:
-
Fix Version/s: (was: 2.1.0-beta)
2.3.0
Maps fail when io.sort.mb is set to high value
--
Key: MAPREDUCE-5028
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5028
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.1.1, 2.0.3-alpha, 0.23.5
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Fix For: 1.2.0, 2.3.0
Attachments: mr-5028-branch1.patch, mr-5028-branch1.patch,
mr-5028-branch1.patch, MR-5028_testapp.patch, mr-5028-trunk.patch,
mr-5028-trunk.patch, mr-5028-trunk.patch, repro-mr-5028.patch
Verified the problem exists on branch-1 with the following configuration:
Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m,
io.sort.mb=1280, dfs.block.size=2147483648
Run teragen to generate 4 GB data
Maps fail when you run wordcount on this configuration with the following
error:
{noformat}
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4468) Encapsulate FairScheduler preemption logic into helper class
[ https://issues.apache.org/jira/browse/MAPREDUCE-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4468: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Encapsulate FairScheduler preemption logic into helper class Key: MAPREDUCE-4468 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4468 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Ryan Hennig Priority: Minor Labels: refactoring, scheduler Fix For: 2.3.0 Attachments: MAPREDUCE-4468.patch Original Estimate: 4h Remaining Estimate: 4h I've extracted the preemption logic from the Fair Scheduler into a helper class so that FairScheduler is closer to following the Single Responsibility Principle. This may eventually evolve into a generalized preemption module which could be leveraged by other schedulers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5170) incorrect exception message if min node size min rack size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5170: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 incorrect exception message if min node size min rack size Key: MAPREDUCE-5170 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5170 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.3-alpha Reporter: Sangjin Lee Priority: Trivial Fix For: 2.3.0 Attachments: MAPREDUCE-5170.patch The exception message for CombineFileInputFormat if min node size min rack size is worded backwards. Currently it reads Minimum split size per node... cannot be smaller than the minimum split size per rack... It should be Minimum split size per node... cannot be LARGER than the minimum split size per rack... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4253) Tests for mapreduce-client-core are lying under mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4253: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Tests for mapreduce-client-core are lying under mapreduce-client-jobclient -- Key: MAPREDUCE-4253 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4253 Project: Hadoop Map/Reduce Issue Type: Task Components: client Affects Versions: 2.0.0-alpha Reporter: Harsh J Assignee: Tsuyoshi OZAWA Fix For: 2.3.0 Attachments: crossing_project_checker.rb, MR-4253.1.patch, MR-4253.2.patch, result.txt Many of the tests for client libs from mapreduce-client-core are lying under mapreduce-client-jobclient. We should investigate if this is the right thing to do and if not, move the tests back into client-core. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4462) Enhance readability of TestFairScheduler.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4462: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Enhance readability of TestFairScheduler.java - Key: MAPREDUCE-4462 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4462 Project: Hadoop Map/Reduce Issue Type: Bug Components: scheduler, test Reporter: Ryan Hennig Priority: Minor Labels: comments, test Fix For: 2.3.0 Attachments: MAPREDUCE-4462.patch Original Estimate: 2h Remaining Estimate: 2h While reading over the unit tests for the Fair Scheduler introduced by MAPREDUCE-3451, I added comments to make the logic of the test easier to grok quickly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3841) Broken Server metrics and Local logs link under the tools menu
[ https://issues.apache.org/jira/browse/MAPREDUCE-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-3841. Resolution: Duplicate Assignee: Jian He Closed as duplicate. Broken Server metrics and Local logs link under the tools menu -- Key: MAPREDUCE-3841 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3841 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Jian He Local logs link redirects to the cluster page and Server metrics opens an empty page on the RM/JHS homepage. So does the links from nodemanager UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He reassigned MAPREDUCE-5441: -- Assignee: Jian He JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5441: --- Attachment: (was: MAPREDUCE-5441.patch) JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5441: --- Attachment: MAPREDUCE-5441.patch JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5441: --- Attachment: MAPREDUCE-5441.patch JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He Attachments: MAPREDUCE-5441.patch When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5441: --- Status: Patch Available (was: Open) JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.0.5-alpha, 2.1.0-beta, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He Attachments: MAPREDUCE-5441.patch When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751973#comment-13751973 ] Jian He commented on MAPREDUCE-5441: Thanks [~rohithsharma] for reporting this problem. Earlier this problem is not easily reproduced on my side because at that time MR choose to ignore the Invalid AMRMToken exception after RM restarts and never explicitly sends the JOB_AM_REBOOT event and keeps alive until signally killed by NM. After that JobClient can just quickly switch to the new AM. Now MR is changed to explicitly send the JOB_AM_REBOOT event in case of Invalid AMRMToken exception(should be fixed later) and JobClient has more chance to see the ERROR state of the JOB which leads JobClient to exit prematurely. Reproduced this problem by putting long sleep in MRAppMaster.showDownJob() for the normal shutDown and MRAppMasterShutdownHook in case of signally shutDown, so that JobClient has great chance to see the ERROR state. Uploaded a patch that in case of REBOOT state of the Job return the external state as RUNNING to prevent JobClient from prematurely exiting The above manual test passed with the patch and failed without. JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He Attachments: MAPREDUCE-5441.patch When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751983#comment-13751983
]
Hadoop QA commented on MAPREDUCE-5441:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600306/MAPREDUCE-5441.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3964//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3964//console
This message is automatically generated.
JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
--
Key: MAPREDUCE-5441
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client
Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta
Reporter: Rohith Sharma K S
Assignee: Jian He
Attachments: MAPREDUCE-5441.patch
When RM issue Reboot command to app master, app master shutdown gracefully.
All the history event are writtent to hdfs with job status set as ERROR.
Jobclient get job state as ERROR and exit.
But RM launches 2nd attempt app master where no client are there to get job
status.In RM UI, job status is displayed as SUCCESS but for client Job is
Failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5460) MR AppMaster command options does not replace @taskid@ with the current task ID.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S reassigned MAPREDUCE-5460:
Assignee: Rohith Sharma K S
MR AppMaster command options does not replace @taskid@ with the current task
ID.
Key: MAPREDUCE-5460
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5460
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Rohith Sharma K S
The description of {{yarn.app.mapreduce.am.command-opts}} in
mapred-default.xml states that occurrences of {{@taskid@}} will be replaced
by the current task ID. This substitution is not happening.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752051#comment-13752051 ] Jian He commented on MAPREDUCE-5441: Good point! just came to my mind and was also just working on it :) JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He Attachments: MAPREDUCE-5441.patch When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5460) MR AppMaster command options does not replace @taskid@ with the current task ID.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752050#comment-13752050
]
Rohith Sharma K S commented on MAPREDUCE-5460:
--
As I see code, there is no validation for replacing @taskid@. Looks description
for yarn.app.mapreduce.am.command-opts is directly copied from
mapred.child.java.opts.
MR AppMaster command options does not replace @taskid@ with the current task
ID.
Key: MAPREDUCE-5460
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5460
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Rohith Sharma K S
The description of {{yarn.app.mapreduce.am.command-opts}} in
mapred-default.xml states that occurrences of {{@taskid@}} will be replaced
by the current task ID. This substitution is not happening.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5441: --- Status: Patch Available (was: Open) JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.0.5-alpha, 2.1.0-beta, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He Attachments: MAPREDUCE-5441.1.patch, MAPREDUCE-5441.patch When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5441: --- Attachment: MAPREDUCE-5441.1.patch New patch fixed the scenario when it's lastAMRetry also return FAILED state JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He Attachments: MAPREDUCE-5441.1.patch, MAPREDUCE-5441.patch When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5441: --- Status: Open (was: Patch Available) JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.0.5-alpha, 2.1.0-beta, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He Attachments: MAPREDUCE-5441.1.patch, MAPREDUCE-5441.patch When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5460) MR AppMaster command options does not replace @taskid@ with the current task ID.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752066#comment-13752066
]
Rohith Sharma K S commented on MAPREDUCE-5460:
--
This issue fix to be done from Yarn framework at the time of lauching master
container, so need to move issue to Yarn.
MR AppMaster command options does not replace @taskid@ with the current task
ID.
Key: MAPREDUCE-5460
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5460
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Rohith Sharma K S
The description of {{yarn.app.mapreduce.am.command-opts}} in
mapred-default.xml states that occurrences of {{@taskid@}} will be replaced
by the current task ID. This substitution is not happening.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752074#comment-13752074 ] Rohith Sharma K S commented on MAPREDUCE-5441: -- +1 I am OK with the patch. JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S Assignee: Jian He Attachments: MAPREDUCE-5441.1.patch, MAPREDUCE-5441.2.patch, MAPREDUCE-5441.patch When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752076#comment-13752076
]
Hadoop QA commented on MAPREDUCE-5441:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600321/MAPREDUCE-5441.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3965//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3965//console
This message is automatically generated.
JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
--
Key: MAPREDUCE-5441
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client
Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta
Reporter: Rohith Sharma K S
Assignee: Jian He
Attachments: MAPREDUCE-5441.1.patch, MAPREDUCE-5441.2.patch,
MAPREDUCE-5441.patch
When RM issue Reboot command to app master, app master shutdown gracefully.
All the history event are writtent to hdfs with job status set as ERROR.
Jobclient get job state as ERROR and exit.
But RM launches 2nd attempt app master where no client are there to get job
status.In RM UI, job status is displayed as SUCCESS but for client Job is
Failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752082#comment-13752082
]
Hadoop QA commented on MAPREDUCE-5441:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600323/MAPREDUCE-5441.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3966//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3966//console
This message is automatically generated.
JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
--
Key: MAPREDUCE-5441
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client
Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta
Reporter: Rohith Sharma K S
Assignee: Jian He
Attachments: MAPREDUCE-5441.1.patch, MAPREDUCE-5441.2.patch,
MAPREDUCE-5441.patch
When RM issue Reboot command to app master, app master shutdown gracefully.
All the history event are writtent to hdfs with job status set as ERROR.
Jobclient get job state as ERROR and exit.
But RM launches 2nd attempt app master where no client are there to get job
status.In RM UI, job status is displayed as SUCCESS but for client Job is
Failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
