[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[
https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655108#comment-13655108
]
Hadoop QA commented on MAPREDUCE-5208:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582757/MAPREDUCE-5208.20130510.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3614//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3614//console
This message is automatically generated.
> SpillRecord and ShuffleHandler should use SecureIOUtils for reading index
> file and map output
> -
>
> Key: MAPREDUCE-5208
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Omkar Vinit Joshi
> Attachments: MAPREDUCE-5208-20130506.patch,
> MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch
>
>
> ShuffleHandler (map output file) and SpillRecord (index file) are reading
> file using unsecured input stream. There exists a possibility for symlink
> attack. related to YARN-578 . Creating this issue to track map reduce changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5208: - Attachment: MAPREDUCE-5208.20130510.patch > SpillRecord and ShuffleHandler should use SecureIOUtils for reading index > file and map output > - > > Key: MAPREDUCE-5208 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5208-20130506.patch, > MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch > > > ShuffleHandler (map output file) and SpillRecord (index file) are reading > file using unsecured input stream. There exists a possibility for symlink > attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5208: - Status: Patch Available (was: Open) > SpillRecord and ShuffleHandler should use SecureIOUtils for reading index > file and map output > - > > Key: MAPREDUCE-5208 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5208-20130506.patch, > MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch > > > ShuffleHandler (map output file) and SpillRecord (index file) are reading > file using unsecured input stream. There exists a possibility for symlink > attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655091#comment-13655091 ] Omkar Vinit Joshi commented on MAPREDUCE-5208: -- Updating test case. Testing ShuffleHandler using HTTP request. Updating error message sent by the server. Instead of just sending "ERROR ON SERVER" now sending all wrapped exceptions. Avoiding explicitly to send the whole stack trace. > SpillRecord and ShuffleHandler should use SecureIOUtils for reading index > file and map output > - > > Key: MAPREDUCE-5208 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5208-20130506.patch, > MAPREDUCE-5208-20130507.patch > > > ShuffleHandler (map output file) and SpillRecord (index file) are reading > file using unsecured input stream. There exists a possibility for symlink > attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4150) Versioning and rolling upgrades for MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4150: Summary: Versioning and rolling upgrades for MR2 (was: Versioning and rolling upgrades for Yarn/MR2) > Versioning and rolling upgrades for MR2 > --- > > Key: MAPREDUCE-4150 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4150 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Ahmed Radwan >Assignee: Karthik Kambatla > > It doesn't seem that Yarn components, for example the ResourceManager or > NodeManager, do build/package version checking before trying to communicate > with each other. > The objective of this ticket is to support the following requirements / use > cases: > - New versions can be marked incompatible with old versions, and services > should be prevented from communicating with each other in such case. This > will avoid non-deterministic behavior/problems resulting from incompatible > components trying to communicate with each other. > - Permitting a policy for running different - but compatible - versions on > the same cluster (for example, in a rolling upgrade scenario). See HDFS-2983 > for the corresponding HDFS implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[
https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655050#comment-13655050
]
Hadoop QA commented on MAPREDUCE-4067:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582729/MAPREDUCE-4067.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3613//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3613//console
This message is automatically generated.
> Replace YarnRemoteException with IOException in MRv2 APIs
> -
>
> Key: MAPREDUCE-4067
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: MAPREDUCE-4067.1.patch, MAPREDUCE-4067.2.patch,
> MAPREDUCE-4067.3.patch, MAPREDUCE-4067.4.patch, MR4067.txt
>
>
> YarnRemoteException is defined as a generic wrapper for all the exceptions in
> yarn. I think we should instead throw IOExceptions in the API, which can
> later be extended for more specialized exceptions without breaking
> compatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-4067: - Attachment: MAPREDUCE-4067.4.patch Create patch based on the latest trunk > Replace YarnRemoteException with IOException in MRv2 APIs > - > > Key: MAPREDUCE-4067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Xuan Gong >Priority: Critical > Attachments: MAPREDUCE-4067.1.patch, MAPREDUCE-4067.2.patch, > MAPREDUCE-4067.3.patch, MAPREDUCE-4067.4.patch, MR4067.txt > > > YarnRemoteException is defined as a generic wrapper for all the exceptions in > yarn. I think we should instead throw IOExceptions in the API, which can > later be extended for more specialized exceptions without breaking > compatibility. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5191) TestQueue#testQueue fails with timeout on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654947#comment-13654947
]
Hitesh Shah commented on MAPREDUCE-5191:
Does it make sense to not use the temp file method in such a scenario to reduce
the time it takes to run? How about just creating a file under target/ with the
name of the test as filename? On a Mac, I saw this test run on an avg of 1
second for multiple runs.
> TestQueue#testQueue fails with timeout on Windows
> -
>
> Key: MAPREDUCE-5191
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5191
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Attachments: MAPREDUCE-5191.2.patch, MAPREDUCE-5191.patch
>
>
> Test times out on my machine after 5 seconds always on the below stack:
> {code}
> testQueue(org.apache.hadoop.mapred.TestQueue) Time elapsed: 5009 sec <<<
> ERROR!
> java.lang.Exception: test timed out after 5000 milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> sun.security.provider.SeedGenerator$ThreadedSeedGenerator.getSeedByte(SeedGenerator.java:330)
> at
> sun.security.provider.SeedGenerator$ThreadedSeedGenerator.getSeedBytes(SeedGenerator.java:319)
> at
> sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117)
> at
> sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
> at
> sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
> at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
> at java.security.SecureRandom.next(SecureRandom.java:455)
> at java.util.Random.nextLong(Random.java:284)
> at java.io.File.generateFile(File.java:1682)
> at java.io.File.createTempFile(File.java:1791)
> at java.io.File.createTempFile(File.java:1828)
> at org.apache.hadoop.mapred.TestQueue.writeFile(TestQueue.java:221)
> at org.apache.hadoop.mapred.TestQueue.testQueue(TestQueue.java:53)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-5217:
--
Resolution: Fixed
Fix Version/s: 1.3.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to branch-1.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: 1.1.2
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Fix For: 1.3.0
>
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654910#comment-13654910
]
Siddharth Seth commented on MAPREDUCE-5217:
---
+1 for the branch-1 patch. Committing based on the comments. Thanks Venkat for
the patch, and Akejandro, Daryn and Jitendra for taking a look.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: 1.1.2
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5239) Update MR App to reflect YarnRemoteException changes after YARN-634
[ https://issues.apache.org/jira/browse/MAPREDUCE-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654903#comment-13654903 ] Hudson commented on MAPREDUCE-5239: --- Integrated in Hadoop-trunk-Commit #3741 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3741/]) YARN-634. Modified YarnRemoteException to be not backed by PB and introduced a separate SerializedException record. Contributed by Siddharth Seth. MAPREDUCE-5239. Updated MR App to reflect YarnRemoteException changes after YARN-634. Contributed by Siddharth Seth. (Revision 1481205) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481205 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/YarnRemoteException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/impl * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/YarnRemoteExceptionFactoryPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/YarnRemoteExceptionFactoryProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/RPCUtil.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/SerializedException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/SerializedExceptionPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/LocalResourceStatus.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/impl/pb/LocalResourceStatusPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerResourceFailedEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalizedResource.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/se
[jira] [Resolved] (MAPREDUCE-5239) Update MR App to reflect YarnRemoteException changes after YARN-634
[ https://issues.apache.org/jira/browse/MAPREDUCE-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5239. Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Committed this to trunk and branch-2 together with YARN-634. Closing this. > Update MR App to reflect YarnRemoteException changes after YARN-634 > --- > > Key: MAPREDUCE-5239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5239 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5239.txt > > > YARN-634 is making YarnRemoteException to be not backed by PB anymore. Need > some MR changes because of that. Tracking MR changes from the patch at > YARN-634. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5018) Support raw binary data with Hadoop streaming
[
https://issues.apache.org/jira/browse/MAPREDUCE-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654863#comment-13654863
]
Jay Hacker commented on MAPREDUCE-5018:
---
[~pratem], you're right, there are cases where it's not efficient. Consider
this though: if you have 100 TB of files in HDFS that you want to md5sum (or
what have you), would you rather do an inefficient distributed md5sum on the
cluster, or copy 100 TB out to a single machine and wait for a single md5sum?
Can you even fit that on one machine?
You still gain reliability: there are multiple copies of each file, and failed
jobs get restarted. It's also just convenient.
Here's the trick to make it efficient: use many files, and set the block size
of individual files big enough to fit the whole file:
{{hadoop fs -D dfs.block.size=1073741824 -put ...}}
Then all reads are local, and you get all the performance Hadoop can give you.
> Support raw binary data with Hadoop streaming
> -
>
> Key: MAPREDUCE-5018
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5018
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/streaming
>Reporter: Jay Hacker
>Priority: Minor
> Attachments: justbytes.jar, MAPREDUCE-5018.patch, mapstream
>
>
> People often have a need to run older programs over many files, and turn to
> Hadoop streaming as a reliable, performant batch system. There are good
> reasons for this:
> 1. Hadoop is convenient: they may already be using it for mapreduce jobs, and
> it is easy to spin up a cluster in the cloud.
> 2. It is reliable: HDFS replicates data and the scheduler retries failed jobs.
> 3. It is reasonably performant: it moves the code to the data, maintaining
> locality, and scales with the number of nodes.
> Historically Hadoop is of course oriented toward processing key/value pairs,
> and so needs to interpret the data passing through it. Unfortunately, this
> makes it difficult to use Hadoop streaming with programs that don't deal in
> key/value pairs, or with binary data in general. For example, something as
> simple as running md5sum to verify the integrity of files will not give the
> correct result, due to Hadoop's interpretation of the data.
> There have been several attempts at binary serialization schemes for Hadoop
> streaming, such as TypedBytes (HADOOP-1722); however, these are still aimed
> at efficiently encoding key/value pairs, and not passing data through
> unmodified. Even the "RawBytes" serialization scheme adds length fields to
> the data, rendering it not-so-raw.
> I often have a need to run a Unix filter on files stored in HDFS; currently,
> the only way I can do this on the raw data is to copy the data out and run
> the filter on one machine, which is inconvenient, slow, and unreliable. It
> would be very convenient to run the filter as a map-only job, allowing me to
> build on existing (well-tested!) building blocks in the Unix tradition
> instead of reimplementing them as mapreduce programs.
> However, most existing tools don't know about file splits, and so want to
> process whole files; and of course many expect raw binary input and output.
> The solution is to run a map-only job with an InputFormat and OutputFormat
> that just pass raw bytes and don't split. It turns out to be a little more
> complicated with streaming; I have attached a patch with the simplest
> solution I could come up with. I call the format "JustBytes" (as "RawBytes"
> was already taken), and it should be usable with most recent versions of
> Hadoop.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654854#comment-13654854
]
Alejandro Abdelnur commented on MAPREDUCE-5217:
---
[~daryn], yep this is for hadoop 1, in Hadoop 2 distcp works just fine.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: 1.1.2
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-5217:
--
Affects Version/s: (was: 2.0.4-alpha)
(was: trunk)
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: 1.1.2
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654794#comment-13654794 ] Sandy Ryza commented on MAPREDUCE-5220: --- Agreed that users shouldn't need to set these variables. protected seems reasonable to me too, but the fact still stands that they were public in MR1, and some user code somewhere conceivably could have used them, and will need to be changed. Maybe it would make sense to change them to public, but deprecate them and mark them as @Private? > Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 > > > Key: MAPREDUCE-5220 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654767#comment-13654767
]
Daryn Sharp commented on MAPREDUCE-5217:
Boilerplate code like this is _wrong_, at least for versions later than 1.x.
It's unreasonable to expect every tool that might be run within the context of
a job to be aware of the internal implementation details of both the UGI (its
env var) and mapreduce.job.credentials.binary. I'd personally like to see the
conf key removed.
If a tools wants to control its token set, my opinion is it should create a new
UGI for the submission. The vast majority of tools, like distcp, have no need
for special logic just in case its running within a job.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated MAPREDUCE-5232: Assignee: Sangjin Lee > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654739#comment-13654739 ] Lohit Vijayarenu commented on MAPREDUCE-5232: - +1 Patch looks good to me > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee > Attachments: MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[
https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654721#comment-13654721
]
Hadoop QA commented on MAPREDUCE-5232:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582659/MAPREDUCE-5232-branch-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3612//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3612//console
This message is automatically generated.
> log classpath and other key properties on child JVM start
> -
>
> Key: MAPREDUCE-5232
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Sangjin Lee
> Attachments: MAPREDUCE-5232-branch-2.patch,
> MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch
>
>
> It would be great if we log vital information such as classpath, etc. upon a
> mapreduce child JVM start. This would help a great deal in terms of
> troubleshooting classpath issues, etc. Today it is pretty difficult to debug
> this unless you preserve the container script.
> Maybe it can log things like classpath, os name/version, java version, etc.
> at the beginning of the child JVM start.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal reassigned MAPREDUCE-5233: Assignee: Mayank Bansal (was: Zhijie Shen) > Functions are changed or removed from Job in jobcontrol > --- > > Key: MAPREDUCE-5233 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Fix For: 2.0.5-beta > > > The functions are removed from Job in jobcontrol: > 1. setMapredJobID(String) > 2. setState(int) > The function signatures are changed: > 1. addDependingJob(ControlledJob) > 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654719#comment-13654719 ] Mayank Bansal commented on MAPREDUCE-5233: -- Taking it over. Thanks, Mayank > Functions are changed or removed from Job in jobcontrol > --- > > Key: MAPREDUCE-5233 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Fix For: 2.0.5-beta > > > The functions are removed from Job in jobcontrol: > 1. setMapredJobID(String) > 2. setState(int) > The function signatures are changed: > 1. addDependingJob(ControlledJob) > 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5095) TestShuffleExceptionCount#testCheckException fails occasionally with JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654710#comment-13654710 ] Hitesh Shah commented on MAPREDUCE-5095: Should abortCalled also be changed to a non-static? > TestShuffleExceptionCount#testCheckException fails occasionally with JDK7 > - > > Key: MAPREDUCE-5095 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5095 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.1.2 > Environment: Open JDK7 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5095.patch > > Original Estimate: 1h > Time Spent: 1h > Remaining Estimate: 0h > > The test fails due a test-order dependency that can be violated when running > with JDK 7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654698#comment-13654698 ] Sangjin Lee commented on MAPREDUCE-5232: Updated the patches. > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee > Attachments: MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-5232: --- Attachment: MAPREDUCE-5232-branch-2.patch > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee > Attachments: MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-5232: --- Attachment: MAPREDUCE-5232.patch > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee > Attachments: MAPREDUCE-5232-branch-2.patch, > MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5239) Update MR App to reflect YarnRemoteException changes after YARN-634
[ https://issues.apache.org/jira/browse/MAPREDUCE-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5239: --- Attachment: MAPREDUCE-5239.txt Attaching MR changes from YARN-634 on behalf of Sid. Reviewing these changes together with those on that JIRA. > Update MR App to reflect YarnRemoteException changes after YARN-634 > --- > > Key: MAPREDUCE-5239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5239 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Attachments: MAPREDUCE-5239.txt > > > YARN-634 is making YarnRemoteException to be not backed by PB anymore. Need > some MR changes because of that. Tracking MR changes from the patch at > YARN-634. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5239) Update MR App to reflect YarnRemoteException changes after YARN-634
Vinod Kumar Vavilapalli created MAPREDUCE-5239: -- Summary: Update MR App to reflect YarnRemoteException changes after YARN-634 Key: MAPREDUCE-5239 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5239 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth YARN-634 is making YarnRemoteException to be not backed by PB anymore. Need some MR changes because of that. Tracking MR changes from the patch at YARN-634. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654643#comment-13654643
]
Alejandro Abdelnur commented on MAPREDUCE-5217:
---
Sidd, you are correct, the patch for branch-1 is needed (as branch-1 has distcp
v1), for branch-2 it is not needed. THX
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654619#comment-13654619 ] Sangjin Lee commented on MAPREDUCE-5232: Thanks [~lohit]. I agree having a single config may be fine. I'll update the patches. > log classpath and other key properties on child JVM start > - > > Key: MAPREDUCE-5232 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee > Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch > > > It would be great if we log vital information such as classpath, etc. upon a > mapreduce child JVM start. This would help a great deal in terms of > troubleshooting classpath issues, etc. Today it is pretty difficult to debug > this unless you preserve the container script. > Maybe it can log things like classpath, os name/version, java version, etc. > at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654606#comment-13654606 ] Sandy Ryza commented on MAPREDUCE-5130: --- Thanks for taking a look, Vinod. bq. Can you please file a ticket if the test is failing on trunk too? Tx. Filed MAPREDUCE-5236 bq. mapreduce.job.jvm.numtasks isn't supported in MR over YARN. Oops. Will take this out. bq. Instead of removing code for computeNumSlotsPer*, you can make it work with simple modifications? computeNumSlotsPer* isn't used anywhere in the code. As there are no more slots in YARN, and it is not a public API, would it not make the most sense to remove it? I could fix the test here and remove it later in MAPREDUCE-5236 if that would be better? > Add missing job config options to mapred-default.xml > > > Key: MAPREDUCE-5130 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, > MAPREDUCE-5130-2.patch, MAPREDUCE-5130.patch > > > I came across that mapreduce.map.child.java.opts and > mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do > a fuller sweep to see what else is missing before posting a patch. > List so far: > mapreduce.map/reduce.child.java.opts > mapreduce.map/reduce.memory.mb > mapreduce.job.jvm.numtasks > mapreduce.input.lineinputformat.linespermap > mapreduce.task.combine.progress.records -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5238) TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandy Ryza updated MAPREDUCE-5238:
--
Description:
{noformat}
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.794 sec <<<
FAILURE!
testGenerateDistCacheData(org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation)
Time elapsed: 16767 sec <<< FAILURE!
java.lang.AssertionError: Wrong permissions for distributed cache file
/user/sandy/testSetupGenerateDistCacheData/distributedCache/26046e44bfac7cec1afce2cef15ee281
expected: but was:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheFiles(TestDistCacheEmulation.java:136)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheData(TestDistCacheEmulation.java:109)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.testGenerateDistCacheData(TestDistCacheEmulation.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
was:
{noformat
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.794 sec <<<
FAILURE!
testGenerateDistCacheData(org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation)
Time elapsed: 16767 sec <<< FAILURE!
java.lang.AssertionError: Wrong permissions for distributed cache file
/user/sandy/testSetupGenerateDistCacheData/distributedCache/26046e44bfac7cec1afce2cef15ee281
expected: but was:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheFiles(TestDistCacheEmulation.java:136)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheData(TestDistCacheEmulation.java:109)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.testGenerateDistCacheData(TestDistCacheEmulation.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
> TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk
>
>
> Key: MAPREDUCE-5238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.4-alpha
>Reporter: Sandy Ryza
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.794 sec
> <<< FAILURE!
> testGenerateDistCacheData(org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation)
> Time elapsed: 16767 sec <<< FAILURE!
> java.lang.AssertionError: Wrong permissions for distributed cache file
> /user/sandy/testSetupGenerateDistCacheData/distributedCache/26046e44bfac7cec1afce2cef15ee281
> expected: but was:
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at
> org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheFiles(TestDistCacheEmulation.java:136)
> at
> org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheData(TestDistCacheE
[jira] [Created] (MAPREDUCE-5238) TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk
Sandy Ryza created MAPREDUCE-5238:
-
Summary: TestDistCacheEmulation.testGenerateDistCacheData is
failing in trunk
Key: MAPREDUCE-5238
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5238
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
{noformat
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.794 sec <<<
FAILURE!
testGenerateDistCacheData(org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation)
Time elapsed: 16767 sec <<< FAILURE!
java.lang.AssertionError: Wrong permissions for distributed cache file
/user/sandy/testSetupGenerateDistCacheData/distributedCache/26046e44bfac7cec1afce2cef15ee281
expected: but was:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheFiles(TestDistCacheEmulation.java:136)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheData(TestDistCacheEmulation.java:109)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.testGenerateDistCacheData(TestDistCacheEmulation.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654491#comment-13654491
]
Hudson commented on MAPREDUCE-4942:
---
Integrated in Hadoop-Mapreduce-trunk #1421 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1421/])
MAPREDUCE-4942. mapreduce.Job has a bunch of methods that throw
InterruptedException so its incompatible with MR1. (rkanter via tucu) (Revision
1480748)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1480748
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java
> mapreduce.Job has a bunch of methods that throw InterruptedException so its
> incompatible with MR1
> -
>
> Key: MAPREDUCE-4942
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.2-alpha
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.0.5-beta
>
> Attachments: MAPREDUCE-4942.patch
>
>
> The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
> {{InterruptedException}} but don't in MR1, which makes them incompatible.
> (Their Javadoc comments are also missing that they throw an
> {{InterruptedException}} anyway)
> I propose that we wrap the {{InterruptedException}} in a
> {{RuntimeException}}.
> {code}
> public float setupProgress() throws IOException, InterruptedException
> public float mapProgress() throws IOException, InterruptedException
> public float reduceProgress() throws IOException, InterruptedException
> public boolean isComplete() throws IOException, InterruptedException
> public boolean isSuccessful() throws IOException, InterruptedException
> public void killJob() throws IOException, InterruptedException
> public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
> IOException, InterruptedException
> public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
> IOException, InterruptedException
> public Counters getCounters() throws IOException, InterruptedException
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654447#comment-13654447
]
Hudson commented on MAPREDUCE-4942:
---
Integrated in Hadoop-Hdfs-trunk #1394 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1394/])
MAPREDUCE-4942. mapreduce.Job has a bunch of methods that throw
InterruptedException so its incompatible with MR1. (rkanter via tucu) (Revision
1480748)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1480748
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java
> mapreduce.Job has a bunch of methods that throw InterruptedException so its
> incompatible with MR1
> -
>
> Key: MAPREDUCE-4942
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.2-alpha
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.0.5-beta
>
> Attachments: MAPREDUCE-4942.patch
>
>
> The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
> {{InterruptedException}} but don't in MR1, which makes them incompatible.
> (Their Javadoc comments are also missing that they throw an
> {{InterruptedException}} anyway)
> I propose that we wrap the {{InterruptedException}} in a
> {{RuntimeException}}.
> {code}
> public float setupProgress() throws IOException, InterruptedException
> public float mapProgress() throws IOException, InterruptedException
> public float reduceProgress() throws IOException, InterruptedException
> public boolean isComplete() throws IOException, InterruptedException
> public boolean isSuccessful() throws IOException, InterruptedException
> public void killJob() throws IOException, InterruptedException
> public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
> IOException, InterruptedException
> public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
> IOException, InterruptedException
> public Counters getCounters() throws IOException, InterruptedException
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654362#comment-13654362
]
Hudson commented on MAPREDUCE-4942:
---
Integrated in Hadoop-Yarn-trunk #205 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/205/])
MAPREDUCE-4942. mapreduce.Job has a bunch of methods that throw
InterruptedException so its incompatible with MR1. (rkanter via tucu) (Revision
1480748)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1480748
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java
> mapreduce.Job has a bunch of methods that throw InterruptedException so its
> incompatible with MR1
> -
>
> Key: MAPREDUCE-4942
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.2-alpha
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.0.5-beta
>
> Attachments: MAPREDUCE-4942.patch
>
>
> The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
> {{InterruptedException}} but don't in MR1, which makes them incompatible.
> (Their Javadoc comments are also missing that they throw an
> {{InterruptedException}} anyway)
> I propose that we wrap the {{InterruptedException}} in a
> {{RuntimeException}}.
> {code}
> public float setupProgress() throws IOException, InterruptedException
> public float mapProgress() throws IOException, InterruptedException
> public float reduceProgress() throws IOException, InterruptedException
> public boolean isComplete() throws IOException, InterruptedException
> public boolean isSuccessful() throws IOException, InterruptedException
> public void killJob() throws IOException, InterruptedException
> public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
> IOException, InterruptedException
> public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
> IOException, InterruptedException
> public Counters getCounters() throws IOException, InterruptedException
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[
https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653642#comment-13653642
]
Hadoop QA commented on MAPREDUCE-4067:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582593/MAPREDUCE-4067.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3611//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3611//console
This message is automatically generated.
> Replace YarnRemoteException with IOException in MRv2 APIs
> -
>
> Key: MAPREDUCE-4067
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: MAPREDUCE-4067.1.patch, MAPREDUCE-4067.2.patch,
> MAPREDUCE-4067.3.patch, MR4067.txt
>
>
> YarnRemoteException is defined as a generic wrapper for all the exceptions in
> yarn. I think we should instead throw IOExceptions in the API, which can
> later be extended for more specialized exceptions without breaking
> compatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4150) Versioning and rolling upgrades for Yarn/MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653638#comment-13653638 ] Karthik Kambatla commented on MAPREDUCE-4150: - Hi Sid Thanks for reaching out. I have some ideas, but haven't had a chance to work on it. Agree we will need changes in both YARN and MR. Please go ahead and create a YARN JIRA to address the YARN part, keen to see your write up and contribute. > Versioning and rolling upgrades for Yarn/MR2 > > > Key: MAPREDUCE-4150 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4150 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Ahmed Radwan >Assignee: Karthik Kambatla > > It doesn't seem that Yarn components, for example the ResourceManager or > NodeManager, do build/package version checking before trying to communicate > with each other. > The objective of this ticket is to support the following requirements / use > cases: > - New versions can be marked incompatible with old versions, and services > should be prevented from communicating with each other in such case. This > will avoid non-deterministic behavior/problems resulting from incompatible > components trying to communicate with each other. > - Permitting a policy for running different - but compatible - versions on > the same cluster (for example, in a rolling upgrade scenario). See HDFS-2983 > for the corresponding HDFS implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4150) Versioning and rolling upgrades for Yarn/MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653622#comment-13653622 ] Siddharth Seth commented on MAPREDUCE-4150: --- Karthik, do you intend to use this jira for work related to MR or YARN ? I'll create a separate JIRA for YARN / move this one over accordingly. Both are likely to require some changes. I have a writeup on YARN rolling upgrades, which I'll post tomorrow. Should be able to club that along with what you have for rolling upgrades to get this started. > Versioning and rolling upgrades for Yarn/MR2 > > > Key: MAPREDUCE-4150 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4150 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Ahmed Radwan >Assignee: Karthik Kambatla > > It doesn't seem that Yarn components, for example the ResourceManager or > NodeManager, do build/package version checking before trying to communicate > with each other. > The objective of this ticket is to support the following requirements / use > cases: > - New versions can be marked incompatible with old versions, and services > should be prevented from communicating with each other in such case. This > will avoid non-deterministic behavior/problems resulting from incompatible > components trying to communicate with each other. > - Permitting a policy for running different - but compatible - versions on > the same cluster (for example, in a rolling upgrade scenario). See HDFS-2983 > for the corresponding HDFS implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-4067: - Attachment: MAPREDUCE-4067.3.patch Fix the test case failure > Replace YarnRemoteException with IOException in MRv2 APIs > - > > Key: MAPREDUCE-4067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Xuan Gong >Priority: Critical > Attachments: MAPREDUCE-4067.1.patch, MAPREDUCE-4067.2.patch, > MAPREDUCE-4067.3.patch, MR4067.txt > > > YarnRemoteException is defined as a generic wrapper for all the exceptions in > yarn. I think we should instead throw IOExceptions in the API, which can > later be extended for more specialized exceptions without breaking > compatibility. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[
https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653600#comment-13653600
]
Hadoop QA commented on MAPREDUCE-4067:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582580/MAPREDUCE-4067.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.app.local.TestLocalContainerAllocator
org.apache.hadoop.mapred.TestClientServiceDelegate
org.apache.hadoop.mapreduce.TestMRJobClient
org.apache.hadoop.mapreduce.security.TestJHSSecurity
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3610//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3610//console
This message is automatically generated.
> Replace YarnRemoteException with IOException in MRv2 APIs
> -
>
> Key: MAPREDUCE-4067
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: MAPREDUCE-4067.1.patch, MAPREDUCE-4067.2.patch,
> MR4067.txt
>
>
> YarnRemoteException is defined as a generic wrapper for all the exceptions in
> yarn. I think we should instead throw IOExceptions in the API, which can
> later be extended for more specialized exceptions without breaking
> compatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
