[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652783#comment-13652783
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582430/MAPREDUCE-5217-branch-1.patch.3
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3603//console
This message is automatically generated.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-branch-1.patch.3
Patch for branch1
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5226) Handle exception related changes in YARN's AMRMProtocol api after YARN-630
[ https://issues.apache.org/jira/browse/MAPREDUCE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652762#comment-13652762 ] Hudson commented on MAPREDUCE-5226: --- Integrated in Hadoop-trunk-Commit #3724 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3724/]) YARN-630. Changed AMRMProtocol api to throw IOException and YarnRemoteException. Contributed by Xuan Gong. MAPREDUCE-5226. Handling YarnRemoteException separately from IOException in MR App's use of AMRMProtocol after YARN-630. Contributed by Xuan Gong. (Revision 1480529) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1480529 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/AMRMProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/AMRMProtocolPBClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/AMRMProtocolPBServiceImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java > Handle exception related changes in YARN's AMRMProtocol api after YARN-630 > -- > > Key: MAPREDUCE-5226 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.0.5-beta > > Attachments: MR-5226.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5226) Handle exception related changes in YARN's AMRMProtocol api after YARN-630
[ https://issues.apache.org/jira/browse/MAPREDUCE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5226. Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Reviewed and committed this together with YARN-630 to trunk and branch-2. Thanks Xuan! > Handle exception related changes in YARN's AMRMProtocol api after YARN-630 > -- > > Key: MAPREDUCE-5226 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.0.5-beta > > Attachments: MR-5226.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5225) SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits
[
https://issues.apache.org/jira/browse/MAPREDUCE-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652750#comment-13652750
]
Hadoop QA commented on MAPREDUCE-5225:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582421/MAPREDUCE-5225.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3602//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3602//console
This message is automatically generated.
> SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits
> ---
>
> Key: MAPREDUCE-5225
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5225
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: MAPREDUCE-5225.1.patch
>
>
> Now, SplitSampler only samples the first maxSplitsSampled splits, caused by
> MAPREDUCE-1820. However, jumping around all splits is in general preferable
> than the first N splits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5226) Handle exception related changes in YARN's AMRMProtocol api after YARN-630
[ https://issues.apache.org/jira/browse/MAPREDUCE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5226: --- Summary: Handle exception related changes in YARN's AMRMProtocol api after YARN-630 (was: Change AMRMProtocol api to throw IOException and YarnRemoteException in MRv2) > Handle exception related changes in YARN's AMRMProtocol api after YARN-630 > -- > > Key: MAPREDUCE-5226 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: MR-5226.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5225) SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5225: --- Attachment: MAPREDUCE-5225.1.patch The patch makes SplitSampler jump by sampling step when it does sampling. Then, the behavior of SplitSampler in both mapred and mapreduce is identical. > SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits > --- > > Key: MAPREDUCE-5225 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5225 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5225.1.patch > > > Now, SplitSampler only samples the first maxSplitsSampled splits, caused by > MAPREDUCE-1820. However, jumping around all splits is in general preferable > than the first N splits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5225) SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5225: --- Status: Patch Available (was: Open) > SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits > --- > > Key: MAPREDUCE-5225 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5225 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5225.1.patch > > > Now, SplitSampler only samples the first maxSplitsSampled splits, caused by > MAPREDUCE-1820. However, jumping around all splits is in general preferable > than the first N splits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652692#comment-13652692
]
Hadoop QA commented on MAPREDUCE-5156:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582387/MAPREDUCE-5156.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-mapreduce-examples:
org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3601//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3601//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3601//console
This message is automatically generated.
> Hadoop-examples-1.x.x.jar cannot run on Yarn
>
>
> Key: MAPREDUCE-5156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Blocker
> Attachments: MAPREDUCE-5156.1.patch, MAPREDUCE-5156.2.patch,
> MAPREDUCE-5156.3.patch
>
>
> M/R examples are run through ProgramDriver.driver. ProgramDriver.driver
> returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the
> function signatures in the example jar and yarn are incompatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsInst should standardize naming rules
[ https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5227: -- Priority: Minor (was: Major) > JobTrackerMetricsInst should standardize naming rules > - > > Key: MAPREDUCE-5227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 1.1.2 >Reporter: Tsuyoshi OZAWA >Assignee: Masahiro Nakagawa >Priority: Minor > > JobTrackerMetricsInst provides users with some metrics, but its naming rules( > "jobs_running", "running_maps", "running_reduces") sometimes confuses users. > It should be standardized. > One concern is backward compatibility, so one idea is to share > MetricMutableGaugeInt object from old and new property name. > e.g. to share runningMaps from "running_maps" and "maps_running". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5227) JobTrackerMetricsInst should standardize naming rules
Tsuyoshi OZAWA created MAPREDUCE-5227: - Summary: JobTrackerMetricsInst should standardize naming rules Key: MAPREDUCE-5227 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.1.2 Reporter: Tsuyoshi OZAWA Assignee: Masahiro Nakagawa JobTrackerMetricsInst provides users with some metrics, but its naming rules( "jobs_running", "running_maps", "running_reduces") sometimes confuses users. It should be standardized. One concern is backward compatibility, so one idea is to share MetricMutableGaugeInt object from old and new property name. e.g. to share runningMaps from "running_maps" and "maps_running". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652580#comment-13652580
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582389/MAPREDUCE-5217-trunk.patch.3
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3600//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3600//console
This message is automatically generated.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2,
> MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5184) Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652558#comment-13652558 ] Zhijie Shen commented on MAPREDUCE-5184: In the document, it is worth mentioning that since hadoop-examples-2.x.jar is the classpath, "hadoop -jar hadoop-examples-1.x.jar"is still using the 2.x jar. Users should either remove 2.x jar from the classpath or set HADOOP_USER_CLASSPATH_FIRST=true and HADOOP_CLASSPATH=...:hadoop-examples-1.x.jar to run 1.x jar actually. > Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2 > > > Key: MAPREDUCE-5184 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5184 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 2.0.5-beta > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5156: --- Status: Patch Available (was: Open) > Hadoop-examples-1.x.x.jar cannot run on Yarn > > > Key: MAPREDUCE-5156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5156.1.patch, MAPREDUCE-5156.2.patch, > MAPREDUCE-5156.3.patch > > > M/R examples are run through ProgramDriver.driver. ProgramDriver.driver > returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the > function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652545#comment-13652545
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
Thanks [~sseth]. I have updated the trunk patch as you suggested and also
changed the variable name to tokenFile as suggested by [~jnp].
I am uploading the trunk patch first and will upload the branch-1 patch with
new variable name after precommit tests with trunk patch are done
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2,
> MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-trunk.patch.3
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2,
> MAPREDUCE-5217-trunk.patch.3
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5156: --- Attachment: MAPREDUCE-5156.3.patch Try to fix the findbugs warning > Hadoop-examples-1.x.x.jar cannot run on Yarn > > > Key: MAPREDUCE-5156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5156.1.patch, MAPREDUCE-5156.2.patch, > MAPREDUCE-5156.3.patch > > > M/R examples are run through ProgramDriver.driver. ProgramDriver.driver > returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the > function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5226) Change AMRMProtocol api to throw IOException and YarnRemoteException in MRv2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5226: - Attachment: MR-5226.1.patch > Change AMRMProtocol api to throw IOException and YarnRemoteException in MRv2 > > > Key: MAPREDUCE-5226 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: MR-5226.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5226) Change AMRMProtocol api to throw IOException and YarnRemoteException in MRv2
Xuan Gong created MAPREDUCE-5226: Summary: Change AMRMProtocol api to throw IOException and YarnRemoteException in MRv2 Key: MAPREDUCE-5226 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652518#comment-13652518 ] Jason Lowe commented on MAPREDUCE-5211: --- I believe 1.x also has a similar behavior where it will concatenate the absolute path of the first input path to the temporary directory. Therefore if we have multi-level merges then the paths will be increasingly lengthy as it keeps tacking on the temporary directory to the input path. > Reducer intermediate files can collide during merge > --- > > Key: MAPREDUCE-5211 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Attachments: MAPREDUCE-5211.branch-0.23.patch > > > The OnDiskMerger.merge method constructs an output path that is not unique to > a reduce attempt, and as a result can result in a file collision with other > reducers from the same app that are running on the same node. In addition > the name of the output file is based on MapOutput.toString which may not be > unique in light of multi-pass merges on disk since the mapId will be null and > the basename ends up as "MapOutput(null, DISK)" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652517#comment-13652517
]
Hadoop QA commented on MAPREDUCE-5130:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582372/MAPREDUCE-5130-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1368 javac
compiler warnings (more than the trunk's current 1366 warnings).
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3599//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3599//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3599//console
This message is automatically generated.
> Add missing job config options to mapred-default.xml
>
>
> Key: MAPREDUCE-5130
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.0.4-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch,
> MAPREDUCE-5130.patch
>
>
> I came across that mapreduce.map.child.java.opts and
> mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do
> a fuller sweep to see what else is missing before posting a patch.
> List so far:
> mapreduce.map/reduce.child.java.opts
> mapreduce.map/reduce.memory.mb
> mapreduce.job.jvm.numtasks
> mapreduce.input.lineinputformat.linespermap
> mapreduce.task.combine.progress.records
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5211: - Priority: Blocker (was: Critical) > Reducer intermediate files can collide during merge > --- > > Key: MAPREDUCE-5211 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Attachments: MAPREDUCE-5211.branch-0.23.patch > > > The OnDiskMerger.merge method constructs an output path that is not unique to > a reduce attempt, and as a result can result in a file collision with other > reducers from the same app that are running on the same node. In addition > the name of the output file is based on MapOutput.toString which may not be > unique in light of multi-pass merges on disk since the mapId will be null and > the basename ends up as "MapOutput(null, DISK)" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652508#comment-13652508 ] Arun C Murthy commented on MAPREDUCE-5211: -- +1 for the branch-0.23 patch. Looks like this existed forever in trunk and got accidentally fixed by MAPREDUCE-2264 which introduced CompressAwarePath which extends Path i.e. with a correct impl of toString - hence the bug in MapOutput.toString got fixed accidentally. However, the fix in trunk is truly accidental, and we'll get paths which have the suffix ".merge" for each level of merge etc. repeated. So, we should apply a similar fix to trunk too. > Reducer intermediate files can collide during merge > --- > > Key: MAPREDUCE-5211 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-5211.branch-0.23.patch > > > The OnDiskMerger.merge method constructs an output path that is not unique to > a reduce attempt, and as a result can result in a file collision with other > reducers from the same app that are running on the same node. In addition > the name of the output file is based on MapOutput.toString which may not be > unique in light of multi-pass merges on disk since the mapId will be null and > the basename ends up as "MapOutput(null, DISK)" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5212) Handle exception related changes in YARN's ClientRMProtocol api after YARN-631
[ https://issues.apache.org/jira/browse/MAPREDUCE-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652486#comment-13652486 ] Hudson commented on MAPREDUCE-5212: --- Integrated in Hadoop-trunk-Commit #3723 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3723/]) YARN-631. Changed ClientRMProtocol api to throw IOException and YarnRemoteException. Contributed by Xuan Gong. MAPREDUCE-5212. Handling YarnRemoteException separately from IOException in MR App after YARN-631. Contributed by Xuan Gong. (Revision 1480494) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1480494 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ClientRMProtocolPBClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ClientRMProtocolPBServiceImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java > Handle exception related changes in YARN's ClientRMProtocol api after YARN-631 > -- > > Key: MAPREDUCE-5212 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5212 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.0.5-beta > > Attachments: MR-5212.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5219) JobStatus#getJobPriority changed to JobStatus#getPriority in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652470#comment-13652470 ] Sandy Ryza commented on MAPREDUCE-5219: --- Checked again and mapreduce.JobStatus doesn't even have a getJobPriority method in MR1. Marking this as invalid. > JobStatus#getJobPriority changed to JobStatus#getPriority in MR2 > > > Key: MAPREDUCE-5219 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5219 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5219-1.patch, MAPREDUCE-5219-1.patch, > MAPREDUCE-5219.patch > > > We should change it back for compatibility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5219) JobStatus#getJobPriority changed to JobStatus#getPriority in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5219: -- Resolution: Invalid Status: Resolved (was: Patch Available) > JobStatus#getJobPriority changed to JobStatus#getPriority in MR2 > > > Key: MAPREDUCE-5219 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5219 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5219-1.patch, MAPREDUCE-5219-1.patch, > MAPREDUCE-5219.patch > > > We should change it back for compatibility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5212) Handle exception related changes in YARN's ClientRMProtocol api after YARN-631
[ https://issues.apache.org/jira/browse/MAPREDUCE-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5212. Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Been reviewing this at YARN-631. Committed this to trunk and branch-2 together with YARN-631. > Handle exception related changes in YARN's ClientRMProtocol api after YARN-631 > -- > > Key: MAPREDUCE-5212 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5212 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.0.5-beta > > Attachments: MR-5212.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5212) Handle exception related changes in YARN's ClientRMProtocol api after YARN-631
[ https://issues.apache.org/jira/browse/MAPREDUCE-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5212: --- Summary: Handle exception related changes in YARN's ClientRMProtocol api after YARN-631 (was: Change ClientRMProtocol api to throw IOException and YarnRemoteException in MR api) > Handle exception related changes in YARN's ClientRMProtocol api after YARN-631 > -- > > Key: MAPREDUCE-5212 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5212 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: MR-5212.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5038) old API CombineFileInputFormat missing fixes that are in new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652443#comment-13652443 ] Arun C Murthy commented on MAPREDUCE-5038: -- I ran the test on hive-0.11, so yes, it includes HIVE-3338. Thanks for checking on this [~sandyr] > old API CombineFileInputFormat missing fixes that are in new API > - > > Key: MAPREDUCE-5038 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5038 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5038-1.patch, MAPREDUCE-5038.patch, > MAPREDUCE-5038-revised-1.patch, MAPREDUCE-5038-revised-1.patch, > MAPREDUCE-5038-revised.patch > > > The following changes patched the CombineFileInputFormat in mapreduce, but > neglected the one in mapred > MAPREDUCE-1597 enabled the CombineFileInputFormat to work on splittable files > MAPREDUCE-2021 solved returning duplicate hostnames in split locations > MAPREDUCE-1806 CombineFileInputFormat does not work with paths not on default > FS > In trunk this is not an issue as the one in mapred extends the one in > mapreduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Attachment: MAPREDUCE-5130-1.patch > Add missing job config options to mapred-default.xml > > > Key: MAPREDUCE-5130 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, > MAPREDUCE-5130.patch > > > I came across that mapreduce.map.child.java.opts and > mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do > a fuller sweep to see what else is missing before posting a patch. > List so far: > mapreduce.map/reduce.child.java.opts > mapreduce.map/reduce.memory.mb > mapreduce.job.jvm.numtasks > mapreduce.input.lineinputformat.linespermap > mapreduce.task.combine.progress.records -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652425#comment-13652425 ] Siddharth Seth commented on MAPREDUCE-5211: --- Jason, the 0.23 patch looks ok. Maybe we should add a utility method to generate intermediate output file names. I think a variant of this does exist in trunk as well. The OnDiskMerger it's first inputPath as the outputPath - which means it could end up overwriting this file in a multi-stage merge ? > Reducer intermediate files can collide during merge > --- > > Key: MAPREDUCE-5211 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-5211.branch-0.23.patch > > > The OnDiskMerger.merge method constructs an output path that is not unique to > a reduce attempt, and as a result can result in a file collision with other > reducers from the same app that are running on the same node. In addition > the name of the output file is based on MapOutput.toString which may not be > unique in light of multi-pass merges on disk since the mapId will be null and > the basename ends up as "MapOutput(null, DISK)" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652416#comment-13652416
]
Siddharth Seth commented on MAPREDUCE-5217:
---
Venkat, the patch looks good. Couple of minor nits though.
- In the trunk patch, the credentials constant can refer to
MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY
- Rename delegationTokenFiles to delegationTokenFile
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652398#comment-13652398
]
Hadoop QA commented on MAPREDUCE-5130:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582366/MAPREDUCE-5130-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3598//console
This message is automatically generated.
> Add missing job config options to mapred-default.xml
>
>
> Key: MAPREDUCE-5130
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.0.4-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130.patch
>
>
> I came across that mapreduce.map.child.java.opts and
> mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do
> a fuller sweep to see what else is missing before posting a patch.
> List so far:
> mapreduce.map/reduce.child.java.opts
> mapreduce.map/reduce.memory.mb
> mapreduce.job.jvm.numtasks
> mapreduce.input.lineinputformat.linespermap
> mapreduce.task.combine.progress.records
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652396#comment-13652396 ] Hudson commented on MAPREDUCE-5157: --- Integrated in Hadoop-trunk-Commit #3721 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3721/]) MAPREDUCE-5157. Bring back old sampler related code so that we can support binary compatibility with hadoop-1 sorter example. Contributed by Zhijie Shen. (Revision 1480474) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1480474 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/InputSampler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestInputSampler.java > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch > > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652388#comment-13652388 ] Sandy Ryza commented on MAPREDUCE-5130: --- Attached new patch that adds in some properties I forgot. To fix the failing test, I took out some of the slots based code in JobConf, which doesn't really make sense in the context of YARN. 1 GB is already the default mapreduce.map|reduce.memory.mb in the code. If we choose to change it, I think we should do it in a different JIRA. > Add missing job config options to mapred-default.xml > > > Key: MAPREDUCE-5130 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130.patch > > > I came across that mapreduce.map.child.java.opts and > mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do > a fuller sweep to see what else is missing before posting a patch. > List so far: > mapreduce.map/reduce.child.java.opts > mapreduce.map/reduce.memory.mb > mapreduce.job.jvm.numtasks > mapreduce.input.lineinputformat.linespermap > mapreduce.task.combine.progress.records -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Attachment: MAPREDUCE-5130-1.patch > Add missing job config options to mapred-default.xml > > > Key: MAPREDUCE-5130 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130.patch > > > I came across that mapreduce.map.child.java.opts and > mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do > a fuller sweep to see what else is missing before posting a patch. > List so far: > mapreduce.map/reduce.child.java.opts > mapreduce.map/reduce.memory.mb > mapreduce.job.jvm.numtasks > mapreduce.input.lineinputformat.linespermap > mapreduce.task.combine.progress.records -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5157: --- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks Zhijie! > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch > > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652383#comment-13652383 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5157: +1, this looks good. Checking it in. > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch > > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5209) ShuffleScheduler log message incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652374#comment-13652374 ] Hudson commented on MAPREDUCE-5209: --- Integrated in Hadoop-trunk-Commit #3720 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3720/]) MAPREDUCE-5209. Fix units in a ShuffleScheduler log message. Contributed by Tsuyoshi OZAWA (Revision 1480464) Result = SUCCESS cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1480464 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java > ShuffleScheduler log message incorrect > -- > > Key: MAPREDUCE-5209 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Radim Kolar >Assignee: Tsuyoshi OZAWA >Priority: Minor > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5209.1.patch > > > In ShuffleScheduler.java line 361 log message is incorrect, there should be > "ms" instead of "s". > LOG.info(host + " freed by " + Thread.currentThread().getName() + " in " > + > (System.currentTimeMillis()-shuffleStart.get()) + "ms"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5209) ShuffleScheduler log message incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-5209: - Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 I committed this. Thanks Tsuyoshi > ShuffleScheduler log message incorrect > -- > > Key: MAPREDUCE-5209 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Radim Kolar >Assignee: Tsuyoshi OZAWA >Priority: Minor > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5209.1.patch > > > In ShuffleScheduler.java line 361 log message is incorrect, there should be > "ms" instead of "s". > LOG.info(host + " freed by " + Thread.currentThread().getName() + " in " > + > (System.currentTimeMillis()-shuffleStart.get()) + "ms"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5225) SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits
Zhijie Shen created MAPREDUCE-5225: -- Summary: SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits Key: MAPREDUCE-5225 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5225 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Now, SplitSampler only samples the first maxSplitsSampled splits, caused by MAPREDUCE-1820. However, jumping around all splits is in general preferable than the first N splits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5148) Syslog missing from Map/Reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652340#comment-13652340 ] yeshavora commented on MAPREDUCE-5148: -- Thanks Arun, I verified the fix. It looks good. > Syslog missing from Map/Reduce tasks > > > Key: MAPREDUCE-5148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5148 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.2.0 >Reporter: yeshavora >Assignee: Arun C Murthy > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5148.patch, Screenshot_MAPREDUCE-5148.png > > > MAPREDUCE-4970 introduced incompatible change and causes syslog to be missing > from tasktracker on old clusters which just have log4j.properties configured -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5038) old API CombineFileInputFormat missing fixes that are in new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652333#comment-13652333 ] Sandy Ryza commented on MAPREDUCE-5038: --- Does the version of Hive that's encountering this include HIVE-3338? > old API CombineFileInputFormat missing fixes that are in new API > - > > Key: MAPREDUCE-5038 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5038 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5038-1.patch, MAPREDUCE-5038.patch, > MAPREDUCE-5038-revised-1.patch, MAPREDUCE-5038-revised-1.patch, > MAPREDUCE-5038-revised.patch > > > The following changes patched the CombineFileInputFormat in mapreduce, but > neglected the one in mapred > MAPREDUCE-1597 enabled the CombineFileInputFormat to work on splittable files > MAPREDUCE-2021 solved returning duplicate hostnames in split locations > MAPREDUCE-1806 CombineFileInputFormat does not work with paths not on default > FS > In trunk this is not an issue as the one in mapred extends the one in > mapreduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5148) Syslog missing from Map/Reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5148: - Resolution: Fixed Fix Version/s: 1.2.1 Status: Resolved (was: Patch Available) Thanks for the review Giri. I just committed this. > Syslog missing from Map/Reduce tasks > > > Key: MAPREDUCE-5148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5148 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.2.0 >Reporter: yeshavora >Assignee: Arun C Murthy > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5148.patch, Screenshot_MAPREDUCE-5148.png > > > MAPREDUCE-4970 introduced incompatible change and causes syslog to be missing > from tasktracker on old clusters which just have log4j.properties configured -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652327#comment-13652327
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582358/MAPREDUCE-5217-branch-1.patch.2
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3597//console
This message is automatically generated.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652321#comment-13652321
]
Jitendra Nath Pandey commented on MAPREDUCE-5217:
-
+1
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-branch-1.patch.2
branch-1 patch after addressing Jitendra's comments
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652317#comment-13652317
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
This fix has been manually test with a secure cluster with branch-1
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652269#comment-13652269 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4067: Revisiting this. I think we are on the correct patch. We should just replace the YarnRemoteException with IOException in all of MR protocols. Clearly the legacy MR clients don't know or care about YarnRemoteExceptions. > Replace YarnRemoteException with IOException in MRv2 APIs > - > > Key: MAPREDUCE-4067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Xuan Gong >Priority: Critical > Attachments: MAPREDUCE-4067.1.patch, MR4067.txt > > > YarnRemoteException is defined as a generic wrapper for all the exceptions in > yarn. I think we should instead throw IOExceptions in the API, which can > later be extended for more specialized exceptions without breaking > compatibility. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652250#comment-13652250
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582343/MAPREDUCE-5217-trunk.patch.2
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3596//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3596//console
This message is automatically generated.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[
https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652241#comment-13652241
]
Hadoop QA commented on MAPREDUCE-5157:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582330/MAPREDUCE-5157.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3593//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3593//console
This message is automatically generated.
> Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
> ---
>
> Key: MAPREDUCE-5157
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch
>
>
> In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in
> hadoop-1. However, after upgrading to hadoop-2,
> org.apache.hadoop.mapred.lib.InputSampler is modified to extend
> org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class,
> Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not
> find Sampler when it runs with hadoop-2 framework.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5156: --- Status: Open (was: Patch Available) Can you please address the findBugs warning? > Hadoop-examples-1.x.x.jar cannot run on Yarn > > > Key: MAPREDUCE-5156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5156.1.patch, MAPREDUCE-5156.2.patch > > > M/R examples are run through ProgramDriver.driver. ProgramDriver.driver > returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the > function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reopened MAPREDUCE-5160: Reopening to correctly close it as a duplicate. > Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find > their inner classes when running on Yarn > --- > > Key: MAPREDUCE-5160 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5160 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > > Aggregatewordcount and Aggregatewordhist of hadoop-1 cannot run on hadoop-2 > due to > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String > args[], Class caller) is not available on hadoop-2 (see MAPREDUCE-5159). > After I changed Aggregatewordcount and Aggregatewordhist to use > createValueAggregatorJob(String args[], Class ValueAggregatorDescriptor>[] descriptors), which is available on hadoop-2, > the two examples could be accepted and run on Yarn. > However, the two examples still failed, because their inner classes, > WordCountPlugInClass and AggregateWordHistogramPlugin, cannot be found in > runtime, respectively. Both the plugin classes extend > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorBaseDescriptor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5160. Resolution: Duplicate > Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find > their inner classes when running on Yarn > --- > > Key: MAPREDUCE-5160 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5160 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > > Aggregatewordcount and Aggregatewordhist of hadoop-1 cannot run on hadoop-2 > due to > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String > args[], Class caller) is not available on hadoop-2 (see MAPREDUCE-5159). > After I changed Aggregatewordcount and Aggregatewordhist to use > createValueAggregatorJob(String args[], Class ValueAggregatorDescriptor>[] descriptors), which is available on hadoop-2, > the two examples could be accepted and run on Yarn. > However, the two examples still failed, because their inner classes, > WordCountPlugInClass and AggregateWordHistogramPlugin, cannot be found in > runtime, respectively. Both the plugin classes extend > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorBaseDescriptor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-trunk.patch.2
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652233#comment-13652233
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
Thanks for the comments Jitendra. I have modified the patch with all the
comments and uploaded a new patch
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
> MAPREDUCE-5217-trunk.patch.2
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5148) Syslog missing from Map/Reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652228#comment-13652228 ] Giridharan Kesavan commented on MAPREDUCE-5148: --- +1 patch looks good. Thanks Arun > Syslog missing from Map/Reduce tasks > > > Key: MAPREDUCE-5148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5148 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.2.0 >Reporter: yeshavora >Assignee: Arun C Murthy > Attachments: MAPREDUCE-5148.patch, Screenshot_MAPREDUCE-5148.png > > > MAPREDUCE-4970 introduced incompatible change and causes syslog to be missing > from tasktracker on old clusters which just have log4j.properties configured -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-5160. Resolution: Fixed Fix Version/s: 2.0.5-beta The issue is fixed with MAPREDUCE-5159. > Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find > their inner classes when running on Yarn > --- > > Key: MAPREDUCE-5160 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5160 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > > Aggregatewordcount and Aggregatewordhist of hadoop-1 cannot run on hadoop-2 > due to > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String > args[], Class caller) is not available on hadoop-2 (see MAPREDUCE-5159). > After I changed Aggregatewordcount and Aggregatewordhist to use > createValueAggregatorJob(String args[], Class ValueAggregatorDescriptor>[] descriptors), which is available on hadoop-2, > the two examples could be accepted and run on Yarn. > However, the two examples still failed, because their inner classes, > WordCountPlugInClass and AggregateWordHistogramPlugin, cannot be found in > runtime, respectively. Both the plugin classes extend > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorBaseDescriptor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5219) JobStatus#getJobPriority changed to JobStatus#getPriority in MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652221#comment-13652221
]
Hadoop QA commented on MAPREDUCE-5219:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582338/MAPREDUCE-5219-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3595//console
This message is automatically generated.
> JobStatus#getJobPriority changed to JobStatus#getPriority in MR2
>
>
> Key: MAPREDUCE-5219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5219
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Affects Versions: 2.0.4-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-5219-1.patch, MAPREDUCE-5219-1.patch,
> MAPREDUCE-5219.patch
>
>
> We should change it back for compatibility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Fang updated MAPREDUCE-5224:
---
Description:
JobTracker today expects the system directory to be in the default file system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionAction() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get("mapred.system.dir",
"/tmp/hadoop/mapred/system"));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We should
change JobTracker to allow that.
was:
JobTracker today expects the system directory to be in the default file system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionAction() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get("mapred.system.dir",
"/tmp/hadoop/mapred/system"));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We should
change JobTracker to allow that.
> JobTracker should allow the system directory to be in non-default FS
>
>
> Key: MAPREDUCE-5224
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Reporter: Xi Fang
>Assignee: Xi Fang
>Priority: Minor
> Fix For: 1-win
>
>
> JobTracker today expects the system directory to be in the default file
> system
> if (fs == null) {
> fs = mrOwner.doAs(new PrivilegedExceptionAction() {
> public FileSystem run() throws IOException {
> return FileSystem.get(conf);
> }});
> }
> ...
> public String getSystemDir() {
> Path sysDir = new Path(conf.get("mapred.system.dir",
> "/tmp/hadoop/mapred/system"));
> return fs.makeQualified(sysDir).toString();
> }
> In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
> Storage), but we would still like the system directory to be in DFS. We
> should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
Xi Fang created MAPREDUCE-5224:
--
Summary: JobTracker should allow the system directory to be in
non-default FS
Key: MAPREDUCE-5224
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Fix For: 1-win
JobTracker today expects the system directory to be in the default file system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionAction() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get("mapred.system.dir",
"/tmp/hadoop/mapred/system"));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We should
change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient
[
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652204#comment-13652204
]
Hadoop QA commented on MAPREDUCE-5222:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582334/mr-5222.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3594//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3594//console
This message is automatically generated.
> Add missing methods to JobClient
> -
>
> Key: MAPREDUCE-5222
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Affects Versions: 2.0.4-alpha
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Fix For: 2.0.5-beta
>
> Attachments: mr-5222.patch, mr-5222.patch
>
>
> JobClient is missing the following two public methods we need to add for
> binary compatibility:
> # static isJobDirValid(Path, FileSystem)
> # Path getStagingAreaDir()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate
[ https://issues.apache.org/jira/browse/MAPREDUCE-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652198#comment-13652198 ] Hudson commented on MAPREDUCE-5159: --- Integrated in Hadoop-trunk-Commit #3719 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3719/]) MAPREDUCE-5159. Change ValueAggregatorJob to add APIs which can support binary compatibility with hadoop-1 examples. Contributed by Zhijie Shen. (Revision 1480394) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1480394 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorJob.java > Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary > compatible with hadoop-2 mapred.lib.aggregate > -- > > Key: MAPREDUCE-5159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5159.1.patch > > > Both examples in hadoop-1 use > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String > args[], Class caller), which no longer exists after upgrading to > hadoop-2. Therefore, they cannot not find the accordant function with their > function signature in the binary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5219) JobStatus#getJobPriority changed to JobStatus#getPriority in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5219: -- Attachment: MAPREDUCE-5219-1.patch > JobStatus#getJobPriority changed to JobStatus#getPriority in MR2 > > > Key: MAPREDUCE-5219 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5219 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5219-1.patch, MAPREDUCE-5219-1.patch, > MAPREDUCE-5219.patch > > > We should change it back for compatibility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate
[ https://issues.apache.org/jira/browse/MAPREDUCE-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5159: --- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Just committed this to trunk and branch-2. Thanks Zhijie! > Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary > compatible with hadoop-2 mapred.lib.aggregate > -- > > Key: MAPREDUCE-5159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5159.1.patch > > > Both examples in hadoop-1 use > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String > args[], Class caller), which no longer exists after upgrading to > hadoop-2. Therefore, they cannot not find the accordant function with their > function signature in the binary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate
[ https://issues.apache.org/jira/browse/MAPREDUCE-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652171#comment-13652171 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5159: +1, this looks good. Checking it in. > Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary > compatible with hadoop-2 mapred.lib.aggregate > -- > > Key: MAPREDUCE-5159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5159.1.patch > > > Both examples in hadoop-1 use > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String > args[], Class caller), which no longer exists after upgrading to > hadoop-2. Therefore, they cannot not find the accordant function with their > function signature in the binary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652168#comment-13652168
]
Jitendra Nath Pandey commented on MAPREDUCE-5217:
-
Also, the job conf is being set twice.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated MAPREDUCE-5217:
Affects Version/s: 2.0.4-alpha
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2, 2.0.4-alpha
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652162#comment-13652162
]
Jitendra Nath Pandey commented on MAPREDUCE-5217:
-
1. Please use the LOG object for logging instead of System.out.
2. Minor: 'delegationToken' variable name should be renamed to tokenFile,
because it actually represents the file name that contains tokens, and could
also contain tokens other than delegation tokens.
> DistCp fails when launched by Oozie in a secure cluster
> ---
>
> Key: MAPREDUCE-5217
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp, security
>Affects Versions: trunk, 1.1.2
> Environment: Hadoop secure cluster
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Attachments: MAPREDUCE-5217-branch-1.patch,
> MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
>
>
> As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
> in the main launcher for Pig, Hive, MR and Sqoop actions.
> if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
> jobConf.set("mapreduce.job.credentials.binary",
> System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
> }
> For Java action, which does not have a main launcher in oozie, the above
> codecan be added by the user as the user purportedly has the code that is
> launched.
> But for DistCp action, the user has no such luxury. The solution attempted in
> MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
> would break MAPREDUCE-3727. So, we have to fix DistCp and
> add the same boilerplate code so that DistCp action can be launched by Oozie
> in a secure cluster.
> The code added checks for an System env. variable to be set which is not
> typically set in normal command line execution of DistCp, DistCp runs fine
> with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5222) Add missing methods to JobClient
[ https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5222: Attachment: mr-5222.patch Thanks Steve. Uploading a patch that incorporates your suggestions. > Add missing methods to JobClient > - > > Key: MAPREDUCE-5222 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 2.0.4-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 2.0.5-beta > > Attachments: mr-5222.patch, mr-5222.patch > > > JobClient is missing the following two public methods we need to add for > binary compatibility: > # static isJobDirValid(Path, FileSystem) > # Path getStagingAreaDir() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[
https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652155#comment-13652155
]
Zhijie Shen commented on MAPREDUCE-5157:
{quote}
Oh and I suppose, with the latest patch, branch-1 sorter works against
hadoop-2, right?
{quote}
Yes, given MAPREDUCE-5156 is fixed together.
> Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
> ---
>
> Key: MAPREDUCE-5157
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch
>
>
> In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in
> hadoop-1. However, after upgrading to hadoop-2,
> org.apache.hadoop.mapred.lib.InputSampler is modified to extend
> org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class,
> Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not
> find Sampler when it runs with hadoop-2 framework.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[
https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652153#comment-13652153
]
Zhijie Shen commented on MAPREDUCE-5157:
{quote}
Did you try to implement the added-back Sampler's class's getSample using the
super class implementation?
{quote}
No, because the super class uses InputSplit from mapreduece.lib, which is an
abstract class, while the implementation here needs to use InputSplit from
mapred.lib, which is an interface. InputSplit from mapred.lib doesn't extend
InputSplit from mapreduece.lib, thus, they are not compatible. Moreover, it is
much more risky to modify the interface classes.
{quote}
So all the added back code is just copy-pasted from branch-1? It did looks like
so, I cross-verfied, except the class definitions which implement from
mapreduce.* counterparts. Just confirming. Right?
{quote}
Yes, copied the copy from Branch-1
{quote}
Jumping around is in general preferable than the first N splits. Can you file a
ticket?
{quote}
I'll file a follow up ticket.
> Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
> ---
>
> Key: MAPREDUCE-5157
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch
>
>
> In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in
> hadoop-1. However, after upgrading to hadoop-2,
> org.apache.hadoop.mapred.lib.InputSampler is modified to extend
> org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class,
> Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not
> find Sampler when it runs with hadoop-2 framework.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652144#comment-13652144 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5157: Oh and I suppose, with the latest patch, branch-1 sorter works against hadoop-2, right? > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch > > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5157: --- Status: Patch Available (was: Open) > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch > > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5157: --- Attachment: MAPREDUCE-5157.1.patch Reattach the patch for Jenkins > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch > > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5157: --- Status: Open (was: Patch Available) HDFS-4305 is committed. Can you reattach the same patch and run it through Jenkins? > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch > > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652137#comment-13652137 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5157: Looked at the patch. Some comments: - Did you try to implement the added-back Sampler's class's getSample using the super class implementation? - So all the added back code is just copy-pasted from branch-1? It did looks like so, I cross-verfied, except the class definitions which implement from mapreduce.* counterparts. Just confirming. Right? - The tests also seem to be strictly following their mapreduce counterparts, which is good except for one thing: There seems to be some difference in SplitSampler between 1.0 and the new one -- in 1.* we are using a SPLIT_STEP so as to only inspect by jumping around splits by that amount but read at a max that many splits as specified by the maxSplitsSampled parameter -- in 2.*, we are sampling the first maxSplitsSampled splits. This seems to be caused by MAPREDUCE-1820. Jumping around is in general preferable than the first N splits. Can you file a ticket? > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5157.1.patch > > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient
[
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652076#comment-13652076
]
Steve Loughran commented on MAPREDUCE-5222:
---
* can you throw {{InterruptedIOException}}?
* go on, add some tests
> Add missing methods to JobClient
> -
>
> Key: MAPREDUCE-5222
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Affects Versions: 2.0.4-alpha
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Fix For: 2.0.5-beta
>
> Attachments: mr-5222.patch
>
>
> JobClient is missing the following two public methods we need to add for
> binary compatibility:
> # static isJobDirValid(Path, FileSystem)
> # Path getStagingAreaDir()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5210) Job submission has strict permission validation
[
https://issues.apache.org/jira/browse/MAPREDUCE-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652055#comment-13652055
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-5210:
Let's not add in new config items. Adding a filesytem API is fine like I first
mentioned. Can you create a sister JIRA in Hadoop Common and propose that
there? If that doesn't go well, we can always do specific checks in MR as the
last resort.
> Job submission has strict permission validation
> ---
>
> Key: MAPREDUCE-5210
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5210
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Amareshwari Sriramadasu
>Assignee: samar
>
> The following code in JobSubmissionFiles.java mandates strict permission on
> job submission :
> {noformat}
> if (fs.exists(stagingArea)) {
> FileStatus fsStatus = fs.getFileStatus(stagingArea);
> String owner = fsStatus.getOwner();
> if (!(owner.equals(currentUser) || owner.equals(realUser))) {
> throw new IOException("The ownership on the staging directory " +
> stagingArea + " is not as expected. " +
> "It is owned by " + owner + ". The directory must " +
> "be owned by the submitter " + currentUser + " or " +
> "by " + realUser);
> }
> {noformat}
> For file systems such as S3, which do not have permission concept, user can
> never submit a job with staging area in S3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5223) ant package shouldn't build task-controller by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652047#comment-13652047 ] Alejandro Abdelnur commented on MAPREDUCE-5223: --- Why do we need a special switch for task-controller? IMO, using the native switch -as proposed by Chris- should be enough, no? > ant package shouldn't build task-controller by default > -- > > Key: MAPREDUCE-5223 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5223 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Arun C Murthy > > ant package shouldn't build task-controller by default, not sure when we > broke this. task-controller should have to be explicitly built via > '-Dtask-controller' option. > This breaks tar creation on all non-Linux platforms. Sigh. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5038) old API CombineFileInputFormat missing fixes that are in new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652044#comment-13652044 ] Sandy Ryza commented on MAPREDUCE-5038: --- Taking a look at this today > old API CombineFileInputFormat missing fixes that are in new API > - > > Key: MAPREDUCE-5038 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5038 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5038-1.patch, MAPREDUCE-5038.patch, > MAPREDUCE-5038-revised-1.patch, MAPREDUCE-5038-revised-1.patch, > MAPREDUCE-5038-revised.patch > > > The following changes patched the CombineFileInputFormat in mapreduce, but > neglected the one in mapred > MAPREDUCE-1597 enabled the CombineFileInputFormat to work on splittable files > MAPREDUCE-2021 solved returning duplicate hostnames in split locations > MAPREDUCE-1806 CombineFileInputFormat does not work with paths not on default > FS > In trunk this is not an issue as the one in mapred extends the one in > mapreduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5223) ant package shouldn't build task-controller by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652043#comment-13652043
]
Chris Nauroth commented on MAPREDUCE-5223:
--
HADOOP-8921 is a long-standing similar bug with a patch available (but not
committed). Do you think this is a duplicate?
As part of HADOOP-8921, I had proposed the following change, but there was
disagreement, so it's not included in the latest HADOOP-8921 patch.
{code}
-
+
{code}
> ant package shouldn't build task-controller by default
> --
>
> Key: MAPREDUCE-5223
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5223
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun C Murthy
>
> ant package shouldn't build task-controller by default, not sure when we
> broke this. task-controller should have to be explicitly built via
> '-Dtask-controller' option.
> This breaks tar creation on all non-Linux platforms. Sigh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5209) ShuffleScheduler log message incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651918#comment-13651918 ] Tsuyoshi OZAWA commented on MAPREDUCE-5209: --- I think this modification doesn't need to include test code. Any idea? > ShuffleScheduler log message incorrect > -- > > Key: MAPREDUCE-5209 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Radim Kolar >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: MAPREDUCE-5209.1.patch > > > In ShuffleScheduler.java line 361 log message is incorrect, there should be > "ms" instead of "s". > LOG.info(host + " freed by " + Thread.currentThread().getName() + " in " > + > (System.currentTimeMillis()-shuffleStart.get()) + "ms"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5209) ShuffleScheduler log message incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651860#comment-13651860
]
Hadoop QA commented on MAPREDUCE-5209:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582289/MAPREDUCE-5209.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3592//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3592//console
This message is automatically generated.
> ShuffleScheduler log message incorrect
> --
>
> Key: MAPREDUCE-5209
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Radim Kolar
>Assignee: Tsuyoshi OZAWA
>Priority: Minor
> Attachments: MAPREDUCE-5209.1.patch
>
>
> In ShuffleScheduler.java line 361 log message is incorrect, there should be
> "ms" instead of "s".
> LOG.info(host + " freed by " + Thread.currentThread().getName() + " in "
> +
> (System.currentTimeMillis()-shuffleStart.get()) + "ms");
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5209) ShuffleScheduler log message incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5209: -- Attachment: MAPREDUCE-5209.1.patch > ShuffleScheduler log message incorrect > -- > > Key: MAPREDUCE-5209 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Radim Kolar >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: MAPREDUCE-5209.1.patch > > > In ShuffleScheduler.java line 361 log message is incorrect, there should be > "ms" instead of "s". > LOG.info(host + " freed by " + Thread.currentThread().getName() + " in " > + > (System.currentTimeMillis()-shuffleStart.get()) + "ms"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5209) ShuffleScheduler log message incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5209: -- Status: Patch Available (was: Open) > ShuffleScheduler log message incorrect > -- > > Key: MAPREDUCE-5209 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Radim Kolar >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: MAPREDUCE-5209.1.patch > > > In ShuffleScheduler.java line 361 log message is incorrect, there should be > "ms" instead of "s". > LOG.info(host + " freed by " + Thread.currentThread().getName() + " in " > + > (System.currentTimeMillis()-shuffleStart.get()) + "ms"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5209) ShuffleScheduler log message incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5209: -- Assignee: Tsuyoshi OZAWA > ShuffleScheduler log message incorrect > -- > > Key: MAPREDUCE-5209 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Radim Kolar >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: MAPREDUCE-5209.1.patch > > > In ShuffleScheduler.java line 361 log message is incorrect, there should be > "ms" instead of "s". > LOG.info(host + " freed by " + Thread.currentThread().getName() + " in " > + > (System.currentTimeMillis()-shuffleStart.get()) + "ms"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[
https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651760#comment-13651760
]
Hadoop QA commented on MAPREDUCE-5221:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582275/MAPREDUCE-5221.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3591//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3591//console
This message is automatically generated.
> Reduce side Combiner is not used when using the new API
> ---
>
> Key: MAPREDUCE-5221
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.4-alpha
>Reporter: Siddharth Seth
>Assignee: Tsuyoshi OZAWA
> Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch,
> MAPREDUCE-5221.3.patch
>
>
> If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this
> will silently ignored on the reduce side since the reduce side usage is only
> aware of the old api combiner.
> This doesn't fail the job - since the new combiner key does not deprecate the
> old key.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5221: -- Attachment: MAPREDUCE-5221.3.patch Fixed to pass compile against v2. > Reduce side Combiner is not used when using the new API > --- > > Key: MAPREDUCE-5221 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi OZAWA > Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch, > MAPREDUCE-5221.3.patch > > > If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this > will silently ignored on the reduce side since the reduce side usage is only > aware of the old api combiner. > This doesn't fail the job - since the new combiner key does not deprecate the > old key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[
https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651749#comment-13651749
]
Hadoop QA commented on MAPREDUCE-5221:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582274/MAPREDUCE-5221.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3590//console
This message is automatically generated.
> Reduce side Combiner is not used when using the new API
> ---
>
> Key: MAPREDUCE-5221
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.4-alpha
>Reporter: Siddharth Seth
>Assignee: Tsuyoshi OZAWA
> Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch
>
>
> If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this
> will silently ignored on the reduce side since the reduce side usage is only
> aware of the old api combiner.
> This doesn't fail the job - since the new combiner key does not deprecate the
> old key.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[
https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651750#comment-13651750
]
Hadoop QA commented on MAPREDUCE-5221:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582273/MAPREDUCE-5221.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1367 javac
compiler warnings (more than the trunk's current 1366 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3589//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3589//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3589//console
This message is automatically generated.
> Reduce side Combiner is not used when using the new API
> ---
>
> Key: MAPREDUCE-5221
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.4-alpha
>Reporter: Siddharth Seth
>Assignee: Tsuyoshi OZAWA
> Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch
>
>
> If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this
> will silently ignored on the reduce side since the reduce side usage is only
> aware of the old api combiner.
> This doesn't fail the job - since the new combiner key does not deprecate the
> old key.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5221: -- Attachment: MAPREDUCE-5221.2.patch Updated to remove needless argument against ReduceTask#getCombinerClass. > Reduce side Combiner is not used when using the new API > --- > > Key: MAPREDUCE-5221 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi OZAWA > Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch > > > If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this > will silently ignored on the reduce side since the reduce side usage is only > aware of the old api combiner. > This doesn't fail the job - since the new combiner key does not deprecate the > old key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5221: -- Status: Patch Available (was: Open) > Reduce side Combiner is not used when using the new API > --- > > Key: MAPREDUCE-5221 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi OZAWA > Attachments: MAPREDUCE-5221.1.patch > > > If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this > will silently ignored on the reduce side since the reduce side usage is only > aware of the old api combiner. > This doesn't fail the job - since the new combiner key does not deprecate the > old key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5221: -- Attachment: MAPREDUCE-5221.1.patch Attached the patch including the fixes and tests. > Reduce side Combiner is not used when using the new API > --- > > Key: MAPREDUCE-5221 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi OZAWA > Attachments: MAPREDUCE-5221.1.patch > > > If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this > will silently ignored on the reduce side since the reduce side usage is only > aware of the old api combiner. > This doesn't fail the job - since the new combiner key does not deprecate the > old key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5221: -- Assignee: Tsuyoshi OZAWA > Reduce side Combiner is not used when using the new API > --- > > Key: MAPREDUCE-5221 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi OZAWA > > If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this > will silently ignored on the reduce side since the reduce side usage is only > aware of the old api combiner. > This doesn't fail the job - since the new combiner key does not deprecate the > old key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5148) Syslog missing from Map/Reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651713#comment-13651713 ] Arun C Murthy commented on MAPREDUCE-5148: -- Btw, for future reference - we need to ensure that the code has right defaults for all configs i.e. in this case it would be required to ensure TaskTracker sets correct defaults for all properties in task-log4j.properties. Furthermore, task-log4j.properties has a bunch of stuff which shouldn't have been blindly copied from the main log4j.properties such as definitions for jobsummary log etc. We should minimize contents of that file and, as above, ensure that all properties defined in task-log4j.properties are correctly represented in code too. > Syslog missing from Map/Reduce tasks > > > Key: MAPREDUCE-5148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5148 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.2.0 >Reporter: yeshavora >Assignee: Arun C Murthy > Attachments: MAPREDUCE-5148.patch, Screenshot_MAPREDUCE-5148.png > > > MAPREDUCE-4970 introduced incompatible change and causes syslog to be missing > from tasktracker on old clusters which just have log4j.properties configured -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5148) Syslog missing from Map/Reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651711#comment-13651711
]
Hadoop QA commented on MAPREDUCE-5148:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582268/MAPREDUCE-5148.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3588//console
This message is automatically generated.
> Syslog missing from Map/Reduce tasks
>
>
> Key: MAPREDUCE-5148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 1.2.0
>Reporter: yeshavora
>Assignee: Arun C Murthy
> Attachments: MAPREDUCE-5148.patch, Screenshot_MAPREDUCE-5148.png
>
>
> MAPREDUCE-4970 introduced incompatible change and causes syslog to be missing
> from tasktracker on old clusters which just have log4j.properties configured
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5148) Syslog missing from Map/Reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5148: - Status: Patch Available (was: Open) > Syslog missing from Map/Reduce tasks > > > Key: MAPREDUCE-5148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5148 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.2.0 >Reporter: yeshavora >Assignee: Arun C Murthy > Attachments: MAPREDUCE-5148.patch, Screenshot_MAPREDUCE-5148.png > > > MAPREDUCE-4970 introduced incompatible change and causes syslog to be missing > from tasktracker on old clusters which just have log4j.properties configured -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5148) Syslog missing from Map/Reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651710#comment-13651710 ] Arun C Murthy commented on MAPREDUCE-5148: -- Spoke too soon, the patch works as expected. I was confused running from a non-tarball i.e. dev-mode build which actually didn't get hadoop-core*.jar on classpath of the MR child task. I think the patch is good to go. Thoughts? > Syslog missing from Map/Reduce tasks > > > Key: MAPREDUCE-5148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5148 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.2.0 >Reporter: yeshavora >Assignee: Arun C Murthy > Attachments: MAPREDUCE-5148.patch, Screenshot_MAPREDUCE-5148.png > > > MAPREDUCE-4970 introduced incompatible change and causes syslog to be missing > from tasktracker on old clusters which just have log4j.properties configured -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5223) ant package shouldn't build task-controller by default
Arun C Murthy created MAPREDUCE-5223: Summary: ant package shouldn't build task-controller by default Key: MAPREDUCE-5223 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5223 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy ant package shouldn't build task-controller by default, not sure when we broke this. task-controller should have to be explicitly built via '-Dtask-controller' option. This breaks tar creation on all non-Linux platforms. Sigh. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
