date:20130509

[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652783#comment-13652783
]

Hadoop QA commented on MAPREDUCE-5217:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12582430/MAPREDUCE-5217-branch-1.patch.3
against trunk revision .

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3603//console

This message is automatically generated.

DistCp fails when launched by Oozie in a secure cluster
---

Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3

As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsInst should standardize naming rules


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated MAPREDUCE-5227:
--

Assignee: (was: Masahiro Nakagawa)

 JobTrackerMetricsInst should standardize naming rules
 -

 Key: MAPREDUCE-5227
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 1.1.2
Reporter: Tsuyoshi OZAWA
Priority: Minor

 JobTrackerMetricsInst provides users with some metrics, but its naming rules( 
 jobs_running, running_maps, running_reduces) sometimes confuses users. 
 It should be standardized.
 One concern is backward compatibility, so one idea is to share 
 MetricMutableGaugeInt object from old and new property name.
 e.g. to share runningMaps from running_maps and maps_running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4584) Umbrella: Preemption and restart of MapReduce tasks

2013-05-09 Thread Chris Douglas (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652832#comment-13652832
 ] 

Chris Douglas commented on MAPREDUCE-4584:
--

[~ozawa]: I've been reading some of the iterations of your patch(es) as you've 
updated them over the last few months. Our proposals are absolutely 
complementary. Your approach (IIRC) involved reusing map tasks to aggregate map 
output on the same host, right? MAPREDUCE-4502 can accomplish more than 
checkpointing by aggregating across partitions.

We added some metadata to {{IFile}} to track which task attempts a segment 
contains. I haven't looked at a recent version of your patch, but that's 
certainly shared functionality.

 Umbrella: Preemption and restart of MapReduce tasks
 ---

 Key: MAPREDUCE-4584
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4584
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: applicationmaster, mrv2, performance, resourcemanager, 
 task
Reporter: Sriram Rao
Assignee: Chris Douglas

 This JIRA will track the implementation of improvements to the handling of 
 intermediate data (e.g., map output). Specifically, it tracks changes in 
 support of preempting running tasks, checkpointing completed work, and 
 spawning one or more tasks to complete the original split/partition. These 
 mechanisms allow one to manage skew in intermediate data, respond to resource 
 abundance or scarcity (particularly with preemption), speculatively execute 
 on the remaining work from checkpointed tasks, and automatically tune 
 parameters for performance.
 Iterations will build on learnings from previous work, including the 
 following:
 Technical reports:
 http://research.yahoo.com/files/yl-2012-002.pdf
 http://research.yahoo.com/files/yl-2012-003.pdf
 Source code:
 http://code.google.com/p/sailfish

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5212) Handle exception related changes in YARN's ClientRMProtocol api after YARN-631


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652860#comment-13652860
 ] 

Hudson commented on MAPREDUCE-5212:
---

Integrated in Hadoop-Yarn-trunk #204 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/204/])
YARN-631. Changed ClientRMProtocol api to throw IOException and 
YarnRemoteException. Contributed by Xuan Gong.
MAPREDUCE-5212. Handling YarnRemoteException separately from IOException in MR 
App after YARN-631. Contributed by Xuan Gong. (Revision 1480494)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480494
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ClientRMProtocolPBClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ClientRMProtocolPBServiceImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java


 Handle exception related changes in YARN's ClientRMProtocol api after YARN-631
 --

 Key: MAPREDUCE-5212
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5212
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
 Fix For: 2.0.5-beta

 Attachments: MR-5212.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652865#comment-13652865
 ] 

Hudson commented on MAPREDUCE-5157:
---

Integrated in Hadoop-Yarn-trunk #204 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/204/])
MAPREDUCE-5157. Bring back old sampler related code so that we can support 
binary compatibility with hadoop-1 sorter example. Contributed by Zhijie Shen. 
(Revision 1480474)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480474
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/InputSampler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestInputSampler.java


 Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
 ---

 Key: MAPREDUCE-5157
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch


 In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in 
 hadoop-1. However, after upgrading to hadoop-2, 
 org.apache.hadoop.mapred.lib.InputSampler is modified to extend 
 org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, 
 Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not 
 find Sampler when it runs with hadoop-2 framework.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5226) Handle exception related changes in YARN's AMRMProtocol api after YARN-630


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652859#comment-13652859
 ] 

Hudson commented on MAPREDUCE-5226:
---

Integrated in Hadoop-Yarn-trunk #204 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/204/])
YARN-630. Changed AMRMProtocol api to throw IOException and 
YarnRemoteException. Contributed by Xuan Gong.
MAPREDUCE-5226. Handling YarnRemoteException separately from IOException in MR 
App's use of AMRMProtocol after YARN-630. Contributed by Xuan Gong. (Revision 
1480529)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480529
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/AMRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/AMRMProtocolPBClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/AMRMProtocolPBServiceImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java


 Handle exception related changes in YARN's AMRMProtocol api after YARN-630
 --

 Key: MAPREDUCE-5226
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
 Fix For: 2.0.5-beta

 Attachments: MR-5226.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652863#comment-13652863
 ] 

Hudson commented on MAPREDUCE-5159:
---

Integrated in Hadoop-Yarn-trunk #204 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/204/])
MAPREDUCE-5159. Change ValueAggregatorJob to add APIs which can support 
binary compatibility with hadoop-1 examples. Contributed by Zhijie Shen. 
(Revision 1480394)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480394
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorJob.java


 Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary 
 compatible with hadoop-2 mapred.lib.aggregate
 --

 Key: MAPREDUCE-5159
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5159.1.patch


 Both examples in hadoop-1 use 
 org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String
  args[], Class? caller), which no longer exists after upgrading to 
 hadoop-2. Therefore, they cannot not find the accordant function with their 
 function signature in the binary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5209) ShuffleScheduler log message incorrect


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652866#comment-13652866
 ] 

Hudson commented on MAPREDUCE-5209:
---

Integrated in Hadoop-Yarn-trunk #204 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/204/])
MAPREDUCE-5209. Fix units in a ShuffleScheduler log message.
Contributed by Tsuyoshi OZAWA (Revision 1480464)

 Result = SUCCESS
cdouglas : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480464
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java


 ShuffleScheduler log message incorrect
 --

 Key: MAPREDUCE-5209
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Radim Kolar
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5209.1.patch


 In ShuffleScheduler.java line 361 log message is incorrect, there should be 
 ms instead of s.
 LOG.info(host +  freed by  + Thread.currentThread().getName() +  in  
 + 
  (System.currentTimeMillis()-shuffleStart.get()) + ms);

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5226) Handle exception related changes in YARN's AMRMProtocol api after YARN-630


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652919#comment-13652919
 ] 

Hudson commented on MAPREDUCE-5226:
---

Integrated in Hadoop-Hdfs-trunk #1393 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1393/])
YARN-630. Changed AMRMProtocol api to throw IOException and 
YarnRemoteException. Contributed by Xuan Gong.
MAPREDUCE-5226. Handling YarnRemoteException separately from IOException in MR 
App's use of AMRMProtocol after YARN-630. Contributed by Xuan Gong. (Revision 
1480529)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480529
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/AMRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/AMRMProtocolPBClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/AMRMProtocolPBServiceImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java


 Handle exception related changes in YARN's AMRMProtocol api after YARN-630
 --

 Key: MAPREDUCE-5226
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
 Fix For: 2.0.5-beta

 Attachments: MR-5226.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5212) Handle exception related changes in YARN's ClientRMProtocol api after YARN-631


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652920#comment-13652920
 ] 

Hudson commented on MAPREDUCE-5212:
---

Integrated in Hadoop-Hdfs-trunk #1393 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1393/])
YARN-631. Changed ClientRMProtocol api to throw IOException and 
YarnRemoteException. Contributed by Xuan Gong.
MAPREDUCE-5212. Handling YarnRemoteException separately from IOException in MR 
App after YARN-631. Contributed by Xuan Gong. (Revision 1480494)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480494
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ClientRMProtocolPBClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ClientRMProtocolPBServiceImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java


 Handle exception related changes in YARN's ClientRMProtocol api after YARN-631
 --

 Key: MAPREDUCE-5212
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5212
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
 Fix For: 2.0.5-beta

 Attachments: MR-5212.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652923#comment-13652923
 ] 

Hudson commented on MAPREDUCE-5159:
---

Integrated in Hadoop-Hdfs-trunk #1393 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1393/])
MAPREDUCE-5159. Change ValueAggregatorJob to add APIs which can support 
binary compatibility with hadoop-1 examples. Contributed by Zhijie Shen. 
(Revision 1480394)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480394
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorJob.java


 Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary 
 compatible with hadoop-2 mapred.lib.aggregate
 --

 Key: MAPREDUCE-5159
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5159.1.patch


 Both examples in hadoop-1 use 
 org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String
  args[], Class? caller), which no longer exists after upgrading to 
 hadoop-2. Therefore, they cannot not find the accordant function with their 
 function signature in the binary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652925#comment-13652925
 ] 

Hudson commented on MAPREDUCE-5157:
---

Integrated in Hadoop-Hdfs-trunk #1393 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1393/])
MAPREDUCE-5157. Bring back old sampler related code so that we can support 
binary compatibility with hadoop-1 sorter example. Contributed by Zhijie Shen. 
(Revision 1480474)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480474
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/InputSampler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestInputSampler.java


 Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
 ---

 Key: MAPREDUCE-5157
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch


 In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in 
 hadoop-1. However, after upgrading to hadoop-2, 
 org.apache.hadoop.mapred.lib.InputSampler is modified to extend 
 org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, 
 Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not 
 find Sampler when it runs with hadoop-2 framework.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5209) ShuffleScheduler log message incorrect


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652926#comment-13652926
 ] 

Hudson commented on MAPREDUCE-5209:
---

Integrated in Hadoop-Hdfs-trunk #1393 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1393/])
MAPREDUCE-5209. Fix units in a ShuffleScheduler log message.
Contributed by Tsuyoshi OZAWA (Revision 1480464)

 Result = FAILURE
cdouglas : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480464
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java


 ShuffleScheduler log message incorrect
 --

 Key: MAPREDUCE-5209
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Radim Kolar
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5209.1.patch


 In ShuffleScheduler.java line 361 log message is incorrect, there should be 
 ms instead of s.
 LOG.info(host +  freed by  + Thread.currentThread().getName() +  in  
 + 
  (System.currentTimeMillis()-shuffleStart.get()) + ms);

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5226) Handle exception related changes in YARN's AMRMProtocol api after YARN-630


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652970#comment-13652970
 ] 

Hudson commented on MAPREDUCE-5226:
---

Integrated in Hadoop-Mapreduce-trunk #1420 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1420/])
YARN-630. Changed AMRMProtocol api to throw IOException and 
YarnRemoteException. Contributed by Xuan Gong.
MAPREDUCE-5226. Handling YarnRemoteException separately from IOException in MR 
App's use of AMRMProtocol after YARN-630. Contributed by Xuan Gong. (Revision 
1480529)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480529
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/AMRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/AMRMProtocolPBClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/AMRMProtocolPBServiceImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java


 Handle exception related changes in YARN's AMRMProtocol api after YARN-630
 --

 Key: MAPREDUCE-5226
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
 Fix For: 2.0.5-beta

 Attachments: MR-5226.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5212) Handle exception related changes in YARN's ClientRMProtocol api after YARN-631


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652971#comment-13652971
 ] 

Hudson commented on MAPREDUCE-5212:
---

Integrated in Hadoop-Mapreduce-trunk #1420 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1420/])
YARN-631. Changed ClientRMProtocol api to throw IOException and 
YarnRemoteException. Contributed by Xuan Gong.
MAPREDUCE-5212. Handling YarnRemoteException separately from IOException in MR 
App after YARN-631. Contributed by Xuan Gong. (Revision 1480494)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480494
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ClientRMProtocolPBClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ClientRMProtocolPBServiceImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java


 Handle exception related changes in YARN's ClientRMProtocol api after YARN-631
 --

 Key: MAPREDUCE-5212
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5212
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
 Fix For: 2.0.5-beta

 Attachments: MR-5212.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652974#comment-13652974
 ] 

Hudson commented on MAPREDUCE-5159:
---

Integrated in Hadoop-Mapreduce-trunk #1420 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1420/])
MAPREDUCE-5159. Change ValueAggregatorJob to add APIs which can support 
binary compatibility with hadoop-1 examples. Contributed by Zhijie Shen. 
(Revision 1480394)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480394
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorJob.java


 Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary 
 compatible with hadoop-2 mapred.lib.aggregate
 --

 Key: MAPREDUCE-5159
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5159.1.patch


 Both examples in hadoop-1 use 
 org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String
  args[], Class? caller), which no longer exists after upgrading to 
 hadoop-2. Therefore, they cannot not find the accordant function with their 
 function signature in the binary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652976#comment-13652976
 ] 

Hudson commented on MAPREDUCE-5157:
---

Integrated in Hadoop-Mapreduce-trunk #1420 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1420/])
MAPREDUCE-5157. Bring back old sampler related code so that we can support 
binary compatibility with hadoop-1 sorter example. Contributed by Zhijie Shen. 
(Revision 1480474)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480474
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/InputSampler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestInputSampler.java


 Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
 ---

 Key: MAPREDUCE-5157
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5157.1.patch, MAPREDUCE-5157.1.patch


 In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in 
 hadoop-1. However, after upgrading to hadoop-2, 
 org.apache.hadoop.mapred.lib.InputSampler is modified to extend 
 org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, 
 Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not 
 find Sampler when it runs with hadoop-2 framework.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5209) ShuffleScheduler log message incorrect


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652977#comment-13652977
 ] 

Hudson commented on MAPREDUCE-5209:
---

Integrated in Hadoop-Mapreduce-trunk #1420 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1420/])
MAPREDUCE-5209. Fix units in a ShuffleScheduler log message.
Contributed by Tsuyoshi OZAWA (Revision 1480464)

 Result = FAILURE
cdouglas : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480464
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java


 ShuffleScheduler log message incorrect
 --

 Key: MAPREDUCE-5209
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5209
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Radim Kolar
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5209.1.patch


 In ShuffleScheduler.java line 361 log message is incorrect, there should be 
 ms instead of s.
 LOG.info(host +  freed by  + Thread.currentThread().getName() +  in  
 + 
  (System.currentTimeMillis()-shuffleStart.get()) + ms);

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5210) Job submission has strict permission validation

2013-05-09 Thread Harsh J (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653007#comment-13653007
 ] 

Harsh J commented on MAPREDUCE-5210:


Changing the FileSystem interface would amount to an incompatible change. Must 
we really introduce an API for an assumed functionality?

Has someone also checked if the original description (which is a check that 
came in with MR security I think, but has no condition wrap) is the only point 
of issue or if there are other points where things have to be conditionally 
wrapped to support a non-true-DFS like S3 to _run_ MR (should we support that)?

 Job submission has strict permission validation
 ---

 Key: MAPREDUCE-5210
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5210
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Amareshwari Sriramadasu
Assignee: samar

 The following code in JobSubmissionFiles.java mandates strict permission on 
 job submission :
 {noformat}
 if (fs.exists(stagingArea)) {
   FileStatus fsStatus = fs.getFileStatus(stagingArea);
   String owner = fsStatus.getOwner();
   if (!(owner.equals(currentUser) || owner.equals(realUser))) {
  throw new IOException(The ownership on the staging directory  +
   stagingArea +  is not as expected.  + 
   It is owned by  + owner + . The directory must  +
   be owned by the submitter  + currentUser +  or  +
   by  + realUser);
   }
 {noformat}
 For file systems such as S3, which do not have permission concept, user can 
 never submit a job with staging area in S3. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsInst should standardize naming rules


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated MAPREDUCE-5227:
--

Assignee: Tsuyoshi OZAWA

 JobTrackerMetricsInst should standardize naming rules
 -

 Key: MAPREDUCE-5227
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 1.1.2
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor

 JobTrackerMetricsInst provides users with some metrics, but its naming rules( 
 jobs_running, running_maps, running_reduces) sometimes confuses users. 
 It should be standardized.
 One concern is backward compatibility, so one idea is to share 
 MetricMutableGaugeInt object from old and new property name.
 e.g. to share runningMaps from running_maps and maps_running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated MAPREDUCE-5227:
--

Summary: JobTrackerMetricsSource and QueueMetrics should standardize naming 
rules  (was: JobTrackerMetricsInst should standardize naming rules)

 JobTrackerMetricsSource and QueueMetrics should standardize naming rules
 

 Key: MAPREDUCE-5227
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 1.1.2
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor

 JobTrackerMetricsInst provides users with some metrics, but its naming rules( 
 jobs_running, running_maps, running_reduces) sometimes confuses users. 
 It should be standardized.
 One concern is backward compatibility, so one idea is to share 
 MetricMutableGaugeInt object from old and new property name.
 e.g. to share runningMaps from running_maps and maps_running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated MAPREDUCE-5227:
--

Attachment: MAPREDUCE-5227.1.patch

1. Moved prefixed attributes to suffixed atrbiutes.
2. (map|reduce)_blacklisted means blacklisted slots actually, therefore renamed 
to map_slots_blacklisted and reduce_slots_blacklisted.

 JobTrackerMetricsSource and QueueMetrics should standardize naming rules
 

 Key: MAPREDUCE-5227
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 1.1.2
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Attachments: MAPREDUCE-5227.1.patch


 JobTrackerMetricsInst provides users with some metrics, but its naming rules( 
 jobs_running, running_maps, running_reduces) sometimes confuses users. 
 It should be standardized.
 One concern is backward compatibility, so one idea is to share 
 MetricMutableGaugeInt object from old and new property name.
 e.g. to share runningMaps from running_maps and maps_running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules

[
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tsuyoshi OZAWA updated MAPREDUCE-5227:
--

Description:
JobTrackerMetricsSource and QueueMetrics provides users with some metrics, but
its naming rules( jobs_running, running_maps, running_reduces) sometimes
confuses users. It should be standardized.

One concern is backward compatibility, so one idea is to share
MetricMutableGaugeInt object from old and new property name.
e.g. to share runningMaps from running_maps and maps_running.

was:
JobTrackerMetricsInst provides users with some metrics, but its naming rules(
jobs_running, running_maps, running_reduces) sometimes confuses users. It
should be standardized.

One concern is backward compatibility, so one idea is to share
MetricMutableGaugeInt object from old and new property name.
e.g. to share runningMaps from running_maps and maps_running.

JobTrackerMetricsSource and QueueMetrics should standardize naming rules

Key: MAPREDUCE-5227
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1.1.2
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
Attachments: MAPREDUCE-5227.1.patch

JobTrackerMetricsSource and QueueMetrics provides users with some metrics,
but its naming rules( jobs_running, running_maps, running_reduces)
sometimes confuses users. It should be standardized.
One concern is backward compatibility, so one idea is to share
MetricMutableGaugeInt object from old and new property name.
e.g. to share runningMaps from running_maps and maps_running.

[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated MAPREDUCE-5227:
--

 Target Version/s: 1.1.3, 1.2.1
Affects Version/s: (was: 1.1.2)
   1.2.1
   1.1.3
   Status: Patch Available  (was: Open)

 JobTrackerMetricsSource and QueueMetrics should standardize naming rules
 

 Key: MAPREDUCE-5227
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 1.1.3, 1.2.1
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Attachments: MAPREDUCE-5227.1.patch


 JobTrackerMetricsSource and QueueMetrics provides users with some metrics, 
 but its naming rules( jobs_running, running_maps, running_reduces) 
 sometimes confuses users. It should be standardized.
 One concern is backward compatibility, so one idea is to share 
 MetricMutableGaugeInt object from old and new property name.
 e.g. to share runningMaps from running_maps and maps_running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules

[
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653043#comment-13653043
]

Hadoop QA commented on MAPREDUCE-5227:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12582487/MAPREDUCE-5227.1.patch
against trunk revision .

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3604//console

This message is automatically generated.

JobTrackerMetricsSource and QueueMetrics should standardize naming rules

Key: MAPREDUCE-5227
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1.1.3, 1.2.1
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
Attachments: MAPREDUCE-5227.1.patch

[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated MAPREDUCE-5227:
--

Attachment: MAPREDUCE-5227-1.1-branch.1.patch

Fixed the filename of the patch to apply to 1.1-branch.

 JobTrackerMetricsSource and QueueMetrics should standardize naming rules
 

 Key: MAPREDUCE-5227
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 1.1.3, 1.2.1
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Attachments: MAPREDUCE-5227-1.1-branch.1.patch, MAPREDUCE-5227.1.patch


 JobTrackerMetricsSource and QueueMetrics provides users with some metrics, 
 but its naming rules( jobs_running, running_maps, running_reduces) 
 sometimes confuses users. It should be standardized.
 One concern is backward compatibility, so one idea is to share 
 MetricMutableGaugeInt object from old and new property name.
 e.g. to share runningMaps from running_maps and maps_running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules

[
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653053#comment-13653053
]

Hadoop QA commented on MAPREDUCE-5227:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12582490/MAPREDUCE-5227-1.1-branch.1.patch
against trunk revision .

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3605//console

This message is automatically generated.

JobTrackerMetricsSource and QueueMetrics should standardize naming rules

Key: MAPREDUCE-5227
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1.1.3, 1.2.1
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
Attachments: MAPREDUCE-5227-1.1-branch.1.patch, MAPREDUCE-5227.1.patch

[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster

[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653059#comment-13653059
]

Alejandro Abdelnur commented on MAPREDUCE-5217:
---

Venkat, Oozie works with distcp on secure clusters. Please check OOZIE-611
which is adding the missing documentation for it. Please check if this solves
your problem.

DistCp fails when launched by Oozie in a secure cluster
---

[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated MAPREDUCE-5227:
--

Attachment: MAPREDUCE-5227-branch-1.1.patch

Fixed typo.

 JobTrackerMetricsSource and QueueMetrics should standardize naming rules
 

 Key: MAPREDUCE-5227
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 1.1.3, 1.2.1
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Attachments: MAPREDUCE-5227-1.1-branch.1.patch, 
 MAPREDUCE-5227.1.patch, MAPREDUCE-5227-branch-1.1.patch


 JobTrackerMetricsSource and QueueMetrics provides users with some metrics, 
 but its naming rules( jobs_running, running_maps, running_reduces) 
 sometimes confuses users. It should be standardized.
 One concern is backward compatibility, so one idea is to share 
 MetricMutableGaugeInt object from old and new property name.
 e.g. to share runningMaps from running_maps and maps_running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules

[
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653064#comment-13653064
]

Hadoop QA commented on MAPREDUCE-5227:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12582492/MAPREDUCE-5227-branch-1.1.patch
against trunk revision .

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3606//console

This message is automatically generated.

JobTrackerMetricsSource and QueueMetrics should standardize naming rules

Key: MAPREDUCE-5227
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1.1.3, 1.2.1
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
Attachments: MAPREDUCE-5227-1.1-branch.1.patch,
MAPREDUCE-5227.1.patch, MAPREDUCE-5227-branch-1.1.patch

[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653113#comment-13653113
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4942:
---

+1

 mapreduce.Job has a bunch of methods that throw InterruptedException so its 
 incompatible with MR1
 -

 Key: MAPREDUCE-4942
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
 Attachments: MAPREDUCE-4942.patch


 The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an 
 {{InterruptedException}} but don't in MR1, which makes them incompatible.  
 (Their Javadoc comments are also missing that they throw an 
 {{InterruptedException}} anyway)
 I propose that we wrap the {{InterruptedException}} in a 
 {{RuntimeException}}.  
 {code}
 public float setupProgress() throws IOException, InterruptedException
 public float mapProgress() throws IOException, InterruptedException
 public float reduceProgress() throws IOException, InterruptedException
 public boolean isComplete() throws IOException, InterruptedException
 public boolean isSuccessful() throws IOException, InterruptedException
 public void killJob() throws IOException, InterruptedException
 public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws 
 IOException, InterruptedException
 public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws 
 IOException, InterruptedException
 public Counters getCounters() throws IOException, InterruptedException
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce

Zhijie Shen created MAPREDUCE-5228:
--

 Summary: Enum Counter is removed from FileInputFormat and 
FileOutputFormat of both mapred and mapreduce
 Key: MAPREDUCE-5228
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen


The enum was used by findCounter(Enum key) to find a specific counter object. 
Now it seems to be replaced by FileInputFormatCounter and 
FileOutputFormatCounter. Now the enum seems to be only used internally, but not 
sure whether it will be used externally when users extend FileFormat.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated MAPREDUCE-5228:
---

Issue Type: Sub-task  (was: Bug)
Parent: MAPREDUCE-5108

 Enum Counter is removed from FileInputFormat and FileOutputFormat of both 
 mapred and mapreduce
 --

 Key: MAPREDUCE-5228
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 The enum was used by findCounter(Enum key) to find a specific counter object. 
 Now it seems to be replaced by FileInputFormatCounter and 
 FileOutputFormatCounter. Now the enum seems to be only used internally, but 
 not sure whether it will be used externally when users extend FileFormat.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653120#comment-13653120
 ] 

Zhijie Shen commented on MAPREDUCE-5220:


Notice this as well, but it seems that the user code should not use 
TaskCompletionEvent.

 Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
 

 Key: MAPREDUCE-5220
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: client
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.0.5-beta




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653125#comment-13653125
 ] 

Hudson commented on MAPREDUCE-4942:
---

Integrated in Hadoop-trunk-Commit #3725 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3725/])
MAPREDUCE-4942. mapreduce.Job has a bunch of methods that throw 
InterruptedException so its incompatible with MR1. (rkanter via tucu) (Revision 
1480748)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1480748
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
* 
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java


 mapreduce.Job has a bunch of methods that throw InterruptedException so its 
 incompatible with MR1
 -

 Key: MAPREDUCE-4942
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
 Attachments: MAPREDUCE-4942.patch


 The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an 
 {{InterruptedException}} but don't in MR1, which makes them incompatible.  
 (Their Javadoc comments are also missing that they throw an 
 {{InterruptedException}} anyway)
 I propose that we wrap the {{InterruptedException}} in a 
 {{RuntimeException}}.  
 {code}
 public float setupProgress() throws IOException, InterruptedException
 public float mapProgress() throws IOException, InterruptedException
 public float reduceProgress() throws IOException, InterruptedException
 public boolean isComplete() throws IOException, InterruptedException
 public boolean isSuccessful() throws IOException, InterruptedException
 public void killJob() throws IOException, InterruptedException
 public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws 
 IOException, InterruptedException
 public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws 
 IOException, InterruptedException
 public Counters getCounters() throws IOException, InterruptedException
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653127#comment-13653127
 ] 

Sandy Ryza commented on MAPREDUCE-5220:
---

Why not?  RunningJob, which is a user-facing interface, has 
getTaskCompletionEvents().

 Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
 

 Key: MAPREDUCE-5220
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: client
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.0.5-beta




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5038) old API CombineFileInputFormat missing fixes that are in new API


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653132#comment-13653132
 ] 

Sandy Ryza commented on MAPREDUCE-5038:
---

I've spent a while attempting to run Hive 0.11.0 tests over a version of Hadoop 
that includes this patch, due to unfamiliarity with ant and the Hive build 
system, have been unable to figure out how to do it.  I've tried copying into 
Hive the MapReduce code that touches the Paths in between Hive and where that 
exception is hit, and I verified that on my setup it produces the correct URL, 
with localhost instead of : in the authority.  Is someone who's more 
familiar with Hive able to figure out where that : in the authority is coming 
from?

 old API CombineFileInputFormat missing fixes that are in new API 
 -

 Key: MAPREDUCE-5038
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5038
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 1.3.0

 Attachments: MAPREDUCE-5038-1.patch, MAPREDUCE-5038.patch, 
 MAPREDUCE-5038-revised-1.patch, MAPREDUCE-5038-revised-1.patch, 
 MAPREDUCE-5038-revised.patch


 The following changes patched the CombineFileInputFormat in mapreduce, but 
 neglected the one in mapred
 MAPREDUCE-1597 enabled the CombineFileInputFormat to work on splittable files
 MAPREDUCE-2021 solved returning duplicate hostnames in split locations
 MAPREDUCE-1806 CombineFileInputFormat does not work with paths not on default 
 FS
 In trunk this is not an issue as the one in mapred extends the one in 
 mapreduce.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5210) Job submission has strict permission validation

2013-05-09 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653141#comment-13653141
 ] 

Steve Loughran commented on MAPREDUCE-5210:
---

S3 is not a filesystem; it, Swift and others are blobstore, with no real notion 
of directories, renames and deletes are non-atomic, time to retrieve data is 
potentially both slow and nondeterministic.

Rather than try and make changes throughout future versions of Hadoop to 
accomodate the use of S3 as a staging area, can I ask a question: Why do you 
need to do this? 

 Job submission has strict permission validation
 ---

 Key: MAPREDUCE-5210
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5210
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Amareshwari Sriramadasu
Assignee: samar

 The following code in JobSubmissionFiles.java mandates strict permission on 
 job submission :
 {noformat}
 if (fs.exists(stagingArea)) {
   FileStatus fsStatus = fs.getFileStatus(stagingArea);
   String owner = fsStatus.getOwner();
   if (!(owner.equals(currentUser) || owner.equals(realUser))) {
  throw new IOException(The ownership on the staging directory  +
   stagingArea +  is not as expected.  + 
   It is owned by  + owner + . The directory must  +
   be owned by the submitter  + currentUser +  or  +
   by  + realUser);
   }
 {noformat}
 For file systems such as S3, which do not have permission concept, user can 
 never submit a job with staging area in S3. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5229) TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce

Zhijie Shen created MAPREDUCE-5229:
--

 Summary: TEMP_DIR_NAME is removed from of FileOutputCommitter of 
mapreduce
 Key: MAPREDUCE-5229
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5229
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen


TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce. As 
FileOutputFormat and FileOutputCommitter may be extend by users, and the 
extended user classes can configured to use. Therefore, this missing public 
static variable may cause compatibility problem when the extended classes refer 
it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5229) TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated MAPREDUCE-5229:
---

Issue Type: Sub-task  (was: Bug)
Parent: MAPREDUCE-5108

 TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce
 -

 Key: MAPREDUCE-5229
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5229
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce. As 
 FileOutputFormat and FileOutputCommitter may be extend by users, and the 
 extended user classes can configured to use. Therefore, this missing public 
 static variable may cause compatibility problem when the extended classes 
 refer it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred

Zhijie Shen created MAPREDUCE-5230:
--

 Summary: createFileSplit is removed from NLineInputFormat of mapred
 Key: MAPREDUCE-5230
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen


createFileSplit is removed from NLineInputFormat of mapred, because it's no 
longer used in the new getSplit implementation. However, since function is 
protected before, there is still the potential risk that the user defined 
format class which extends old NLineInputFormat uses the protected function.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5231) Constructor of DBInputFormat.DBRecordReader in mapred is changed

Zhijie Shen created MAPREDUCE-5231:
--

 Summary: Constructor of DBInputFormat.DBRecordReader in mapred is 
changed
 Key: MAPREDUCE-5231
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5231
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen


The constructor of DBInputFormat.DBRecordReader in mapred is changed from MR1 
to RM2. Though MAPREDUCE-716 tried to deal with the API difference. However, if 
DBInputFormat.DBRecordReader is extended, the incompatibility around the 
constructor is still there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4942:
--

   Resolution: Fixed
Fix Version/s: 2.0.5-beta
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks Robert. Committed to trunk and branch-2.

 mapreduce.Job has a bunch of methods that throw InterruptedException so its 
 incompatible with MR1
 -

 Key: MAPREDUCE-4942
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-4942.patch


 The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an 
 {{InterruptedException}} but don't in MR1, which makes them incompatible.  
 (Their Javadoc comments are also missing that they throw an 
 {{InterruptedException}} anyway)
 I propose that we wrap the {{InterruptedException}} in a 
 {{RuntimeException}}.  
 {code}
 public float setupProgress() throws IOException, InterruptedException
 public float mapProgress() throws IOException, InterruptedException
 public float reduceProgress() throws IOException, InterruptedException
 public boolean isComplete() throws IOException, InterruptedException
 public boolean isSuccessful() throws IOException, InterruptedException
 public void killJob() throws IOException, InterruptedException
 public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws 
 IOException, InterruptedException
 public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws 
 IOException, InterruptedException
 public Counters getCounters() throws IOException, InterruptedException
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xi Fang updated MAPREDUCE-5224:
---

Attachment: MAPREDUCE-5224.patch

 JobTracker should allow the system directory to be in non-default FS
 

 Key: MAPREDUCE-5224
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
 Fix For: 1-win

 Attachments: MAPREDUCE-5224.patch


  JobTracker today expects the system directory to be in the default file 
 system
 if (fs == null) {
   fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
 public FileSystem run() throws IOException {
   return FileSystem.get(conf);
   }});
 }
 ...
   public String getSystemDir() {
 Path sysDir = new Path(conf.get(mapred.system.dir, 
 /tmp/hadoop/mapred/system));  
 return fs.makeQualified(sysDir).toString();
   }
 In Cloud like Azure the default file system is set as ASV (Windows Azure Blob 
 Storage), but we would still like the system directory to be in DFS. We 
 should change JobTracker to allow that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xi Fang updated MAPREDUCE-5224:
---

Attachment: (was: MAPREDUCE-5224.patch)

 JobTracker should allow the system directory to be in non-default FS
 

 Key: MAPREDUCE-5224
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
 Fix For: 1-win


  JobTracker today expects the system directory to be in the default file 
 system
 if (fs == null) {
   fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
 public FileSystem run() throws IOException {
   return FileSystem.get(conf);
   }});
 }
 ...
   public String getSystemDir() {
 Path sysDir = new Path(conf.get(mapred.system.dir, 
 /tmp/hadoop/mapred/system));  
 return fs.makeQualified(sysDir).toString();
   }
 In Cloud like Azure the default file system is set as ASV (Windows Azure Blob 
 Storage), but we would still like the system directory to be in DFS. We 
 should change JobTracker to allow that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5232) log classpath and other key properties on child JVM start

Sangjin Lee created MAPREDUCE-5232:
--

 Summary: log classpath and other key properties on child JVM start
 Key: MAPREDUCE-5232
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee


It would be great if we log vital information such as classpath, etc. upon a 
mapreduce child JVM start. This would help a great deal in terms of 
troubleshooting classpath issues, etc. Today it is pretty difficult to debug 
this unless you preserve the container script.

Maybe it can log things like classpath, os name/version, java version, etc. at 
the beginning of the child JVM start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653219#comment-13653219
 ] 

Alejandro Abdelnur commented on MAPREDUCE-5222:
---

[~ste...@apache.org], I'm not sure throwing an {{InterruptedIOException}} is 
the right thing to do, this exception is to signal an interruption during IO, 
which is not the case. The goal here is to keep API backwards compatibitility 
with Hadoop 1, in other places what has been done is to wrap the 
{{InterruptedException}} with a {{RuntimeException}}. IMO, we should do the 
same here.

 Add missing methods to JobClient 
 -

 Key: MAPREDUCE-5222
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 2.0.5-beta

 Attachments: mr-5222.patch, mr-5222.patch


 JobClient is missing the following two public methods we need to add for 
 binary compatibility:
 # static isJobDirValid(Path, FileSystem)
 # Path getStagingAreaDir()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653227#comment-13653227
 ] 

Sandy Ryza commented on MAPREDUCE-5232:
---

+1 to this

 log classpath and other key properties on child JVM start
 -

 Key: MAPREDUCE-5232
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee

 It would be great if we log vital information such as classpath, etc. upon a 
 mapreduce child JVM start. This would help a great deal in terms of 
 troubleshooting classpath issues, etc. Today it is pretty difficult to debug 
 this unless you preserve the container script.
 Maybe it can log things like classpath, os name/version, java version, etc. 
 at the beginning of the child JVM start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5218) Annotate (comment) internal classes as Private


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-5218:
--

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks Karthik. Committed to branch-1.

 Annotate (comment) internal classes as Private
 --

 Key: MAPREDUCE-5218
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5218
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.2
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Minor
 Fix For: 1.3.0

 Attachments: mr-5218-v0.patch


 The following classes are intended for internal use and it would be nice to 
 explicitly state that in comments/annotation.
 # TaskUmbilicalProtocol
 # TaskInProgress
 # MapReducePolicyProvider
 # MRAdmin?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5222) Add missing methods to JobClient

2013-05-09 Thread Karthik Kambatla (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-5222:


Attachment: mr-5222-rte.patch

Throwing RuntimeException instead of InterruptedIOException.

 Add missing methods to JobClient 
 -

 Key: MAPREDUCE-5222
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 2.0.5-beta

 Attachments: mr-5222.patch, mr-5222.patch, mr-5222-rte.patch


 JobClient is missing the following two public methods we need to add for 
 binary compatibility:
 # static isJobDirValid(Path, FileSystem)
 # Path getStagingAreaDir()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol

Zhijie Shen created MAPREDUCE-5233:
--

 Summary: Functions are changed or removed from Job in jobcontrol
 Key: MAPREDUCE-5233
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen


The functions are removed from Job in jobcontrol:

1. setMapredJobID(String)
2. setState(int)

The function signatures are changed:

1. addDependingJob(ControlledJob)
2. getMapredJobID()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xi Fang updated MAPREDUCE-5224:
---

Attachment: MAPREDUCE-5224.1.patch

 JobTracker should allow the system directory to be in non-default FS
 

 Key: MAPREDUCE-5224
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
 Fix For: 1-win

 Attachments: MAPREDUCE-5224.1.patch


  JobTracker today expects the system directory to be in the default file 
 system
 if (fs == null) {
   fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
 public FileSystem run() throws IOException {
   return FileSystem.get(conf);
   }});
 }
 ...
   public String getSystemDir() {
 Path sysDir = new Path(conf.get(mapred.system.dir, 
 /tmp/hadoop/mapred/system));  
 return fs.makeQualified(sysDir).toString();
   }
 In Cloud like Azure the default file system is set as ASV (Windows Azure Blob 
 Storage), but we would still like the system directory to be in DFS. We 
 should change JobTracker to allow that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xi Fang updated MAPREDUCE-5224:
---

Attachment: (was: MAPREDUCE-5224.1.patch)

 JobTracker should allow the system directory to be in non-default FS
 

 Key: MAPREDUCE-5224
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
 Fix For: 1-win


  JobTracker today expects the system directory to be in the default file 
 system
 if (fs == null) {
   fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
 public FileSystem run() throws IOException {
   return FileSystem.get(conf);
   }});
 }
 ...
   public String getSystemDir() {
 Path sysDir = new Path(conf.get(mapred.system.dir, 
 /tmp/hadoop/mapred/system));  
 return fs.makeQualified(sysDir).toString();
   }
 In Cloud like Azure the default file system is set as ASV (Windows Azure Blob 
 Storage), but we would still like the system directory to be in DFS. We 
 should change JobTracker to allow that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653277#comment-13653277
 ] 

Alejandro Abdelnur commented on MAPREDUCE-5222:
---

+1 pending Jenkins. Steve?

 Add missing methods to JobClient 
 -

 Key: MAPREDUCE-5222
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 2.0.5-beta

 Attachments: mr-5222.patch, mr-5222.patch, mr-5222-rte.patch


 JobClient is missing the following two public methods we need to add for 
 binary compatibility:
 # static isJobDirValid(Path, FileSystem)
 # Path getStagingAreaDir()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653280#comment-13653280
 ] 

Hadoop QA commented on MAPREDUCE-5222:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12582521/mr-5222-rte.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3607//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3607//console

This message is automatically generated.

 Add missing methods to JobClient 
 -

 Key: MAPREDUCE-5222
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 2.0.5-beta

 Attachments: mr-5222.patch, mr-5222.patch, mr-5222-rte.patch


 JobClient is missing the following two public methods we need to add for 
 binary compatibility:
 # static isJobDirValid(Path, FileSystem)
 # Path getStagingAreaDir()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5234) Signature changes for getTaskId of TaskReport in mapred

Zhijie Shen created MAPREDUCE-5234:
--

 Summary: Signature changes for getTaskId of TaskReport in mapred
 Key: MAPREDUCE-5234
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5234
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen


TaskReport in mapred of MR2 extends TaskReport in mapreduce, and inherits 
getTaskId, which return TaskID object. in MR1, this function returns String.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5234) Signature changes for getTaskId of TaskReport in mapred


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653287#comment-13653287
 ] 

Zhijie Shen commented on MAPREDUCE-5234:


Note that there's another function getTaskID, which was and is supposed to 
return TaskID.

 Signature changes for getTaskId of TaskReport in mapred
 ---

 Key: MAPREDUCE-5234
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5234
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta


 TaskReport in mapred of MR2 extends TaskReport in mapreduce, and inherits 
 getTaskId, which return TaskID object. in MR1, this function returns String.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4366) mapred metrics shows negative count of waiting maps and reduces


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653310#comment-13653310
 ] 

Sandy Ryza commented on MAPREDUCE-4366:
---

Uploaded a new patch that makes the variable naming more clear.  [~acmurthy], 
is that a satisfactory explanation?

 mapred metrics shows negative count of waiting maps and reduces
 ---

 Key: MAPREDUCE-4366
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4366
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 1.0.2
Reporter: Thomas Graves
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4366-branch-1-1.patch, 
 MAPREDUCE-4366-branch-1.patch


 Negative waiting_maps and waiting_reduces count is observed in the mapred 
 metrics.  MAPREDUCE-1238 partially fixed this but it appears there is still 
 issues as we are seeing it, but not as bad.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5235) MAX_GROUP_LIMIT is removed from Counters in mapred

Zhijie Shen created MAPREDUCE-5235:
--

 Summary: MAX_GROUP_LIMIT is removed from Counters in mapred
 Key: MAPREDUCE-5235
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5235
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen


MAX_GROUP_LIMIT is removed from Counters in mapred in MR2. Though it seems not 
to be the variable that will be referred by the user code. It was actually 
configurable value MR1. We should investigate why the upper bound doesn't need 
to be checked in MR2.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5236) JobConf should not support disabling memory limits

Sandy Ryza created MAPREDUCE-5236:
-

 Summary: JobConf should not support disabling memory limits
 Key: MAPREDUCE-5236
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5236
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza


In MR1, a special value of -1 could be given for 
mapreduce.job.map|reduce.memory.mb to disable memory limits.  In MR2, this 
makes no sense, as with slots gone, this value is used for requesting resources 
and scheduling.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on MAPREDUCE-5224 started by Xi Fang.

 JobTracker should allow the system directory to be in non-default FS
 

 Key: MAPREDUCE-5224
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
 Fix For: 1-win


  JobTracker today expects the system directory to be in the default file 
 system
 if (fs == null) {
   fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
 public FileSystem run() throws IOException {
   return FileSystem.get(conf);
   }});
 }
 ...
   public String getSystemDir() {
 Path sysDir = new Path(conf.get(mapred.system.dir, 
 /tmp/hadoop/mapred/system));  
 return fs.makeQualified(sysDir).toString();
   }
 In Cloud like Azure the default file system is set as ASV (Windows Azure Blob 
 Storage), but we would still like the system directory to be in DFS. We 
 should change JobTracker to allow that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-5232:
---

Attachment: MAPREDUCE-5232-branch-2.patch

Patch for branch-2

 log classpath and other key properties on child JVM start
 -

 Key: MAPREDUCE-5232
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee
 Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch


 It would be great if we log vital information such as classpath, etc. upon a 
 mapreduce child JVM start. This would help a great deal in terms of 
 troubleshooting classpath issues, etc. Today it is pretty difficult to debug 
 this unless you preserve the container script.
 Maybe it can log things like classpath, os name/version, java version, etc. 
 at the beginning of the child JVM start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-5232:
---

Attachment: MAPREDUCE-5232.patch

Patch for the trunk

 log classpath and other key properties on child JVM start
 -

 Key: MAPREDUCE-5232
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee
 Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch


 It would be great if we log vital information such as classpath, etc. upon a 
 mapreduce child JVM start. This would help a great deal in terms of 
 troubleshooting classpath issues, etc. Today it is pretty difficult to debug 
 this unless you preserve the container script.
 Maybe it can log things like classpath, os name/version, java version, etc. 
 at the beginning of the child JVM start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-5232:
---

Status: Patch Available  (was: Open)

 log classpath and other key properties on child JVM start
 -

 Key: MAPREDUCE-5232
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee
 Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch


 It would be great if we log vital information such as classpath, etc. upon a 
 mapreduce child JVM start. This would help a great deal in terms of 
 troubleshooting classpath issues, etc. Today it is pretty difficult to debug 
 this unless you preserve the container script.
 Maybe it can log things like classpath, os name/version, java version, etc. 
 at the beginning of the child JVM start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653385#comment-13653385
 ] 

Sangjin Lee commented on MAPREDUCE-5232:


I posted the proposed patch. I'm not 100% happy with the config names, and am 
open to suggestions.

 log classpath and other key properties on child JVM start
 -

 Key: MAPREDUCE-5232
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee
 Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch


 It would be great if we log vital information such as classpath, etc. upon a 
 mapreduce child JVM start. This would help a great deal in terms of 
 troubleshooting classpath issues, etc. Today it is pretty difficult to debug 
 this unless you preserve the container script.
 Maybe it can log things like classpath, os name/version, java version, etc. 
 at the beginning of the child JVM start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653387#comment-13653387
 ] 

Zhijie Shen commented on MAPREDUCE-5222:


Why testIsJobDirValid throws UnsupportedOperationException? job.xml and 
job.split are still among the submission resources, though dir may be 
different.

 Add missing methods to JobClient 
 -

 Key: MAPREDUCE-5222
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 2.0.5-beta

 Attachments: mr-5222.patch, mr-5222.patch, mr-5222-rte.patch


 JobClient is missing the following two public methods we need to add for 
 binary compatibility:
 # static isJobDirValid(Path, FileSystem)
 # Path getStagingAreaDir()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5237) Three functions are removed in ClusterStatus

Zhijie Shen created MAPREDUCE-5237:
--

 Summary: Three functions are removed in ClusterStatus 
 Key: MAPREDUCE-5237
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen


The three functions are:

CollectionString getGraylistedTrackerNames()
int getGraylistedTrackers()
State getJobTrackerState()

The're tracker related, such that they are no longer used in MR2. Maybe we 
should add them and throw UnsupportedOperationException?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653400#comment-13653400
 ] 

Hadoop QA commented on MAPREDUCE-5232:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12582546/MAPREDUCE-5232.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3608//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3608//console

This message is automatically generated.

 log classpath and other key properties on child JVM start
 -

 Key: MAPREDUCE-5232
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee
 Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch


 It would be great if we log vital information such as classpath, etc. upon a 
 mapreduce child JVM start. This would help a great deal in terms of 
 troubleshooting classpath issues, etc. Today it is pretty difficult to debug 
 this unless you preserve the container script.
 Maybe it can log things like classpath, os name/version, java version, etc. 
 at the beginning of the child JVM start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653412#comment-13653412
 ] 

Zhijie Shen commented on MAPREDUCE-5220:


I mean why users need to set the variables in TaskCompletionEvent. The 
variables should be set by the system, and users just read the them. 
protected should be the reasonable access control. 

 Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
 

 Key: MAPREDUCE-5220
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: client
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.0.5-beta




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653434#comment-13653434
 ] 

Alejandro Abdelnur commented on MAPREDUCE-5222:
---

(holding on my +1), on [~zjshen]'s comment, missed that. he has a point, 
throwing a UOE would break existing apps at runtime, isn't there any proper 
behavior we can give to that method?

 Add missing methods to JobClient 
 -

 Key: MAPREDUCE-5222
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 2.0.5-beta

 Attachments: mr-5222.patch, mr-5222.patch, mr-5222-rte.patch


 JobClient is missing the following two public methods we need to add for 
 binary compatibility:
 # static isJobDirValid(Path, FileSystem)
 # Path getStagingAreaDir()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653446#comment-13653446
 ] 

Sandy Ryza commented on MAPREDUCE-5130:
---

Alejandro convinced me that the slots based code changes should be done in a 
different JIRA.  Uploading a new patch without them.

 Add missing job config options to mapred-default.xml
 

 Key: MAPREDUCE-5130
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, 
 MAPREDUCE-5130-2.patch, MAPREDUCE-5130.patch


 I came across that mapreduce.map.child.java.opts and 
 mapreduce.reduce.child.java.opts were missing in mapred-default.xml.  I'll do 
 a fuller sweep to see what else is missing before posting a patch.
 List so far:
 mapreduce.map/reduce.child.java.opts
 mapreduce.map/reduce.memory.mb
 mapreduce.job.jvm.numtasks
 mapreduce.input.lineinputformat.linespermap
 mapreduce.task.combine.progress.records

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated MAPREDUCE-5130:
--

Attachment: MAPREDUCE-5130-2.patch

 Add missing job config options to mapred-default.xml
 

 Key: MAPREDUCE-5130
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, 
 MAPREDUCE-5130-2.patch, MAPREDUCE-5130.patch


 I came across that mapreduce.map.child.java.opts and 
 mapreduce.reduce.child.java.opts were missing in mapred-default.xml.  I'll do 
 a fuller sweep to see what else is missing before posting a patch.
 List so far:
 mapreduce.map/reduce.child.java.opts
 mapreduce.map/reduce.memory.mb
 mapreduce.job.jvm.numtasks
 mapreduce.input.lineinputformat.linespermap
 mapreduce.task.combine.progress.records

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xi Fang updated MAPREDUCE-5224:
---

Attachment: MAPREDUCE-5224.patch

 JobTracker should allow the system directory to be in non-default FS
 

 Key: MAPREDUCE-5224
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
 Fix For: 1-win

 Attachments: MAPREDUCE-5224.patch


  JobTracker today expects the system directory to be in the default file 
 system
 if (fs == null) {
   fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
 public FileSystem run() throws IOException {
   return FileSystem.get(conf);
   }});
 }
 ...
   public String getSystemDir() {
 Path sysDir = new Path(conf.get(mapred.system.dir, 
 /tmp/hadoop/mapred/system));  
 return fs.makeQualified(sysDir).toString();
   }
 In Cloud like Azure the default file system is set as ASV (Windows Azure Blob 
 Storage), but we would still like the system directory to be in DFS. We 
 should change JobTracker to allow that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653460#comment-13653460
 ] 

Hadoop QA commented on MAPREDUCE-5130:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12582556/MAPREDUCE-5130-2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 1367 javac 
compiler warnings (more than the trunk's current 1366 warnings).

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-tools/hadoop-gridmix:

  org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3609//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3609//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3609//console

This message is automatically generated.

 Add missing job config options to mapred-default.xml
 

 Key: MAPREDUCE-5130
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, 
 MAPREDUCE-5130-2.patch, MAPREDUCE-5130.patch


 I came across that mapreduce.map.child.java.opts and 
 mapreduce.reduce.child.java.opts were missing in mapred-default.xml.  I'll do 
 a fuller sweep to see what else is missing before posting a patch.
 List so far:
 mapreduce.map/reduce.child.java.opts
 mapreduce.map/reduce.memory.mb
 mapreduce.job.jvm.numtasks
 mapreduce.input.lineinputformat.linespermap
 mapreduce.task.combine.progress.records

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653473#comment-13653473
 ] 

Xi Fang commented on MAPREDUCE-5224:


The original motivation of this JIRA is trying to fix the following scenario. 
In Azure, the default file system is set as ASV (Windows Azure Blob Storage), 
but we would still like the system directory to be in DFS, because we don't 
want to put such files in ASV that charge Azure customers fee. Thus, we want to 
change JobTracker.java to allow that.

The problem in the current JobTracker.java is that we want to use 
makeQualified() to assemble a path. But getSystemDir() uses the wrong fs object 
to call fs.makeQualified(), if default (e.g. Azure in our scanerio) and 
mapred.system.dir are using different file systems. In the proposed fix, we 
rely on FileSystem.get() to choose the appropriate file system according to 
mapred.system.dir. It falls back on the default file system if the scheme is 
not there. 

Although the original motivation is trying to fix the problem for Azure, this 
fix also applies to other scenarios where the default file system and 
mapred.system.dir are supposed to use different file systems.

A unit test will follow.


 JobTracker should allow the system directory to be in non-default FS
 

 Key: MAPREDUCE-5224
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
 Fix For: 1-win

 Attachments: MAPREDUCE-5224.patch


  JobTracker today expects the system directory to be in the default file 
 system
 if (fs == null) {
   fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
 public FileSystem run() throws IOException {
   return FileSystem.get(conf);
   }});
 }
 ...
   public String getSystemDir() {
 Path sysDir = new Path(conf.get(mapred.system.dir, 
 /tmp/hadoop/mapred/system));  
 return fs.makeQualified(sysDir).toString();
   }
 In Cloud like Azure the default file system is set as ASV (Windows Azure Blob 
 Storage), but we would still like the system directory to be in DFS. We 
 should change JobTracker to allow that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml