Author: todd
Date: Mon Sep 10 18:45:45 2012
New Revision: 1383030
URL: http://svn.apache.org/viewvc?rev=1383030&view=rev
Log:
Merge trunk into QJM branch
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/ (props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/CHANGES.txt
(contents, props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/conf/ (props
changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
(contents, props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/c++/ (props
changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/block_forensics/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/build-contrib.xml
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/build.xml
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/data_join/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/eclipse-plugin/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/index/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/vaidya/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/examples/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/java/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/test/mapred/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc/
(props changed)
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/webapps/job/
(props changed)
Propchange: hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/
------------------------------------------------------------------------------
Merged /hadoop/common/trunk/hadoop-mapreduce-project:r1380987-1383029
Modified: hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/CHANGES.txt?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
--- hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/CHANGES.txt
(original)
+++ hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/CHANGES.txt Mon
Sep 10 18:45:45 2012
@@ -1,6 +1,6 @@
Hadoop MapReduce Change Log
-Trunk (unreleased changes)
+Trunk (Unreleased)
INCOMPATIBLE CHANGES
@@ -126,79 +126,29 @@ Trunk (unreleased changes)
MAPREDUCE-3868. Make Raid Compile. (Weiyan Wang via schen)
-Branch-2 ( Unreleased changes )
+Release 2.0.3-alpha - Unreleased
INCOMPATIBLE CHANGES
NEW FEATURES
- MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
- command-line. (ahmed via tucu)
-
- MAPREDUCE-4417. add support for encrypted shuffle (tucu)
-
IMPROVEMENTS
- MAPREDUCE-4157. ResourceManager should not kill apps that are well behaved
- (Jason Lowe via bobby)
-
- MAPREDUCE-4511. Add IFile readahead (ahmed via tucu)
-
- MAPREDUCE-4408. allow jobs to set a JAR that is in the distributed cached
- (rkanter via tucu)
-
- MAPREDUCE-2786. Add compression option for TestDFSIO.
- (Plamen Jeliazkov via shv)
+ OPTIMIZATIONS
BUG FIXES
- MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default
value in
- YarnConfiguration. (ahmed via tucu)
-
- MAPREDUCE-4406. Users should be able to specify the MiniCluster
ResourceManager
- and JobHistoryServer ports. (ahmed via tucu)
-
- MAPREDUCE-4407. Add hadoop-yarn-server-tests-<version>-tests.jar to hadoop
dist
- package. (ahmed via tucu)
-
- MAPREDUCE-4465. Update description of yarn.nodemanager.address property.
- (bowang via tucu)
-
- MAPREDUCE-4342. Distributed Cache gives inconsistent result if cache files
- get deleted from tasktracker. (mayank_bansal via tucu)
-
- MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter
via tucu)
-
- MAPREDUCE-4494. TestFifoScheduler failing with Metrics source
QueueMetrics,q0=default
- already exists!. (ahmed.radwan via tucu)
-
- MAPREDUCE-4484. Incorrect IS_MINI_YARN_CLUSTER property name in
YarnConfiguration.
- (ahmed.radwan via tucu)
-
- MAPREDUCE-4562. Support for "FileSystemCounter" legacy counter group name
- for compatibility reasons is creating incorrect counter name.
- (Jarek Jarcec Cecho via tomwhite)
-
- MAPREDUCE-4068. Jars in lib subdirectory of the submittable JAR are not
added to the
- classpath (rkanter via tucu)
-
- MAPREDUCE-4577. HDFS-3672 broke
- TestCombineFileInputFormat.testMissingBlocks() test. (atm)
-
- MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov
via tucu)
-
- MAPREDUCE-4608. hadoop-mapreduce-client is missing some dependencies.
- (tucu via tomwhite)
-
- MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in
- MR2. (tomwhite)
-
-Release 2.1.0-alpha - Unreleased
+Release 2.0.2-alpha - 2012-09-07
INCOMPATIBLE CHANGES
NEW FEATURES
+ MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
+ command-line. (ahmed via tucu)
+
+ MAPREDUCE-4417. add support for encrypted shuffle (tucu)
+
MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu)
@@ -208,6 +158,17 @@ Release 2.1.0-alpha - Unreleased
IMPROVEMENTS
+ MAPREDUCE-4157. ResourceManager should not kill apps that are well behaved
+ (Jason Lowe via bobby)
+
+ MAPREDUCE-4511. Add IFile readahead (ahmed via tucu)
+
+ MAPREDUCE-4408. allow jobs to set a JAR that is in the distributed cached
+ (rkanter via tucu)
+
+ MAPREDUCE-2786. Add compression option for TestDFSIO.
+ (Plamen Jeliazkov via shv)
+
MAPREDUCE-4440. Changed SchedulerApp and SchedulerNode to be a minimal
interface to allow schedulers to maintain their own. (acmurthy)
@@ -249,10 +210,56 @@ Release 2.1.0-alpha - Unreleased
MAPREDUCE-4579. Split TestTaskAttempt into two so as to pass tests on
jdk7. (Thomas Graves via vinodkv)
- OPTIMIZATIONS
+ MAPREDUCE-4638. MR AM supplies MapReduce jars in classpath rather than
+ rely on YARN. (acmurthy)
BUG FIXES
+ MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default
value in
+ YarnConfiguration. (ahmed via tucu)
+
+ MAPREDUCE-4406. Users should be able to specify the MiniCluster
ResourceManager
+ and JobHistoryServer ports. (ahmed via tucu)
+
+ MAPREDUCE-4407. Add hadoop-yarn-server-tests-<version>-tests.jar to hadoop
dist
+ package. (ahmed via tucu)
+
+ MAPREDUCE-4465. Update description of yarn.nodemanager.address property.
+ (bowang via tucu)
+
+ MAPREDUCE-4342. Distributed Cache gives inconsistent result if cache files
+ get deleted from tasktracker. (mayank_bansal via tucu)
+
+ MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter
via tucu)
+
+ MAPREDUCE-4494. TestFifoScheduler failing with Metrics source
QueueMetrics,q0=default
+ already exists!. (ahmed.radwan via tucu)
+
+ MAPREDUCE-4484. Incorrect IS_MINI_YARN_CLUSTER property name in
YarnConfiguration.
+ (ahmed.radwan via tucu)
+
+ MAPREDUCE-4562. Support for "FileSystemCounter" legacy counter group name
+ for compatibility reasons is creating incorrect counter name.
+ (Jarek Jarcec Cecho via tomwhite)
+
+ MAPREDUCE-4068. Jars in lib subdirectory of the submittable JAR are not
added to the
+ classpath (rkanter via tucu)
+
+ MAPREDUCE-4577. HDFS-3672 broke
+ TestCombineFileInputFormat.testMissingBlocks() test. (atm)
+
+ MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov
via tucu)
+
+ MAPREDUCE-4608. hadoop-mapreduce-client is missing some dependencies.
+ (tucu via tomwhite)
+
+ MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in
+ MR2. (tomwhite)
+
+ MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby)
+
+ MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar()
(rkanter via tucu)
+
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on
HDFS. (tomwhite)
@@ -880,6 +887,12 @@ Release 0.23.3 - UNRELEASED
mapreduce.reduce.maxattempts defaults are set to 4 as well as
mapreduce.job.maxtaskfailures.per.tracker. (Ravi Prakash via jeagles)
+ MAPREDUCE-4633. history server doesn't set permissions on all subdirs
+ (tgraves via bobby)
+
+ MAPREDUCE-4641. Exception in commitJob marks job as successful in job
+ history (Jason Lowe via bobby)
+
Release 0.23.2 - UNRELEASED
INCOMPATIBLE CHANGES
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/CHANGES.txt
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt:r1380987-1383029
Propchange: hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/conf/
------------------------------------------------------------------------------
Merged /hadoop/common/trunk/hadoop-mapreduce-project/conf:r1380987-1383029
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
Mon Sep 10 18:45:45 2012
@@ -759,7 +759,8 @@ public class JobImpl implements org.apac
job.getCommitter().commitJob(job.getJobContext());
} catch (IOException e) {
LOG.error("Could not do commit for Job", e);
- job.logJobHistoryFinishedEvent();
+ job.addDiagnostic("Job commit failed: " + e.getMessage());
+ job.abortJob(org.apache.hadoop.mapreduce.JobStatus.State.FAILED);
return job.finished(JobState.FAILED);
}
job.logJobHistoryFinishedEvent();
@@ -1199,7 +1200,7 @@ public class JobImpl implements org.apac
}
}
- private void abortJob(
+ protected void abortJob(
org.apache.hadoop.mapreduce.JobStatus.State finalState) {
try {
committer.abortJob(jobContext, finalState);
@@ -1501,7 +1502,7 @@ public class JobImpl implements org.apac
}
}
- private void addDiagnostic(String diag) {
+ protected void addDiagnostic(String diag) {
diagnostics.add(diag);
}
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
Mon Sep 10 18:45:45 2012
@@ -19,9 +19,11 @@
package org.apache.hadoop.mapreduce.v2.app.job.impl;
import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.io.IOException;
@@ -91,8 +93,6 @@ public class TestJobImpl {
when(mockJob.getCommitter()).thenReturn(mockCommitter);
when(mockJob.getEventHandler()).thenReturn(mockEventHandler);
when(mockJob.getJobContext()).thenReturn(mockJobContext);
- doNothing().when(mockJob).setFinishTime();
- doNothing().when(mockJob).logJobHistoryFinishedEvent();
when(mockJob.finished(JobState.KILLED)).thenReturn(JobState.KILLED);
when(mockJob.finished(JobState.FAILED)).thenReturn(JobState.FAILED);
when(mockJob.finished(JobState.SUCCEEDED)).thenReturn(JobState.SUCCEEDED);
@@ -103,11 +103,13 @@ public class TestJobImpl {
// commitJob stubbed out, so this can't happen
}
doNothing().when(mockEventHandler).handle(any(JobHistoryEvent.class));
+ JobState jobState = JobImpl.checkJobCompleteSuccess(mockJob);
Assert.assertNotNull("checkJobCompleteSuccess incorrectly returns null " +
- "for successful job",
- JobImpl.checkJobCompleteSuccess(mockJob));
+ "for successful job", jobState);
Assert.assertEquals("checkJobCompleteSuccess returns incorrect state",
- JobState.FAILED, JobImpl.checkJobCompleteSuccess(mockJob));
+ JobState.FAILED, jobState);
+ verify(mockJob).abortJob(
+ eq(org.apache.hadoop.mapreduce.JobStatus.State.FAILED));
}
@Test
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
Mon Sep 10 18:45:45 2012
@@ -61,11 +61,6 @@ public class JHAdminConfig {
MR_HISTORY_PREFIX + "datestring.cache.size";
public static final int DEFAULT_MR_HISTORY_DATESTRING_CACHE_SIZE = 200000;
- //TODO REMOVE debug-mode
- /** Equivalent to 0.20 mapreduce.jobhistory.debug.mode */
- public static final String MR_HISTORY_DEBUG_MODE =
- MR_HISTORY_PREFIX + "debug-mode";
-
/** Path where history files should be stored for DONE jobs. **/
public static final String MR_HISTORY_DONE_DIR =
MR_HISTORY_PREFIX + "done-dir";
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
Mon Sep 10 18:45:45 2012
@@ -79,6 +79,13 @@ public class JobHistoryUtils {
public static final FsPermission HISTORY_DONE_FILE_PERMISSION =
FsPermission.createImmutable((short) 0770); // rwx------
+
+ /**
+ * Umask for the done dir and derivatives.
+ */
+ public static final FsPermission HISTORY_DONE_DIR_UMASK = FsPermission
+ .createImmutable((short) (0770 ^ 0777));
+
/**
* Permissions for the intermediate done directory.
@@ -336,20 +343,19 @@ public class JobHistoryUtils {
/**
* Gets the timestamp component based on millisecond time.
* @param millisecondTime
- * @param debugMode
* @return the timestamp component based on millisecond time
*/
- public static String timestampDirectoryComponent(long millisecondTime,
boolean debugMode) {
+ public static String timestampDirectoryComponent(long millisecondTime) {
Calendar timestamp = Calendar.getInstance();
timestamp.setTimeInMillis(millisecondTime);
String dateString = null;
- dateString = String.format(
- TIMESTAMP_DIR_FORMAT,
- timestamp.get(Calendar.YEAR),
- // months are 0-based in Calendar, but people will expect January
- // to be month #1.
- timestamp.get(debugMode ? Calendar.HOUR : Calendar.MONTH) + 1,
- timestamp.get(debugMode ? Calendar.MINUTE : Calendar.DAY_OF_MONTH));
+ dateString = String
+ .format(TIMESTAMP_DIR_FORMAT,
+ timestamp.get(Calendar.YEAR),
+ // months are 0-based in Calendar, but people will expect January
to
+ // be month #1.
+ timestamp.get(Calendar.MONTH) + 1,
+ timestamp.get(Calendar.DAY_OF_MONTH));
dateString = dateString.intern();
return dateString;
}
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
Mon Sep 10 18:45:45 2012
@@ -179,6 +179,12 @@ public class MRApps extends Apps {
Apps.addToEnvironment(environment, Environment.CLASSPATH.name(), c
.trim());
}
+ for (String c : conf.getStrings(
+ MRJobConfig.MAPREDUCE_APPLICATION_CLASSPATH,
+ MRJobConfig.DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH)) {
+ Apps.addToEnvironment(environment, Environment.CLASSPATH.name(), c
+ .trim());
+ }
} finally {
if (classpathFileStream != null) {
classpathFileStream.close();
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
Mon Sep 10 18:45:45 2012
@@ -140,11 +140,19 @@ public class TestMRApps {
Map<String, String> environment = new HashMap<String, String>();
MRApps.setClasspath(environment, job.getConfiguration());
assertTrue(environment.get("CLASSPATH").startsWith("$PWD:"));
- String confClasspath =
job.getConfiguration().get(YarnConfiguration.YARN_APPLICATION_CLASSPATH);
- if (confClasspath != null) {
- confClasspath = confClasspath.replaceAll(",\\s*", ":").trim();
+ String yarnAppClasspath =
+ job.getConfiguration().get(
+ YarnConfiguration.YARN_APPLICATION_CLASSPATH);
+ if (yarnAppClasspath != null) {
+ yarnAppClasspath = yarnAppClasspath.replaceAll(",\\s*", ":").trim();
}
- assertTrue(environment.get("CLASSPATH").contains(confClasspath));
+ assertTrue(environment.get("CLASSPATH").contains(yarnAppClasspath));
+ String mrAppClasspath =
+
job.getConfiguration().get(MRJobConfig.MAPREDUCE_APPLICATION_CLASSPATH);
+ if (mrAppClasspath != null) {
+ mrAppClasspath = mrAppClasspath.replaceAll(",\\s*", ":").trim();
+ }
+ assertTrue(environment.get("CLASSPATH").contains(mrAppClasspath));
}
@Test public void testSetClasspathWithUserPrecendence() {
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
Mon Sep 10 18:45:45 2012
@@ -587,4 +587,18 @@ public interface MRJobConfig {
MR_AM_SECURITY_SERVICE_AUTHORIZATION_CLIENT =
"security.job.client.protocol.acl";
+ /**
+ * CLASSPATH for all YARN MapReduce applications.
+ */
+ public static final String MAPREDUCE_APPLICATION_CLASSPATH =
+ "mapreduce.application.classpath";
+
+ /**
+ * Default CLASSPATH for all YARN MapReduce applications.
+ */
+ public static final String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH = {
+ "$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*",
+ "$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*",
+ };
+
}
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
Mon Sep 10 18:45:45 2012
@@ -1203,6 +1203,8 @@
<!-- end of node health script variables -->
+<!-- MR YARN Application properties -->
+
<property>
<name>mapreduce.job.counters.limit</name>
<value>120</value>
@@ -1312,6 +1314,13 @@
<description>The amount of memory the MR AppMaster needs.</description>
</property>
+<property>
+ <description>CLASSPATH for MR applications. A comma-separated list
+ of CLASSPATH entries</description>
+ <name>mapreduce.application.classpath</name>
+
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
+</property>
+
<!-- jobhistory properties -->
<property>
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml:r1380987-1383029
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
Mon Sep 10 18:45:45 2012
@@ -358,7 +358,6 @@ public class HistoryFileManager extends
private Configuration conf;
- private boolean debugMode;
private String serialNumberFormat;
private Path doneDirPrefixPath = null; // folder for completed jobs
@@ -379,8 +378,7 @@ public class HistoryFileManager extends
public void init(Configuration conf) {
this.conf = conf;
- debugMode = conf.getBoolean(JHAdminConfig.MR_HISTORY_DEBUG_MODE, false);
- int serialNumberLowDigits = debugMode ? 1 : 3;
+ int serialNumberLowDigits = 3;
serialNumberFormat = ("%0"
+ (JobHistoryUtils.SERIAL_NUMBER_DIRECTORY_DIGITS +
serialNumberLowDigits)
+ "d");
@@ -392,6 +390,7 @@ public class HistoryFileManager extends
doneDirPrefixPath = FileContext.getFileContext(conf).makeQualified(
new Path(doneDirPrefix));
doneDirFc = FileContext.getFileContext(doneDirPrefixPath.toUri(), conf);
+ doneDirFc.setUMask(JobHistoryUtils.HISTORY_DONE_DIR_UMASK);
mkdir(doneDirFc, doneDirPrefixPath, new FsPermission(
JobHistoryUtils.HISTORY_DONE_DIR_PERMISSION));
} catch (IOException e) {
@@ -779,8 +778,8 @@ public class HistoryFileManager extends
}
private Path canonicalHistoryLogPath(JobId id, long millisecondTime) {
- String timestampComponent = JobHistoryUtils.timestampDirectoryComponent(
- millisecondTime, debugMode);
+ String timestampComponent = JobHistoryUtils
+ .timestampDirectoryComponent(millisecondTime);
return new Path(doneDirPrefixPath, JobHistoryUtils.historyLogSubdirectory(
id, timestampComponent, serialNumberFormat));
}
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
Mon Sep 10 18:45:45 2012
@@ -43,7 +43,7 @@ import org.apache.hadoop.yarn.api.record
import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.util.ProtoUtils;
-import org.hadoop.yarn.client.YarnClientImpl;
+import org.apache.hadoop.yarn.client.YarnClientImpl;
public class ResourceMgrDelegate extends YarnClientImpl {
private static final Log LOG = LogFactory.getLog(ResourceMgrDelegate.class);
Modified:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java?rev=1383030&r1=1383029&r2=1383030&view=diff
==============================================================================
---
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
(original)
+++
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
Mon Sep 10 18:45:45 2012
@@ -58,8 +58,12 @@ public class MiniMRClientClusterFactory
Job job = Job.getInstance(conf);
job.addFileToClassPath(appJar);
- String callerJar = JarFinder.getJar(caller);
- job.setJar(callerJar);
+
+ Path callerJar = new Path(JarFinder.getJar(caller));
+ Path remoteCallerJar = new Path(testRootDir, callerJar.getName());
+ fs.copyFromLocalFile(callerJar, remoteCallerJar);
+ fs.setPermission(remoteCallerJar, new FsPermission("744"));
+ job.addFileToClassPath(remoteCallerJar);
MiniMRYarnCluster miniMRYarnCluster = new MiniMRYarnCluster(caller
.getName(), noOfNMs);
Propchange: hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/c++/
------------------------------------------------------------------------------
Merged /hadoop/common/trunk/hadoop-mapreduce-project/src/c++:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/block_forensics/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/block_forensics:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/build-contrib.xml
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/build-contrib.xml:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/build.xml
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/build.xml:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/data_join/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/eclipse-plugin/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/eclipse-plugin:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/index/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/index:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/contrib/vaidya/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/vaidya:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/examples/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/examples:r1380987-1383029
Propchange: hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/java/
------------------------------------------------------------------------------
Merged /hadoop/common/trunk/hadoop-mapreduce-project/src/java:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/test/mapred/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc:r1380987-1383029
Propchange:
hadoop/common/branches/HDFS-3077/hadoop-mapreduce-project/src/webapps/job/
------------------------------------------------------------------------------
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job:r1380987-1383029
