[jira] [Resolved] (MAPREDUCE-6327) [Event producers] Implement MapReduce AM writing MR events/counters to v2 ATS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-6327. Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Committed the patch to branch YARN-2928. Thanks, Junping! Also thanks for review, Sangjin and Robert! Let's continue the the improvement in the follow up jira: MAPREDUCE-6318 [Event producers] Implement MapReduce AM writing MR events/counters to v2 ATS - Key: MAPREDUCE-6327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6327 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sangjin Lee Assignee: Junping Du Fix For: 2.8.0 Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch, YARN-3046-v5.patch, YARN-3046-v6.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6202) TestMRTimelineEventHandling fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen resolved MAPREDUCE-6202.
Resolution: Duplicate
TestMRTimelineEventHandling fails on trunk
--
Key: MAPREDUCE-6202
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6202
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Robert Kanter
Assignee: Zhijie Shen
Currently, {{TestMRTimelineEventHandling}} is failing on trunk:
{noformat}
Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 181.186 sec
FAILURE! - in org.apache.hadoop.mapred.TestMRTimelineEventHandling
testMRTimelineEventHandling(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
Time elapsed: 97.149 sec ERROR!
java.io.IOException: Job didn't finish in 30 seconds
at
org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
at
org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:53)
testMapreduceJobTimelineServiceEnabled(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
Time elapsed: 83.884 sec ERROR!
java.io.IOException: Job didn't finish in 30 seconds
at
org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
at
org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled(TestMRTimelineEventHandling.java:107)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6202) TestMRTimelineEventHandling fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen resolved MAPREDUCE-6202.
Resolution: Duplicate
Close it as the duplicate of MAPREDUCE-6189
TestMRTimelineEventHandling fails on trunk
--
Key: MAPREDUCE-6202
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6202
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Robert Kanter
Currently, {{TestMRTimelineEventHandling}} is failing on trunk:
{noformat}
Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 181.186 sec
FAILURE! - in org.apache.hadoop.mapred.TestMRTimelineEventHandling
testMRTimelineEventHandling(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
Time elapsed: 97.149 sec ERROR!
java.io.IOException: Job didn't finish in 30 seconds
at
org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
at
org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:53)
testMapreduceJobTimelineServiceEnabled(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
Time elapsed: 83.884 sec ERROR!
java.io.IOException: Job didn't finish in 30 seconds
at
org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
at
org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled(TestMRTimelineEventHandling.java:107)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6167) Prior 2.4 MR has compatibility issue because o.a.h.http.HttpConfig.setPolicy is removed
Zhijie Shen created MAPREDUCE-6167: -- Summary: Prior 2.4 MR has compatibility issue because o.a.h.http.HttpConfig.setPolicy is removed Key: MAPREDUCE-6167 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6167 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen a. In the following scenarios: 1. Either insecure or secure; 2. MR 2.2 with either old or new shuffle handler on NM; 3. Submitting via new client. We will see the following console exception: 14/11/17 23:47:45 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/zjshen/.staging/zjshen/.staging/job_1416270549965_0014 java.lang.NoSuchMethodError: org.apache.hadoop.http.HttpConfig.getSchemePrefix()Ljava/lang/String; at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:428) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:302) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) b. In the following scenarios: 1. Either insecure or secure; 2. MR 2.2 with old shuffle on NM; 3. Submitting via old client. We will see the following exception in the AM Log: 2014-11-17 15:09:06,157 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1416264695865_0007_01 2014-11-17 15:09:06,436 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.NoSuchMethodError: org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364) 2014-11-17 15:09:06,439 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler. The two exceptions are actually the same problem, but using the old client prevents it happening during app submission. o.a.h.http.HttpConfig.setPolicy is removed by YARN-1553 in 2.4. It could be a hadoop-common issue, but keep it in MR now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6148) Consolidate copying regular files/jars, log4j file and tar ball files
Zhijie Shen created MAPREDUCE-6148: -- Summary: Consolidate copying regular files/jars, log4j file and tar ball files Key: MAPREDUCE-6148 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6148 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Zhijie Shen Assignee: Junping Du There're code duplication among copying regular files/jars, log4j file and tar ball files. We need to consolidate them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6142) Test failure in TestJobHistoryEventHandler and TestMRTimelineEventHandling
Zhijie Shen created MAPREDUCE-6142:
--
Summary: Test failure in TestJobHistoryEventHandler and
TestMRTimelineEventHandling
Key: MAPREDUCE-6142
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6142
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Affects Versions: 2.6.0
Reporter: Zhijie Shen
Assignee: Zhijie Shen
{code}
Running org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 28.122 sec
FAILURE! - in org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
testTimelineEventHandling(org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler)
Time elapsed: 6.014 sec FAILURE!
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler.testTimelineEventHandling(TestJobHistoryEventHandler.java:467)
Results :
Failed tests:
TestJobHistoryEventHandler.testTimelineEventHandling:467 expected:1 but
was:0
Tests run: 12, Failures: 1, Errors: 0, Skipped: 0
{code}
{code}
Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 204.488 sec
FAILURE! - in org.apache.hadoop.mapred.TestMRTimelineEventHandling
testMRTimelineEventHandling(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
Time elapsed: 54.84 sec FAILURE!
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:58)
Results :
Failed tests:
TestMRTimelineEventHandling.testMRTimelineEventHandling:58 expected:1 but
was:0
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6018) Create a framework specific config to enable timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-6018. Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Committed the patch to trunk, branch-2 and branch-2.6. Thanks Robert for the patch, and Vinod for input about config name! Create a framework specific config to enable timeline server Key: MAPREDUCE-6018 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6018 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Jonathan Eagles Assignee: Robert Kanter Fix For: 2.6.0 Attachments: MAPREDUCE-6018.patch, MAPREDUCE-6018.patch, MAPREDUCE-6018.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6124) Make MR setup the timeline domain and put entities into it
Zhijie Shen created MAPREDUCE-6124: -- Summary: Make MR setup the timeline domain and put entities into it Key: MAPREDUCE-6124 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6124 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen After YARN-2102, we should make MR client optionally define an domain, and put its generated entities into this domain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6080) JHS checks YARN application ACLs to determine user's access to aggregated logs
Zhijie Shen created MAPREDUCE-6080: -- Summary: JHS checks YARN application ACLs to determine user's access to aggregated logs Key: MAPREDUCE-6080 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6080 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 2.5.0, 3.0.0 Reporter: Zhijie Shen While JHS uses JobACLsManager to check user's access tot the job history information, it uses ApplicationACLsManager to justify whether the user has access to the aggregated log, because it directly imports AggregatedLogsBlock into the log web page. In most cases, the two manager can do consistent access control. However we observed case that YARN acls is enabled while MR cluster acls is not. Therefore, the user can view all the job information except accessing the aggregated logs from JHS. It confuses the user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6044) Fully qualified intermediate done directory will break per-user dir creation on Windows
Zhijie Shen created MAPREDUCE-6044:
--
Summary: Fully qualified intermediate done directory will break
per-user dir creation on Windows
Key: MAPREDUCE-6044
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6044
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 3.0.0, 2.6.0
Environment: Windows
Reporter: Zhijie Shen
Assignee: Zhijie Shen
After MAPREDUCE-6032, the string of the intermediate done dir will be a fully
qualified path.
The following code in JobHistroyUtils tries to concat this path and user name
to create a per-user dir path, using File.separator as the seperator (on
Windows, it is \).
{code}
public static String getHistoryIntermediateDoneDirForUser(Configuration conf)
throws IOException {
return getConfiguredHistoryIntermediateDoneDirPrefix(conf) + File.separator
+ UserGroupInformation.getCurrentUser().getShortUserName();
}
{code}
Therefore, an intermediate done dir for user will become
hdfs://localhost:9201/mapred/history/done_intermediate\user. With the scheme
available in the path, Path class will not replace \ with /, and finally FS
cannot handle this path correctly: it will take done_intermediateuser as a
single directory name.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5940) Avoid negative elapsed time in JHS/MRAM web UI and services
Zhijie Shen created MAPREDUCE-5940: -- Summary: Avoid negative elapsed time in JHS/MRAM web UI and services Key: MAPREDUCE-5940 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5940 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mr-am, webapps Reporter: Zhijie Shen Assignee: Zhijie Shen Recently we observed a rare bug that an elapsed time of a reducer is going to be negative on JHS web UI and via REST APIs. While the real reason for this bug seems to be clock asynchronization on different hosts, the web frontend should have masked the negative values. However, in the current code, *org.apache.hadoop.mapreduce.v2.app.webapp.dao.** only check whether the elapsed time is -1 or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
Zhijie Shen created MAPREDUCE-5933: -- Summary: Enable MR AM to post history events to the timeline server Key: MAPREDUCE-5933 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Reporter: Zhijie Shen Assignee: Zhijie Shen Nowadays, MR AM collects the history events and writes it to HDFS for JHS to source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5934) Make JHS source the timeline server for job history information
Zhijie Shen created MAPREDUCE-5934: -- Summary: Make JHS source the timeline server for job history information Key: MAPREDUCE-5934 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5934 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Reporter: Zhijie Shen Assignee: Zhijie Shen After MAPREDUCE-5933, JHS can source the timeline server to get the job history information. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5924) Windows: Sort Job failed due to 'Invalid event: TA_COMMIT_PENDING at COMMIT_PENDING'
Zhijie Shen created MAPREDUCE-5924:
--
Summary: Windows: Sort Job failed due to 'Invalid event:
TA_COMMIT_PENDING at COMMIT_PENDING'
Key: MAPREDUCE-5924
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5924
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Post the issue on behalf of [~yeshavora]:
The Sort job over 1GB data failed with below error
{code}
2014-06-09 09:15:38,746 INFO [Socket Reader #1 for port 63415]
SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
job_1402304714683_0002 (auth:SIMPLE)
2014-06-09 09:15:38,750 INFO [IPC Server handler 13 on 63415]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update
from attempt_1402304714683_0002_r_15_1000
2014-06-09 09:15:38,751 ERROR [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this
event at current state for attempt_1402304714683_0002_r_15_1000
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
TA_COMMIT_PENDING at COMMIT_PENDING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1058)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:145)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1271)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1263)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:722)
2014-06-09 09:15:38,753 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1402304714683_0002Job
Transitioned from RUNNING to ERROR
{code}
The JobHistory Url prints job state = ERROR
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5858) MR should make use of the timeline server
Zhijie Shen created MAPREDUCE-5858: -- Summary: MR should make use of the timeline server Key: MAPREDUCE-5858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5858 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Zhijie Shen Assignee: Zhijie Shen Now MR relies on its own JobHistoryServer for MR specific history information. Given the timeline server is ready, we should gradually migrate MR historic data to it as well. relieving MR from maintaining its own history server daemon. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
Zhijie Shen created MAPREDUCE-5831:
--
Summary: Old MR client is not compatible with new MR application
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.3.0, 2.2.0
Reporter: Zhijie Shen
Priority: Critical
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR 2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one nodes
of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since Hadoop
2.3.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5833) TestRMContainerAllocator fails ocassionally
Zhijie Shen created MAPREDUCE-5833:
--
Summary: TestRMContainerAllocator fails ocassionally
Key: MAPREDUCE-5833
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5833
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
testReportedAppProgress and testReportedAppProgressWithOnlyMaps have race
conditions.
{code}
Stacktrace
java.util.NoSuchElementException: null
at java.util.Collections$EmptyIterator.next(Collections.java:2998)
at
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgress(TestRMContainerAllocator.java:535)
{code}
{code}
Error Message
Task state is not correct (timedout) expected:RUNNING but was:SCHEDULED
Stacktrace
junit.framework.AssertionFailedError: Task state is not correct (timedout)
expected:RUNNING but was:SCHEDULED
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:393)
at
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgressWithOnlyMaps(TestRMContainerAllocator.java:700)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5768) TestUberAM fails on trunk
Zhijie Shen created MAPREDUCE-5768: -- Summary: TestUberAM fails on trunk Key: MAPREDUCE-5768 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5768 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Error Message Number of sylog* files expected same:4 was not:8 Stacktrace java.lang.AssertionError: Number of sylog* files expected same:4 was not:8 at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotSame(Assert.java:641) at org.junit.Assert.assertSame(Assert.java:580) at org.apache.hadoop.mapreduce.v2.TestMRJobs.testContainerRollingLog(TestMRJobs.java:523) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5681) TestJHSSecurity fails
Zhijie Shen created MAPREDUCE-5681:
--
Summary: TestJHSSecurity fails
Key: MAPREDUCE-5681
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5681
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
{code}
---
T E S T S
---
Running org.apache.hadoop.mapreduce.security.TestJHSSecurity
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.763 sec
FAILURE! - in org.apache.hadoop.mapreduce.security.TestJHSSecurity
testDelegationToken(org.apache.hadoop.mapreduce.security.TestJHSSecurity) Time
elapsed: 1.56 sec ERROR!
java.lang.NullPointerException: null
at java.util.Hashtable.get(Hashtable.java:334)
at java.util.Properties.getProperty(Properties.java:932)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:874)
at org.apache.hadoop.http.HttpServer.initSpnego(HttpServer.java:892)
at org.apache.hadoop.http.HttpServer.access$100(HttpServer.java:101)
at org.apache.hadoop.http.HttpServer$Builder.build(HttpServer.java:323)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:232)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.initializeWebApp(HistoryClientService.java:149)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.serviceStart(HistoryClientService.java:118)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceStart(JobHistoryServer.java:175)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.security.TestJHSSecurity.testDelegationToken(TestJHSSecurity.java:100)
Results :
Tests in error:
TestJHSSecurity.testDelegationToken:100 ? NullPointer
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
{code}
Did some preliminary investigation, in HistoryClientService:
{code}
.withHttpSpnegoPrincipalKey(
JHAdminConfig.MR_WEBAPP_SPNEGO_USER_NAME_KEY)
{code}
MR_WEBAPP_SPNEGO_USER_NAME_KEY seems not to be in the configuration.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Resolved] (MAPREDUCE-5681) TestJHSSecurity fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-5681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen resolved MAPREDUCE-5681.
Resolution: Duplicate
TestJHSSecurity fails on trunk
--
Key: MAPREDUCE-5681
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5681
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
{code}
---
T E S T S
---
Running org.apache.hadoop.mapreduce.security.TestJHSSecurity
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.763 sec
FAILURE! - in org.apache.hadoop.mapreduce.security.TestJHSSecurity
testDelegationToken(org.apache.hadoop.mapreduce.security.TestJHSSecurity)
Time elapsed: 1.56 sec ERROR!
java.lang.NullPointerException: null
at java.util.Hashtable.get(Hashtable.java:334)
at java.util.Properties.getProperty(Properties.java:932)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:874)
at org.apache.hadoop.http.HttpServer.initSpnego(HttpServer.java:892)
at org.apache.hadoop.http.HttpServer.access$100(HttpServer.java:101)
at org.apache.hadoop.http.HttpServer$Builder.build(HttpServer.java:323)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:232)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.initializeWebApp(HistoryClientService.java:149)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.serviceStart(HistoryClientService.java:118)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceStart(JobHistoryServer.java:175)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.security.TestJHSSecurity.testDelegationToken(TestJHSSecurity.java:100)
Results :
Tests in error:
TestJHSSecurity.testDelegationToken:100 ? NullPointer
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
{code}
Did some preliminary investigation, in HistoryClientService:
{code}
.withHttpSpnegoPrincipalKey(
JHAdminConfig.MR_WEBAPP_SPNEGO_USER_NAME_KEY)
{code}
MR_WEBAPP_SPNEGO_USER_NAME_KEY seems not to be in the configuration.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Created] (MAPREDUCE-5576) MR AM unregistration should be failed due to UnknownHostException on getting history url
Zhijie Shen created MAPREDUCE-5576: -- Summary: MR AM unregistration should be failed due to UnknownHostException on getting history url Key: MAPREDUCE-5576 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5576 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Before RMCommunicator sends the request to RM to finish the application, it will try to get the JHS url, which may throw UnknownHostException. The current code path will skip sending the request to RM when the exception is raised, which sounds not a reasonable behavior, because RM's unregistering an AM will not affected by the tracking URL. The URL can be empty or null. AFAIK, the impact of null URL will be that the URL to redirect users from RM web page to JHS will be unavailable, and the job report will not show the URL as well. However, is it much much better than failing an application because of UnknownHostException here? Anyway, users can go to JHS directly to find the application history info. Therefore, the reasonable code path here should be catching UnknownHostException and set historyUrl = null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5557) QueueACLs doesn't work as root allows *
Zhijie Shen created MAPREDUCE-5557: -- Summary: QueueACLs doesn't work as root allows * Key: MAPREDUCE-5557 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5557 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Even if we specify acl for default queue, say user1, user2 can still submit and kill applications on default queue, because the queue checked user2 don't have the access to it, it then checked whether user2 has the access to it's parent recursively, and finally it found user2 have the access to root. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5559) Reconsidering the policy of ignoring the blacklist after reaching the thread
Zhijie Shen created MAPREDUCE-5559: -- Summary: Reconsidering the policy of ignoring the blacklist after reaching the thread Key: MAPREDUCE-5559 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5559 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Zhijie Shen Assignee: Zhijie Shen Nowadays, when RM AM find the number of blacklisted nodes reaches one threshold, the blacklist will be totally ignored. The newly assigned containers on the blacklisted nodes will be allocated. This may be not the best practice. We need to reconsider of it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5562) MR AM should exit when unregister() throws exception
Zhijie Shen created MAPREDUCE-5562: -- Summary: MR AM should exit when unregister() throws exception Key: MAPREDUCE-5562 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5562 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
Zhijie Shen created MAPREDUCE-5551: -- Summary: Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5547) Job history should not be flushed to JHS until AM gets unregistered
Zhijie Shen created MAPREDUCE-5547: -- Summary: Job history should not be flushed to JHS until AM gets unregistered Key: MAPREDUCE-5547 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5547 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (MAPREDUCE-5535) TestClusterMRNotification.testMR is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen resolved MAPREDUCE-5535.
Resolution: Duplicate
Will fix together in MAPREDUCE-5538.
TestClusterMRNotification.testMR is failing
---
Key: MAPREDUCE-5535
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5535
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
{code}
testMR(org.apache.hadoop.mapred.TestClusterMRNotification) Time elapsed:
35.222 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:0
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at
org.apache.hadoop.mapred.NotificationTestCase.testMR(NotificationTestCase.java:163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5538) MRAppMaster#shutDownJob shouldn't send job end notification before checking isLastRetry
Zhijie Shen created MAPREDUCE-5538: -- Summary: MRAppMaster#shutDownJob shouldn't send job end notification before checking isLastRetry Key: MAPREDUCE-5538 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5538 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5514) TestRMContainerAllocator fails on trunk
Zhijie Shen created MAPREDUCE-5514: -- Summary: TestRMContainerAllocator fails on trunk Key: MAPREDUCE-5514 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5514 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5499) Fix synchronization issues of the setters/getters of *PBImpl which take in/return lists
Zhijie Shen created MAPREDUCE-5499: -- Summary: Fix synchronization issues of the setters/getters of *PBImpl which take in/return lists Key: MAPREDUCE-5499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5499 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Xuan Gong Similar to YARN-609. There're the following *PBImpls which need to be fixed: 1. GetDiagnosticsResponsePBImpl 2. GetTaskAttemptCompletionEventsResponsePBImpl 3. GetTaskReportsResposnePBImpl 4. CounterGroupPBImpl 5. JobReportPBImpl 6. TaskReportPBImpl -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5459) Update the doc of running MRv1 examples jar on YARN
Zhijie Shen created MAPREDUCE-5459:
--
Summary: Update the doc of running MRv1 examples jar on YARN
Key: MAPREDUCE-5459
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5459
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
In addition to adding two env vars: HADOOP_USER_CLASSPATH_FIRST and
HADOOP_CLASSPATH, we still need to add
{code}
property
namemapreduce.job.user.classpath.first/name
valuetrue/value
/property
{code}
in mapred-site.xml to make sure that the MRv1 examples jar runs correctly on
YARN. Some examples will use Java reflection to find the classes in the
examples jar dynamically when they are running. With this configuration, the
MRv1 examples jar will appear before the MRv2 examples jar in CLASSPATH of the
processes in YARN containers. Therefore, the classes found via reflection will
be picked from MRv1 examples jar instead of MRv2 examples jar as well.
MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm needs to be updated to document
this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5108) Changes needed for Binary Compatibility for MR applications via YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-5108. Resolution: Fixed Close the ticket as all the subtasks are closed. Changes needed for Binary Compatibility for MR applications via YARN Key: MAPREDUCE-5108 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5108 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Arun C Murthy Assignee: Zhijie Shen Priority: Blocker Attachments: Binary Backward Compatibility.pdf, mr1_mr2_api_diff.tar.gz, MR_API_DIFF_v2.tar.gz As we get ready to ship out a beta/stable version of hadoop-2, it makes sense to spend time reviewing support for existing MR applications (hadoop-1) to migrate seamlessly. We've done various pieces of work over time, let's track progress and document things clearly. [~zjshen] has done a bunch of testing and results look very promising so far. The aim is to support applications using org.apache.hadoop.mapred.* api in a binary compatible manner in hadoop-2 - thus, users can just take existing MR applications jars, point them at YARN clusters and things just work. Clearly, we might have some corner cases (haven't seen many so far), including semantics (not just apis); however the intent is to, at least, document them throughly if not actually fix them as feasible. Also, it's clear that we will *not* be able to support org.apache.hadoop.mapreduce api in a *binary compatible* manner due to the interface changes we made in hadoop-0.21 (sigh), and hence, users using the _new_ apis will have to re-compile (i.e. source compatible only). Net, given that vast majority of users use the org.apache.hadoop.mapred api, it's a very reasonable way to ease migration to hadoop-2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5327) TestMRJobs and TestUberAM fail at verifying counters
Zhijie Shen created MAPREDUCE-5327:
--
Summary: TestMRJobs and TestUberAM fail at verifying counters
Key: MAPREDUCE-5327
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5327
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Priority: Critical
See the test report in YARN-829 and YARN-830:
* https://builds.apache.org/job/PreCommit-YARN-Build/1269//testReport/
* https://builds.apache.org/job/PreCommit-YARN-Build/1270//testReport/
The failure seems to be related to:
{code}
Assert
.assertTrue(counters.findCounter(JobCounter.SLOTS_MILLIS_MAPS) != null
counters.findCounter(JobCounter.SLOTS_MILLIS_MAPS).getValue() !=
0);
{code}
in TestMRJobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen resolved MAPREDUCE-4942.
Resolution: Fixed
Close the ticket. Actually, these methods are binary compatible with both MRv1
and 0.23. However, due to the conflicting method signature (throwing or not
throwing InterruptedException), these methods are source incompatible either
with MRv1 or 0.23.
Again, it is better be compatible with MRv1, as it has larger user base, while
0.23 users don't need much change to fix the code. Assume that a MRv1
application is not binary compatible because of mapreduce.Counter, whose binary
incompatibility will not be fixed. The user can recompile the code against MRv2
jars, and recompilation will not fail at mapreduce.Job given this ticket fixed.
Will document this imcompatible change in MAPREDUCE-5184 as well.
mapreduce.Job has a bunch of methods that throw InterruptedException so its
incompatible with MR1
-
Key: MAPREDUCE-4942
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.1.0-beta
Attachments: MAPREDUCE-4942.patch
The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
{{InterruptedException}} but don't in MR1, which makes them incompatible.
(Their Javadoc comments are also missing that they throw an
{{InterruptedException}} anyway)
I propose that we wrap the {{InterruptedException}} in a
{{RuntimeException}}.
{code}
public float setupProgress() throws IOException, InterruptedException
public float mapProgress() throws IOException, InterruptedException
public float reduceProgress() throws IOException, InterruptedException
public boolean isComplete() throws IOException, InterruptedException
public boolean isSuccessful() throws IOException, InterruptedException
public void killJob() throws IOException, InterruptedException
public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public Counters getCounters() throws IOException, InterruptedException
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-5275. Resolution: Fixed CacheToken is the class that provides user facing APIs. Whether the methods as annotated private or not, users are likely to refer them in their custom routines, because they are public. We fixed MAPREDUCE-5220 for the similar reason. IMHO, it is good to keep the rule of binary compatibility consistent. In addition, adding the methods back does no harm to MRv2 if they are actually not used. Feel free to reopen it, if you still have some concern. Thanks! Mapreduce API: TokenCache incompatibility issues with MR1 - Key: MAPREDUCE-5275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.1.0-beta Attachments: MAPREDUCE-5275-trunk-1.patch, MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, MAPREDUCE-5275-trunk-4.patch There're following incompatibility issues: * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is removed * Credentials loadTokens(String, Configuration) changes to Credentials loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5296) Mapred API: Function signature change in JobControl
Zhijie Shen created MAPREDUCE-5296: -- Summary: Mapred API: Function signature change in JobControl Key: MAPREDUCE-5296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5296 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen String addJob(Job) - String addJob(ControlledJob) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen reopened MAPREDUCE-5220: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5281) Mapreduce API: Counter changes from non-abstract class to interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-5281. Resolution: Won't Fix Mapreduce API: Counter changes from non-abstract class to interface --- Key: MAPREDUCE-5281 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5281 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Therefore, significant changes in Counter: 1. Two Constructors are removed; 2. Following methods are removed: * boolean equals(Object) * int hashCode() * void readFields(DataInput) * void write(DataOutput) Fix of this issue may break 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5284) Mapreduce API: CounterGroup changes from non-abstract class to interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-5284. Resolution: Won't Fix Mapreduce API: CounterGroup changes from non-abstract class to interface Key: MAPREDUCE-5284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5284 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Therefore, constructors and implemented methods are removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5299) Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
Zhijie Shen created MAPREDUCE-5299: -- Summary: Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent Key: MAPREDUCE-5299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5299 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Move the add-on patch of MAPREDUCE-5220 here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
Zhijie Shen created MAPREDUCE-5300: -- Summary: Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5280) Mapreduce API: ClusterMetrics incompatibility issues with MR1
Zhijie Shen created MAPREDUCE-5280: -- Summary: Mapreduce API: ClusterMetrics incompatibility issues with MR1 Key: MAPREDUCE-5280 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5280 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen 1. Constructor has one fewer parameters: numGraylistedTrackers 2. getGrayListedTaskTrackerCount() is removed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5281) Mapreduce API: Counter changes from non-abstract class to interface
Zhijie Shen created MAPREDUCE-5281: -- Summary: Mapreduce API: Counter changes from non-abstract class to interface Key: MAPREDUCE-5281 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5281 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Therefore, significant changes in Counter: 1. Two Constructors are removed; 2. Following methods are removed: * boolean equals(Object) * int hashCode() * void readFields(DataInput) * void write(DataOutput) Fix of this issue may break 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5284) Mapreduce API: CounterGroup changes from non-abstract class to interface
Zhijie Shen created MAPREDUCE-5284: -- Summary: Mapreduce API: CounterGroup changes from non-abstract class to interface Key: MAPREDUCE-5284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5284 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Therefore, constructors and implemented methods are removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5272) A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1
Zhijie Shen created MAPREDUCE-5272:
--
Summary: A Minor Error in Javadoc of TestMRWithDistributedCache in
Branch-1
Key: MAPREDUCE-5272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5272
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Trivial
{code}
/**
* Tests the use of the
* {@link org.apache.hadoop.mapreduce.filecache.DistributedCache} within the
* full MR flow as well as the LocalJobRunner. This ought to be part of the
* filecache package, but that package is not currently in mapred, so cannot
* depend on MR for testing.
*/
{code}
It should be org.apache.hadoop.filecache.DistributedCache instead. Branch-1
doesn't have org.apache.hadoop.mapreduce.filecache.DistributedCache
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5273) Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce
Zhijie Shen created MAPREDUCE-5273: -- Summary: Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce Key: MAPREDUCE-5273 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5273 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two protected variables are removed from CombineFileRecordReader in both mapred and mapreduce: * FileSystem fs * ClassRecordReaderK, V rrClass -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5274) String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce
Zhijie Shen created MAPREDUCE-5274: -- Summary: String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce Key: MAPREDUCE-5274 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5274 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce after upgrading to M/R 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
Zhijie Shen created MAPREDUCE-5275: -- Summary: Mapreduce API: TokenCache incompatibility issues with MR1 Key: MAPREDUCE-5275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen There're following incompatibility issues: * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is removed * Credentials loadTokens(String, Configuration) changes to Credentials loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
Zhijie Shen created MAPREDUCE-5263: -- Summary: filecache.DistributedCache incompatiblity issues with MR1 Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen A couple of methods and variables have been removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4251) API Incompatibility - Sampler
[ https://issues.apache.org/jira/browse/MAPREDUCE-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-4251. Resolution: Duplicate It duplicates MAPREDUCE-5157. Reopen it if it is needed for 22 also. API Incompatibility - Sampler - Key: MAPREDUCE-4251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4251 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv1 Affects Versions: 0.22.0 Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: MAPREDUCE-4251.patch org.apache.hadoop.mapred.lib.InputSampler#Sampler in Hadoop 0.20 has been moved to org.apache.hadoop.mapreduce.lib.partition.InputSampler#Sampler in Hadoop 0.22 The arguments of the getSample method in the Sampler class have also been changed, 0.22 use the new InputFormat, and 0.20 use the deprecated InputFormat, 0.22 use org.apache.hadoop.mapreduce.Job and 0.20use org.apache.hadoop.mapred.JobConf. So the programs compiled with old api has to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5243) MRAdmin is removed from M/R while RMAdmin is added to Yarn
Zhijie Shen created MAPREDUCE-5243: -- Summary: MRAdmin is removed from M/R while RMAdmin is added to Yarn Key: MAPREDUCE-5243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5243 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Though in the 2.x mapred script, MRAdmin will not be called, MRAdmin class is better to be there in case users call it programmatically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5244) Two functions change their visibility in JobStatus
Zhijie Shen created MAPREDUCE-5244: -- Summary: Two functions change their visibility in JobStatus Key: MAPREDUCE-5244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5244 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two functions change their visibility in JobStatus from public to protected: void setRunState(int) void setSchedulingInfo(String) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
Zhijie Shen created MAPREDUCE-5245: -- Summary: A number of public static variables are removed from JobConf Key: MAPREDUCE-5245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen A number of public static variables are removed from JobConf: boolean DEFAULT_MAPREDUCE_RECOVER_JOB String MAPREDUCE_RECOVER_JOB String WORKFLOW_ADJACENCY_PREFIX_PATTERN String WORKFLOW_ADJACENCY_PREFIX_STRING String WORKFLOW_ID String WORKFLOW_NAME String WORKFLOW_NODE_NAME String WORKFLOW_TAGS The workflow related variables are moved to MRJobConfig. The follwing public static variables becomes default: String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY The variables there are no longer referred internally in 2.x, but they might be used by users as they were public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5231) Constructor of DBInputFormat.DBRecordReader in mapred is changed
Zhijie Shen created MAPREDUCE-5231: -- Summary: Constructor of DBInputFormat.DBRecordReader in mapred is changed Key: MAPREDUCE-5231 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5231 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen The constructor of DBInputFormat.DBRecordReader in mapred is changed from MR1 to RM2. Though MAPREDUCE-716 tried to deal with the API difference. However, if DBInputFormat.DBRecordReader is extended, the incompatibility around the constructor is still there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
Zhijie Shen created MAPREDUCE-5233: -- Summary: Functions are changed or removed from Job in jobcontrol Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5234) Signature changes for getTaskId of TaskReport in mapred
Zhijie Shen created MAPREDUCE-5234: -- Summary: Signature changes for getTaskId of TaskReport in mapred Key: MAPREDUCE-5234 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5234 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen TaskReport in mapred of MR2 extends TaskReport in mapreduce, and inherits getTaskId, which return TaskID object. in MR1, this function returns String. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5235) MAX_GROUP_LIMIT is removed from Counters in mapred
Zhijie Shen created MAPREDUCE-5235: -- Summary: MAX_GROUP_LIMIT is removed from Counters in mapred Key: MAPREDUCE-5235 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5235 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen MAX_GROUP_LIMIT is removed from Counters in mapred in MR2. Though it seems not to be the variable that will be referred by the user code. It was actually configurable value MR1. We should investigate why the upper bound doesn't need to be checked in MR2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5237) Three functions are removed in ClusterStatus
Zhijie Shen created MAPREDUCE-5237: -- Summary: Three functions are removed in ClusterStatus Key: MAPREDUCE-5237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen The three functions are: CollectionString getGraylistedTrackerNames() int getGraylistedTrackers() State getJobTrackerState() The're tracker related, such that they are no longer used in MR2. Maybe we should add them and throw UnsupportedOperationException? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-5160. Resolution: Fixed Fix Version/s: 2.0.5-beta The issue is fixed with MAPREDUCE-5159. Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn --- Key: MAPREDUCE-5160 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5160 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.0.5-beta Aggregatewordcount and Aggregatewordhist of hadoop-1 cannot run on hadoop-2 due to org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String args[], Class? caller) is not available on hadoop-2 (see MAPREDUCE-5159). After I changed Aggregatewordcount and Aggregatewordhist to use createValueAggregatorJob(String args[], Class? extends ValueAggregatorDescriptor[] descriptors), which is available on hadoop-2, the two examples could be accepted and run on Yarn. However, the two examples still failed, because their inner classes, WordCountPlugInClass and AggregateWordHistogramPlugin, cannot be found in runtime, respectively. Both the plugin classes extend org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorBaseDescriptor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5225) SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits
Zhijie Shen created MAPREDUCE-5225: -- Summary: SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits Key: MAPREDUCE-5225 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5225 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Now, SplitSampler only samples the first maxSplitsSampled splits, caused by MAPREDUCE-1820. However, jumping around all splits is in general preferable than the first N splits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
Zhijie Shen created MAPREDUCE-5156: -- Summary: Hadoop-examples-1.x.x.jar cannot run on Yarn Key: MAPREDUCE-5156 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen M/R examples are run through ProgramDriver.driver. ProgramDriver.driver returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
Zhijie Shen created MAPREDUCE-5157: -- Summary: Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib Key: MAPREDUCE-5157 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in hadoop-1. However, after upgrading to hadoop-2, org.apache.hadoop.mapred.lib.InputSampler is modified to extend org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate
Zhijie Shen created MAPREDUCE-5159: -- Summary: Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate Key: MAPREDUCE-5159 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Both examples in hadoop-1 use org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String args[], Class? caller), which no longer exists after upgrading to hadoop-2. Therefore, they cannot not find the accordant function with their function signature in the binary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
Zhijie Shen created MAPREDUCE-5160: -- Summary: Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn Key: MAPREDUCE-5160 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5160 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Aggregatewordcount and Aggregatewordhist of hadoop-1 cannot run on hadoop-2 due to org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String args[], Class? caller) is not available on hadoop-2 (see MAPREDUCE-5159). After I changed Aggregatewordcount and Aggregatewordhist to use createValueAggregatorJob(String args[], Class? extends ValueAggregatorDescriptor[] descriptors), which is available on hadoop-2, the two examples could be accepted and run on Yarn. However, the two examples still failed, because their inner classes, WordCountPlugInClass and AggregateWordHistogramPlugin, cannot be found in runtime, respectively. Both the plugin classes extend org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorBaseDescriptor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5145) Change default max-attempts to be more than one for MR jobs as well
Zhijie Shen created MAPREDUCE-5145: -- Summary: Change default max-attempts to be more than one for MR jobs as well Key: MAPREDUCE-5145 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5145 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen We need to give the AM of MR jobs the chance to retry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5140) The mapping between yarn and mapreduce states needs to be updated when application store is delayed
Zhijie Shen created MAPREDUCE-5140: -- Summary: The mapping between yarn and mapreduce states needs to be updated when application store is delayed Key: MAPREDUCE-5140 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5140 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen In YARN-514, application store needs to be delayed to unblock application submission, such that a new state of MRApp needs to be created. On mapreduce side, there's some function to map yarn states to mapreduce ones. This mapping needs to be updated due to the newly added state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
