[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154233#comment-14154233
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672181/MAPREDUCE-5933.patch
against trunk revision 14d60da.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1310 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.web.TestWebHDFS
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4925//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4925//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4925//artifact/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4925//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6116) Start container with auxiliary service data race condition
[
https://issues.apache.org/jira/browse/MAPREDUCE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or updated MAPREDUCE-6116:
-
Component/s: nodemanager
> Start container with auxiliary service data race condition
> --
>
> Key: MAPREDUCE-6116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6116
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.4.0
> Environment: HDP 2.1 on SLES 11
>Reporter: Andrew Or
>
> This shares the same symptoms as MAPREDUCE-2947, which is supposedly fixed.
> The stack trace I ran into is very similar:
> {code}
> Exception in thread "ContainerLauncher #1" java.lang.Error:
> org.apache.hadoop.yarn.exceptions.YarnException:
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException:
> java.lang.IllegalArgumentException
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:224)
> at
> org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:93)
> at
> org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:63)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ... 2 more
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:236)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:147)
> at java.nio.ByteBuffer.get(ByteBuffer.java:694)
> at com.google.protobuf.ByteString.copyFrom(ByteString.java:217)
> at com.google.protobuf.ByteString.copyFrom(ByteString.java:229)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:196)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.convertToProtoFormat(ContainerLaunchContextPBImpl.java:101)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$2$1.next(ContainerLaunchContextPBImpl.java:312)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$2$1.next(ContainerLaunchContextPBImpl.java:300)
> at
> com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
> at
> com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
> at
> org.apache.hadoop.yarn.proto.YarnProtos$ContainerLaunchContextProto$Builder.addAllServiceData(YarnProtos.java:32918)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.addServiceDataToProto(ContainerLaunchContextPBImpl.java:323)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToBuilder(ContainerLaunchContextPBImpl.java:112)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToProto(ContainerLaunchContextPBImpl.java:128)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.getProto(ContainerLaunchContextPBImpl.java:70)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.convertToProtoFormat(StartContainerRequestPBImpl.java:156)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.mergeLocalToBuilder(StartContainerRequestPBImpl.java:85)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.mergeLocalToProto(StartContainerRequestPBImpl.java:95)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.getProto(StartContainerRequestPBImpl.java:57)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.convertToProtoFormat(StartContainersRequestPBImpl.java:137)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.addLocalRequestsToProto(StartContainersRequestPBImpl.java:97)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.mergeLocalToBuilder(StartContainersRequestPBImpl.java:79)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.mergeLocalToProto(StartContainersRequestPBImpl.java:72)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.getProto(StartContainersRequestPBImpl.java:48)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.Containe
[jira] [Created] (MAPREDUCE-6116) Start container with auxiliary service data race condition
Andrew Or created MAPREDUCE-6116:
Summary: Start container with auxiliary service data race condition
Key: MAPREDUCE-6116
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6116
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Environment: HDP 2.1 on SLES 11
Reporter: Andrew Or
This shares the same symptoms as MAPREDUCE-2947, which is supposedly fixed. The
stack trace I ran into is very similar:
{code}
Exception in thread "ContainerLauncher #1" java.lang.Error:
org.apache.hadoop.yarn.exceptions.YarnException:
java.lang.IllegalArgumentException
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException:
java.lang.IllegalArgumentException
at
org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
at
org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:224)
at
org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:93)
at
org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:63)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
... 2 more
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:236)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:147)
at java.nio.ByteBuffer.get(ByteBuffer.java:694)
at com.google.protobuf.ByteString.copyFrom(ByteString.java:217)
at com.google.protobuf.ByteString.copyFrom(ByteString.java:229)
at
org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:196)
at
org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.convertToProtoFormat(ContainerLaunchContextPBImpl.java:101)
at
org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$2$1.next(ContainerLaunchContextPBImpl.java:312)
at
org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$2$1.next(ContainerLaunchContextPBImpl.java:300)
at
com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
at
com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
at
org.apache.hadoop.yarn.proto.YarnProtos$ContainerLaunchContextProto$Builder.addAllServiceData(YarnProtos.java:32918)
at
org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.addServiceDataToProto(ContainerLaunchContextPBImpl.java:323)
at
org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToBuilder(ContainerLaunchContextPBImpl.java:112)
at
org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToProto(ContainerLaunchContextPBImpl.java:128)
at
org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.getProto(ContainerLaunchContextPBImpl.java:70)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.convertToProtoFormat(StartContainerRequestPBImpl.java:156)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.mergeLocalToBuilder(StartContainerRequestPBImpl.java:85)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.mergeLocalToProto(StartContainerRequestPBImpl.java:95)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.getProto(StartContainerRequestPBImpl.java:57)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.convertToProtoFormat(StartContainersRequestPBImpl.java:137)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.addLocalRequestsToProto(StartContainersRequestPBImpl.java:97)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.mergeLocalToBuilder(StartContainersRequestPBImpl.java:79)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.mergeLocalToProto(StartContainersRequestPBImpl.java:72)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.getProto(StartContainersRequestPBImpl.java:48)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:93)
at
org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:201)
... 5 more
{code}
What I was doing in my application is calling
`ContainerLaunchContext#setServiceData` with
[jira] [Updated] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5933:
-
Attachment: MAPREDUCE-5933.patch
{quote}5. The countersToJSON method already turns the Counters into JSON so the
ATS can display them.
Yeah, my question is whether we really want to make counters' details
visualization-friendly via timeline read APIs, while the JHS will query this
data, reconstruct counter objects and show them there. If we don't, the easiest
way is to write counters into bytes when storing, and recover them from bytes
when retrieving in.{quote}
The problem with writing them as bytes is that the ATS doesn't know anything
about Counters. It would have to know how to read them back and then convert
them to JSON for display. The way I have it now, the ATS doesn't have to know
anything extra.
{quote}It seems that you published all the events related to job summary, which
may not be enough{quote}
In the new patch, I've added another method which processes the remaining event
types that were missing. I've also verified that they show up in the ATS
output for a job.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6072) Remove INSTALL document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153259#comment-14153259 ] Hudson commented on MAPREDUCE-6072: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1912 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1912/]) MAPREDUCE-6072. Remove INSTALL document (Akira AJISAKA via aw) (aw: rev 4666440b1aa6f394334cae43b95779948b7d6b07) * BUILDING.txt * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/INSTALL > Remove INSTALL document > --- > > Key: MAPREDUCE-6072 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6072 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.6.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: newbie > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6072.2.patch, MAPREDUCE-6072.3.patch, > MAPREDUCE-6072.patch > > > "./hadoop-mapreduce-project/INSTALL" has become stale. The document shows > * svn as SCM, now we use git > * ant to compile hadoop-mapreduce-examples > The document should be updated to link BUILDING.txt and web docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6094) TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153258#comment-14153258
]
Hudson commented on MAPREDUCE-6094:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1912 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1912/])
MAPREDUCE-6094. TestMRCJCFileInputFormat.testAddInputPath() fails on trunk.
Contributed by Akira AJISAKA (jlowe: rev
23fbfe36c31674907a26c8ecc6fd47888dfcc1ea)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRCJCFileInputFormat.java
* hadoop-mapreduce-project/CHANGES.txt
> TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
> --
>
> Key: MAPREDUCE-6094
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6094
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Sangjin Lee
>Assignee: Akira AJISAKA
>Priority: Minor
> Fix For: 2.6.0
>
> Attachments: MAPREDUCE-6094.patch
>
>
> {noformat}
> Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.624 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
> testAddInputPath(org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat)
> Time elapsed: 0.886 sec <<< ERROR!
> java.io.IOException: No FileSystem for scheme: s3
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2583)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2629)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at
> org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat.testAddInputPath(TestMRCJCFileInputFormat.java:55)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6094) TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153201#comment-14153201
]
Hudson commented on MAPREDUCE-6094:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1887 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1887/])
MAPREDUCE-6094. TestMRCJCFileInputFormat.testAddInputPath() fails on trunk.
Contributed by Akira AJISAKA (jlowe: rev
23fbfe36c31674907a26c8ecc6fd47888dfcc1ea)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRCJCFileInputFormat.java
> TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
> --
>
> Key: MAPREDUCE-6094
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6094
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Sangjin Lee
>Assignee: Akira AJISAKA
>Priority: Minor
> Fix For: 2.6.0
>
> Attachments: MAPREDUCE-6094.patch
>
>
> {noformat}
> Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.624 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
> testAddInputPath(org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat)
> Time elapsed: 0.886 sec <<< ERROR!
> java.io.IOException: No FileSystem for scheme: s3
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2583)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2629)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at
> org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat.testAddInputPath(TestMRCJCFileInputFormat.java:55)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6072) Remove INSTALL document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153202#comment-14153202 ] Hudson commented on MAPREDUCE-6072: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1887 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1887/]) MAPREDUCE-6072. Remove INSTALL document (Akira AJISAKA via aw) (aw: rev 4666440b1aa6f394334cae43b95779948b7d6b07) * hadoop-mapreduce-project/CHANGES.txt * BUILDING.txt * hadoop-mapreduce-project/INSTALL > Remove INSTALL document > --- > > Key: MAPREDUCE-6072 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6072 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.6.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: newbie > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6072.2.patch, MAPREDUCE-6072.3.patch, > MAPREDUCE-6072.patch > > > "./hadoop-mapreduce-project/INSTALL" has become stale. The document shows > * svn as SCM, now we use git > * ant to compile hadoop-mapreduce-examples > The document should be updated to link BUILDING.txt and web docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6094) TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153087#comment-14153087
]
Hudson commented on MAPREDUCE-6094:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #696 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/696/])
MAPREDUCE-6094. TestMRCJCFileInputFormat.testAddInputPath() fails on trunk.
Contributed by Akira AJISAKA (jlowe: rev
23fbfe36c31674907a26c8ecc6fd47888dfcc1ea)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRCJCFileInputFormat.java
> TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
> --
>
> Key: MAPREDUCE-6094
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6094
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Sangjin Lee
>Assignee: Akira AJISAKA
>Priority: Minor
> Fix For: 2.6.0
>
> Attachments: MAPREDUCE-6094.patch
>
>
> {noformat}
> Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.624 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
> testAddInputPath(org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat)
> Time elapsed: 0.886 sec <<< ERROR!
> java.io.IOException: No FileSystem for scheme: s3
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2583)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2629)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at
> org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat.testAddInputPath(TestMRCJCFileInputFormat.java:55)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6072) Remove INSTALL document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153088#comment-14153088 ] Hudson commented on MAPREDUCE-6072: --- FAILURE: Integrated in Hadoop-Yarn-trunk #696 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/696/]) MAPREDUCE-6072. Remove INSTALL document (Akira AJISAKA via aw) (aw: rev 4666440b1aa6f394334cae43b95779948b7d6b07) * hadoop-mapreduce-project/INSTALL * BUILDING.txt * hadoop-mapreduce-project/CHANGES.txt > Remove INSTALL document > --- > > Key: MAPREDUCE-6072 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6072 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.6.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: newbie > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6072.2.patch, MAPREDUCE-6072.3.patch, > MAPREDUCE-6072.patch > > > "./hadoop-mapreduce-project/INSTALL" has become stale. The document shows > * svn as SCM, now we use git > * ant to compile hadoop-mapreduce-examples > The document should be updated to link BUILDING.txt and web docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[
https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152925#comment-14152925
]
Hadoop QA commented on MAPREDUCE-6096:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671987/MAPREDUCE-6096-v3.patch
against trunk revision 0577eb3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4924//console
This message is automatically generated.
> SummarizedJob class NPEs with some jhist files
> --
>
> Key: MAPREDUCE-6096
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: trunk
>Reporter: zhangyubiao
> Labels: easyfix, patch
> Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096-v3.patch,
> MAPREDUCE-6096.patch,
> job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist
>
>
> When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's
> map-reduce-client-core project
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and
> HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like
> job_1408862281971_489761-1410883171851_XXX.jhist)
> and it throw an Exception Just Like
> Exception in thread "pool-1-thread-1" java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626)
> at
> com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70)
> After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is
> NULL ， So I change the order of
> attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to
> TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus())
> and it works well .
> So I wonder If we can change all attempt.getTaskStatus() after
> TaskStatus.State.XXX.toString() ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[
https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152916#comment-14152916
]
Hadoop QA commented on MAPREDUCE-6096:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671987/MAPREDUCE-6096-v3.patch
against trunk revision 0577eb3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4923//console
This message is automatically generated.
> SummarizedJob class NPEs with some jhist files
> --
>
> Key: MAPREDUCE-6096
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: trunk
>Reporter: zhangyubiao
> Labels: easyfix, patch
> Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096-v3.patch,
> MAPREDUCE-6096.patch,
> job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist
>
>
> When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's
> map-reduce-client-core project
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and
> HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like
> job_1408862281971_489761-1410883171851_XXX.jhist)
> and it throw an Exception Just Like
> Exception in thread "pool-1-thread-1" java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626)
> at
> com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70)
> After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is
> NULL ， So I change the order of
> attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to
> TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus())
> and it works well .
> So I wonder If we can change all attempt.getTaskStatus() after
> TaskStatus.State.XXX.toString() ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: MAPREDUCE-6096-v3.patch > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096-v3.patch, > MAPREDUCE-6096.patch, > job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096.patch, > job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152905#comment-14152905 ] zhangyubiao commented on MAPREDUCE-6096: Thanks for reviewed。I summit a jhist File to Test parse. and v3 code is update. > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: trunk >Reporter: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096.patch > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
