[jira] [Updated] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[
https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-6316:
-
Status: Open (was: Patch Available)
A problem occurs when you access
http://rmhost:rmport/proxy/application_1434432414806_0003/mapreduce/attempts/job_1434432414806_0003/m/RUNNING
{code}
2015-06-15 22:46:56,369 ERROR [378354947@qtp-596129348-3]
org.apache.hadoop.mapreduce.v2.app.webapp.AppController: Failed to render
attempts page with task type : m for job id : job_1434432414806_0003
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.webapp.TaskPage$AttemptsBlock.render(TaskPage.java:74)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
at
org.apache.hadoop.mapreduce.v2.app.webapp.AppController.attempts(AppController.java:296)
{code}
You should test all links at least manually. Ideally please try to add some
unit tests.
> Task Attempt List entries should link to the task overview
> --
>
> Key: MAPREDUCE-6316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Siqi Li
>Assignee: Siqi Li
> Labels: BB2015-05-TBR
> Attachments: AM attempt page.png, AM task page.png, All Attempts
> page.png, MAPREDUCE-6316.v1.patch, MAPREDUCE-6316.v2.patch,
> MAPREDUCE-6316.v3.patch, MAPREDUCE-6316.v4.patch, MAPREDUCE-6316.v5.patch,
> MAPREDUCE-6316.v6.patch, Task Overview page.png
>
>
> Typical workflow is to click on the list of failed attempts. Then you want to
> look at the counters, or the list of attempts of just one task in general. If
> each entry task attempt id linked the task id portion of it back to the task,
> we would not have to go through the list of tasks to search for the task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6363: Labels: (was: BB2015-05-TBR) > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587289#comment-14587289 ] Brahma Reddy Battula commented on MAPREDUCE-6363: - Thanks a lot [~ajisakaa] for review and commiting!!! And thanks to others for looking into this issue.. > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6400) TestReduceFetchFromPartialMem fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula reassigned MAPREDUCE-6400:
---
Assignee: Brahma Reddy Battula
> TestReduceFetchFromPartialMem fails
> ---
>
> Key: MAPREDUCE-6400
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6400
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
>Reporter: Akira AJISAKA
>Assignee: Brahma Reddy Battula
>
> TestReduceFetchFromPartialMem fails.
> {noformat}
> Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.978 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
> testReduceFromPartialMem(org.apache.hadoop.mapred.TestReduceFetchFromPartialMem)
> Time elapsed: 69.214 sec <<< ERROR!
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:867)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.runJob(TestReduceFetchFromPartialMem.java:300)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:93)
> Results :
> Tests in error:
> TestReduceFetchFromPartialMem.testReduceFromPartialMem:93->runJob:300 » IO
> Job...
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6400) TestReduceFetchFromPartialMem fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated MAPREDUCE-6400:
--
Component/s: test
> TestReduceFetchFromPartialMem fails
> ---
>
> Key: MAPREDUCE-6400
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6400
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
>Reporter: Akira AJISAKA
>
> TestReduceFetchFromPartialMem fails.
> {noformat}
> Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.978 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
> testReduceFromPartialMem(org.apache.hadoop.mapred.TestReduceFetchFromPartialMem)
> Time elapsed: 69.214 sec <<< ERROR!
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:867)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.runJob(TestReduceFetchFromPartialMem.java:300)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:93)
> Results :
> Tests in error:
> TestReduceFetchFromPartialMem.testReduceFromPartialMem:93->runJob:300 » IO
> Job...
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6400) TestReduceFetchFromPartialMem fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587218#comment-14587218
]
Akira AJISAKA commented on MAPREDUCE-6400:
--
I used {{git bisect}} and found this issue was broken by MAPREDUCE-6174.
> TestReduceFetchFromPartialMem fails
> ---
>
> Key: MAPREDUCE-6400
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6400
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Akira AJISAKA
>
> TestReduceFetchFromPartialMem fails.
> {noformat}
> Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.978 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
> testReduceFromPartialMem(org.apache.hadoop.mapred.TestReduceFetchFromPartialMem)
> Time elapsed: 69.214 sec <<< ERROR!
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:867)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.runJob(TestReduceFetchFromPartialMem.java:300)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:93)
> Results :
> Tests in error:
> TestReduceFetchFromPartialMem.testReduceFromPartialMem:93->runJob:300 » IO
> Job...
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6400) TestReduceFetchFromPartialMem fails
Akira AJISAKA created MAPREDUCE-6400:
Summary: TestReduceFetchFromPartialMem fails
Key: MAPREDUCE-6400
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6400
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.8.0
Reporter: Akira AJISAKA
TestReduceFetchFromPartialMem fails.
{noformat}
Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.978 sec <<<
FAILURE! - in org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
testReduceFromPartialMem(org.apache.hadoop.mapred.TestReduceFetchFromPartialMem)
Time elapsed: 69.214 sec <<< ERROR!
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:867)
at
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.runJob(TestReduceFetchFromPartialMem.java:300)
at
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:93)
Results :
Tests in error:
TestReduceFetchFromPartialMem.testReduceFromPartialMem:93->runJob:300 » IO
Job...
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6396) TestPipeApplication fails by NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587125#comment-14587125
]
Hudson commented on MAPREDUCE-6396:
---
FAILURE: Integrated in Hadoop-trunk-Commit #8019 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8019/])
MAPREDUCE-6396. TestPipeApplication fails by NullPointerException. Contributed
by Brahma Reddy Battula. (aajisaka: rev
85cc644f92e0e55293993c43e381f161bca5ce32)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/TestPipeApplication.java
* hadoop-mapreduce-project/CHANGES.txt
> TestPipeApplication fails by NullPointerException
> -
>
> Key: MAPREDUCE-6396
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6396
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6396.patch
>
>
> TestPipeApplication#testApplication and TestPipeApplication#testRunner fail
> on trunk.
> {noformat}
> Running org.apache.hadoop.mapred.pipes.TestPipeApplication
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 5.758 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.pipes.TestPipeApplication
> testApplication(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time
> elapsed: 1.157 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.fs.permission.FsPermission.applyUMask(FsPermission.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:498)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:526)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:298)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:777)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testApplication(TestPipeApplication.java:180)
> testRunner(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time elapsed:
> 0.035 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.fs.permission.FsPermission.applyUMask(FsPermission.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:498)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:526)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:298)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:777)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testRunner(TestPipeApplication.java:98)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6396) TestPipeApplication fails by NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6396:
-
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed this to trunk. Thanks all who contributed to this issue.
> TestPipeApplication fails by NullPointerException
> -
>
> Key: MAPREDUCE-6396
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6396
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6396.patch
>
>
> TestPipeApplication#testApplication and TestPipeApplication#testRunner fail
> on trunk.
> {noformat}
> Running org.apache.hadoop.mapred.pipes.TestPipeApplication
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 5.758 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.pipes.TestPipeApplication
> testApplication(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time
> elapsed: 1.157 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.fs.permission.FsPermission.applyUMask(FsPermission.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:498)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:526)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:298)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:777)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testApplication(TestPipeApplication.java:180)
> testRunner(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time elapsed:
> 0.035 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.fs.permission.FsPermission.applyUMask(FsPermission.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:498)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:526)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:298)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:777)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testRunner(TestPipeApplication.java:98)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6396) TestPipeApplication fails by NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587037#comment-14587037
]
Akira AJISAKA commented on MAPREDUCE-6396:
--
Thanks [~brahmareddy] for creating the patch and thanks [~zxu] for the comment.
+1, committing this.
> TestPipeApplication fails by NullPointerException
> -
>
> Key: MAPREDUCE-6396
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6396
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE-6396.patch
>
>
> TestPipeApplication#testApplication and TestPipeApplication#testRunner fail
> on trunk.
> {noformat}
> Running org.apache.hadoop.mapred.pipes.TestPipeApplication
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 5.758 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.pipes.TestPipeApplication
> testApplication(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time
> elapsed: 1.157 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.fs.permission.FsPermission.applyUMask(FsPermission.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:498)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:526)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:298)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:777)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testApplication(TestPipeApplication.java:180)
> testRunner(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time elapsed:
> 0.035 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.fs.permission.FsPermission.applyUMask(FsPermission.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:498)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:526)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:298)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:777)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testRunner(TestPipeApplication.java:98)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587004#comment-14587004 ] Hudson commented on MAPREDUCE-6363: --- FAILURE: Integrated in Hadoop-trunk-Commit #8018 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8018/]) MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java * hadoop-mapreduce-project/CHANGES.txt > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6363: - Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks [~brahmareddy] for the contribution and [~haridsv] for the comment. > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6363: - Summary: [NNBench] Lease mismatch error when running with multiple mappers (was: [NNBench mark] Lease mismatch error when running with multiple mappers) > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6363) [NNBench mark] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586981#comment-14586981 ] Akira AJISAKA commented on MAPREDUCE-6363: -- Committing this. > [NNBench mark] Lease mismatch error when running with multiple mappers > -- > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6399) UI Improvement - Counters Statistics Page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin J. Price updated MAPREDUCE-6399: -- Summary: UI Improvement - Counters Statistics Page (was: UI Improvement - Counters Page) > UI Improvement - Counters Statistics Page > - > > Key: MAPREDUCE-6399 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6399 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Kevin J. Price >Priority: Minor > > Current, the 'counters' page on the web UI will show totals for each counter, > and clicking on a counter gets individual values from each task. > For the purposes of job performance tuning, it would be very helpful to be > able to quickly see statistic aggregations on the counter. Specifically, I'm > thinking a view where, for each counter, we get: > MAX, MIN, MEAN, MEDIAN, MODE, STDEV (or some subset of those) > The max and min values could include links to corresponding task(s) counters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6316: --- Status: Patch Available (was: Open) > Task Attempt List entries should link to the task overview > -- > > Key: MAPREDUCE-6316 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Siqi Li >Assignee: Siqi Li > Labels: BB2015-05-TBR > Attachments: AM attempt page.png, AM task page.png, All Attempts > page.png, MAPREDUCE-6316.v1.patch, MAPREDUCE-6316.v2.patch, > MAPREDUCE-6316.v3.patch, MAPREDUCE-6316.v4.patch, MAPREDUCE-6316.v5.patch, > MAPREDUCE-6316.v6.patch, Task Overview page.png > > > Typical workflow is to click on the list of failed attempts. Then you want to > look at the counters, or the list of attempts of just one task in general. If > each entry task attempt id linked the task id portion of it back to the task, > we would not have to go through the list of tasks to search for the task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6399) UI Improvement - Counters Page
Kevin J. Price created MAPREDUCE-6399: - Summary: UI Improvement - Counters Page Key: MAPREDUCE-6399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6399 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Kevin J. Price Priority: Minor Current, the 'counters' page on the web UI will show totals for each counter, and clicking on a counter gets individual values from each task. For the purposes of job performance tuning, it would be very helpful to be able to quickly see statistic aggregations on the counter. Specifically, I'm thinking a view where, for each counter, we get: MAX, MIN, MEAN, MEDIAN, MODE, STDEV (or some subset of those) The max and min values could include links to corresponding task(s) counters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3384) Add warning message for underflow or overflow
[
https://issues.apache.org/jira/browse/MAPREDUCE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585835#comment-14585835
]
Brahma Reddy Battula commented on MAPREDUCE-3384:
-
[~devaraj.k] thanks for review..
{quote}Can we also have key as part of the warning message?{quote}
Ok.we can have.
{quote}
I am thinking, if we have more number of values where overflow/underflow can
happen, we would be filling with these logs. Can we limit having a some flag in
the reduce method to log only once for overflow and underflow for each key? any
thoughts?{quote}
I am thinking like following,, please let me know your opinion...
{code}
public void reduce(Key key, Iterable values,
Context context) throws IOException, InterruptedException {
int sum = 0;
boolean isOverflow=false;
boolean isUnderflow=false;
for (IntWritable val : values) {
int tempVal = val.get();
int oldSum = sum;
sum += tempVal;
if ((!isOverflow && oldSum > 0 && tempVal > 0) && (oldSum > sum)) {
LOG.warn("Arithmetic overflow occurs while reducing"+tempVal);
isOverflow=True;
}
if ((!isUnderflow && oldSum < 0 && tempVal < 0) && (oldSum < sum)) {
LOG.warn("Arithmetic underflow occurs while reducing"+tempVal);
isUnderflow=True;
}
}
result.set(sum);
context.write(key, result);
}
{code}
> Add warning message for underflow or overflow
> -
>
> Key: MAPREDUCE-3384
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3384
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: JiangKai
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE-3384-002.patch, MAPREDUCE-3384-003.patch,
> MAPREDUCE-3384.patch
>
>
> When we call the function reduce() of LongSumReducer,the result may overflow
> or underflow.
> We should have a warning message to users if overflow/underflow occurs for
> all these classes,
> {code:xml}
> /java/org/apache/hadoop/mapred/lib/LongSumReducer.java
> .../apache/hadoop/mapreduce/lib/reduce/IntSumReducer.java
> .../apache/hadoop/mapreduce/lib/reduce/LongSumReducer.java
> .../test/java/org/apache/hadoop/fs/AccumulatingReducer.java
> .../hadoop/mapred/nativetask/kvtest/HashSumReducer.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6380) AggregatedLogDeletionService will throw exception when there are some other directories in remote-log-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585711#comment-14585711 ] Mohammad Shahid Khan commented on MAPREDUCE-6380: - Correct the listStatus(Path f) will throw FileNotFoundException, instead of propagating the failing the same catch the FileNotFoundException and should log that the directory does not exits. > AggregatedLogDeletionService will throw exception when there are some other > directories in remote-log-dir > - > > Key: MAPREDUCE-6380 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6380 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Zhang Wei >Priority: Trivial > > AggregatedLogDeletionService will throw FileNotFoundException when there are > some extraneous directories put in remote-log-dir. The deletion function will > try to listStatus against the "extraneous-dir + suffix" dir. I think it > would be better if the function can ignore these directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6396) TestPipeApplication fails by NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585535#comment-14585535
]
zhihai xu commented on MAPREDUCE-6396:
--
thanks for working on this issue, [~brahmareddy].
the patch looks good to me, +1 (non-binding)
> TestPipeApplication fails by NullPointerException
> -
>
> Key: MAPREDUCE-6396
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6396
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE-6396.patch
>
>
> TestPipeApplication#testApplication and TestPipeApplication#testRunner fail
> on trunk.
> {noformat}
> Running org.apache.hadoop.mapred.pipes.TestPipeApplication
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 5.758 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.pipes.TestPipeApplication
> testApplication(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time
> elapsed: 1.157 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.fs.permission.FsPermission.applyUMask(FsPermission.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:498)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:526)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:298)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:777)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testApplication(TestPipeApplication.java:180)
> testRunner(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time elapsed:
> 0.035 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.fs.permission.FsPermission.applyUMask(FsPermission.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:498)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:526)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:298)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:777)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testRunner(TestPipeApplication.java:98)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
