date:20160107

[jira] [Reopened] (MAPREDUCE-6566) Add retry support to mapreduce CLI tool

2016-01-07 Thread Akihiro Suda (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akihiro Suda reopened MAPREDUCE-6566:
-

The test is flaky, so I would like to open this issue again.
{code}
---
Test set: org.apache.hadoop.mapreduce.tools.TestCLI
---
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.699 sec <<< 
FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI)  Time elapsed: 0.025 sec  
<<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
{code}

{code}
$ cat 
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/surefire-reports/org.apache.hadoop.mapreduce.tools.TestCLI-output.txt
Killed job job_1234654654_001
The job job_1234654654_002 has already been killed.
Could not mark the job job_1234654654_003 as killed, as it has already failed.
Killed job job_1234654654_004
Error: Invalid task-type: JOB_SETUP
Usage: job [-list-attempt-ids   ]. Valid values 
for  are MAP REDUCE. Valid values for  are pending, 
running, completed, failed, killed
Error: Invalid task-type: JOB_CLEANUP
Usage: job [-list-attempt-ids   ]. Valid values 
for  are MAP REDUCE. Valid values for  are pending, 
running, completed, failed, killed
Error: Invalid task-state: complete
Usage: job [-list-attempt-ids   ]. Valid values 
for  are MAP REDUCE. Valid values for  are pending, 
running, completed, failed, killed
{code}

https://github.com/apache/hadoop/blob/d4e766de931c1cc478561a211215c517ba1f341c/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java#L175

> Add retry support to mapreduce CLI tool
> ---
>
> Key: MAPREDUCE-6566
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6566
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6566.001.patch, MAPREDUCE-6566.002.patch
>
>
> MAPREDUCE-6251 added support for retries to JobClient. However the MR CLI 
> class doesn't use the JobClient. It would be useful to add support for 
> retries to the CLI class as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hadoop-Mapreduce-trunk-Java8 - Build # 863 - Still Failing

2016-01-07 Thread Apache Jenkins Server

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/863/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 10101 lines...]
Running org.apache.hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.161 sec - 
in org.apache.hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.jobhistory.TestEvents
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.587 sec - in 
org.apache.hadoop.mapreduce.jobhistory.TestEvents
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.jobhistory.TestJobSummary
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.358 sec - in 
org.apache.hadoop.mapreduce.jobhistory.TestJobSummary
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.056 sec - 
in org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler

Results :

Failed tests: 
  
TestTaskAttempt.testMRAppHistoryForTAFailedInAssigned:177->testTaskAttemptAssignedKilledHistory:388
 No Ta Started JH Event

Tests run: 340, Failures: 1, Errors: 0, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  2.156 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:37 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 25.589 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  5.123 s]
[INFO] Apache Hadoop MapReduce App ... FAILURE [10:45 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SKIPPED
[INFO] Apache Hadoop MapReduce JobClient . SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 12:56 min
[INFO] Finished at: 2016-01-08T00:05:09+00:00
[INFO] Final Memory: 41M/268M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-app: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-app
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testMRAppHistoryForTAFailedInAssigned

Error Message:
No Ta Started JH Event

Stack Trace:
java.lang.AssertionError: No Ta Started JH Event
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testTaskAttemptAssignedKilledHistory(TestTaskAttempt.java:388)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testMRAppHistoryForTAFailedInAssigned(TestTaskAttempt.java:177)

Hadoop-Mapreduce-trunk - Build # 2798 - Still Failing

2016-01-07 Thread Apache Jenkins Server

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2798/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 32910 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.051 sec - in 
org.apache.hadoop.mapred.pipes.TestPipes
Running org.apache.hadoop.mapred.pipes.TestPipesNonJavaInputFormat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.475 sec - in 
org.apache.hadoop.mapred.pipes.TestPipesNonJavaInputFormat
Running org.apache.hadoop.mapred.pipes.TestPipeApplication
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.932 sec - in 
org.apache.hadoop.mapred.pipes.TestPipeApplication
Running org.apache.hadoop.mapred.TestJavaSerialization
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.581 sec - in 
org.apache.hadoop.mapred.TestJavaSerialization
Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.357 sec - in 
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
Running org.apache.hadoop.ipc.TestMRCJCSocketFactory
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.222 sec - in 
org.apache.hadoop.ipc.TestMRCJCSocketFactory

Results :

Failed tests: 
  TestNetworkedJob.testNetworkedJob:174 expected:<[[Thu Jan 07 08:50:59 + 
2016] Application is Activated, waiting for resources to be assigned for AM.  
Details : AM Partition =  ; Partition Resource = 
 ; Queue's Absolute capacity = 100.0 % ; Queue's 
Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> 
but was:<[]>

Tests run: 527, Failures: 1, Errors: 0, Skipped: 11

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  3.540 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:59 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 29.697 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  5.042 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [09:10 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SUCCESS [05:35 min]
[INFO] Apache Hadoop MapReduce JobClient . FAILURE [  01:41 h]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:58 h
[INFO] Finished at: 2016-01-07T09:30:02+00:00
[INFO] Final Memory: 35M/1024M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-jobclient
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob

Error Message:
expected:<[[Thu Jan 07 08:50:59 + 2016] Application is Activated, waiting 
for resources to be assigned for AM.  Details : AM Partition = 
 ; Partition Resource =  ; Queue's 
Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's 
Absolute max capacity = 100.0 % ; ]> but was:<[]>

Stack Trace:
org.junit.ComparisonFailure: expected:<[[Thu Jan 07 08:50:59 + 2016] 
Application is Activated,

Hadoop-Mapreduce-trunk - Build # 2799 - Still Failing

2016-01-07 Thread Apache Jenkins Server

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2799/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 32912 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.75 sec - in 
org.apache.hadoop.mapred.TestJobSysDirWithDFS
Running org.apache.hadoop.mapred.TestReduceTask
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.382 sec - in 
org.apache.hadoop.mapred.TestReduceTask
Running org.apache.hadoop.mapred.TestBadRecords
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.047 sec - in 
org.apache.hadoop.mapred.TestBadRecords
Running org.apache.hadoop.mapred.TestMiniMRDFSCaching
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.047 sec - in 
org.apache.hadoop.mapred.TestMiniMRDFSCaching
Running org.apache.hadoop.io.TestSequenceFileMergeProgress
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.366 sec - in 
org.apache.hadoop.io.TestSequenceFileMergeProgress
Running org.apache.hadoop.conf.TestNoDefaultsJobConf
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.986 sec - in 
org.apache.hadoop.conf.TestNoDefaultsJobConf

Results :

Failed tests: 
  TestNetworkedJob.testNetworkedJob:174 expected:<[[Thu Jan 07 11:30:54 + 
2016] Application is Activated, waiting for resources to be assigned for AM.  
Details : AM Partition =  ; Partition Resource = 
 ; Queue's Absolute capacity = 100.0 % ; Queue's 
Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> 
but was:<[]>

Tests run: 527, Failures: 1, Errors: 0, Skipped: 11

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  2.801 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:53 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 28.729 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  4.888 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [09:15 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SUCCESS [05:37 min]
[INFO] Apache Hadoop MapReduce JobClient . FAILURE [  01:43 h]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:00 h
[INFO] Finished at: 2016-01-07T11:59:32+00:00
[INFO] Final Memory: 34M/750M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-jobclient
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob

Error Message:
expected:<[[Thu Jan 07 11:30:54 + 2016] Application is Activated, waiting 
for resources to be assigned for AM.  Details : AM Partition = 
 ; Partition Resource =  ; Queue's 
Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's 
Absolute max capacity = 100.0 % ; ]> but was:<[]>

Stack Trace:
org.junit.ComparisonFailure: expected:<[[Thu Jan 07 11:30:54 + 2016] 
Application is Activated, waiting for resources to be assigned for AM.  Details 
: AM Partition =  ;

Hadoop-Mapreduce-trunk - Build # 2802 - Still Failing

2016-01-07 Thread Apache Jenkins Server

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2802/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 32950 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.712 sec - in 
org.apache.hadoop.mapred.TestClientRedirect
Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.63 sec - in 
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.585 sec - in 
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
Running org.apache.hadoop.mapred.TestSequenceFileAsBinaryOutputFormat
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.996 sec - in 
org.apache.hadoop.mapred.TestSequenceFileAsBinaryOutputFormat

Results :

Failed tests: 
  TestNetworkedJob.testNetworkedJob:174 expected:<[[Fri Jan 08 02:26:04 + 
2016] Scheduler has assigned a container for AM, waiting for AM container to be 
launched]> but was:<[]>

Tests in error: 
  TestMiniMRClientCluster.setup:80 » YarnRuntime 
org.apache.hadoop.yarn.exceptio...
  TestMiniMRClientCluster.cleanup:93 NullPointer

Tests run: 527, Failures: 1, Errors: 2, Skipped: 11

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  2.908 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:55 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 29.512 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  4.883 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [09:21 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SUCCESS [05:36 min]
[INFO] Apache Hadoop MapReduce JobClient . FAILURE [  01:40 h]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:57 h
[INFO] Finished at: 2016-01-08T02:33:34+00:00
[INFO] Final Memory: 35M/738M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-jobclient
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
3 tests failed.
FAILED:  
org.apache.hadoop.mapred.TestMiniMRClientCluster.org.apache.hadoop.mapred.TestMiniMRClientCluster

Error Message:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to intialize 
existing directories

Stack Trace:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to intialize 
existing directories
at 
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:460)
at 
org.apache.hadoop.fs.DelegateToFileSystem.listStatus(DelegateToFileSystem.java:168)
at org.apache.hadoop.fs.ChecksumFs.listStatus(ChecksumFs.java:521)
at 
org.apache.hadoop.fs.AbstractFileSystem$1.(AbstractFileSystem.java:890)
at 
org.apache.hadoop.fs.AbstractFileSystem.listStatusIterator(AbstractFileSystem.java:888)
at org.apache.hadoop.fs.FileContext$22.next(FileContext.java:1492)

[jira] [Resolved] (MAPREDUCE-6508) TestNetworkedJob fails consistently due to delegation token changes on RM.

2016-01-07 Thread Akira AJISAKA (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA resolved MAPREDUCE-6508.
--
Resolution: Fixed

> TestNetworkedJob fails consistently due to delegation token changes on RM.
> --
>
> Key: MAPREDUCE-6508
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6508
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6508.00.patch, MAPREDUCE-6508.01.patch
>
>
> {noformat}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 84.215 sec 
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time elapsed: 
> 31.537 sec  <<< ERROR!
> java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: 
> java.io.IOException: Delegation Token can be issued only with kerberos 
> authentication
>   at 
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getDelegationToken(ClientRMService.java:1044)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getDelegationToken(ApplicationClientProtocolPBServiceImpl.java:325)
>   at 
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2236)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2232)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2230)
> Caused by: java.io.IOException: Delegation Token can be issued only with 
> kerberos authentication
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getDelegationToken(ClientRMService.java:1017)
>   ... 10 more
>   at org.apache.hadoop.ipc.Client.call(Client.java:1448)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1379)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>   at com.sun.proxy.$Proxy84.getDelegationToken(Unknown Source)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getDelegationToken(ApplicationClientProtocolPBClientImpl.java:339)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:251)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy85.getDelegationToken(Unknown Source)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getRMDelegationToken(YarnClientImpl.java:541)
>   at 
> org.apache.hadoop.mapred.ResourceMgrDelegate.getDelegationToken(ResourceMgrDelegate.java:177)
>   at 
> org.apache.hadoop.mapred.YARNRunner.getDelegationToken(YARNRunner.java:231)
>   at 
> org.apache.hadoop.mapreduce.Cluster.getDelegationToken(Cluster.java:401)
>   at org.apache.hadoop.mapred.JobClient$16.run(JobClient.java:1234)
>   at org.apache.hadoop.mapred.JobClient$16.run(JobClient.java:1231)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
>   at 
> org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:1230)
>   at 
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:260)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hadoop-Mapreduce-trunk-Java8 - Build # 861 - Still Failing

2016-01-07 Thread Apache Jenkins Server

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/861/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 10893 lines...]
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.TestValueIterReset
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.579 sec - in 
org.apache.hadoop.mapreduce.TestValueIterReset
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 134.886 sec - 
in org.apache.hadoop.mapreduce.TestMapReduceLazyOutput

Results :

Failed tests: 
  TestNetworkedJob.testNetworkedJob:174 expected:<[[Thu Jan 07 15:58:11 + 
2016] Scheduler has assigned a container for AM, waiting for AM container to be 
launched]> but was:<[]>

Tests in error: 
  TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled:169 » IO 
Jo...
  TestLineRecordReaderJobs.testDefaultRecordDelimiters:127->createAndRunJob:83 
» NoClassDefFound
  TestLineRecordReaderJobs.testCustomRecordDelimiters:105->createAndRunJob:83 » 
NoClassDefFound

Tests run: 527, Failures: 1, Errors: 3, Skipped: 11

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  2.247 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:40 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 26.700 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  5.574 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [10:54 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SUCCESS [06:11 min]
[INFO] Apache Hadoop MapReduce JobClient . FAILURE [  01:40 h]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:59 h
[INFO] Finished at: 2016-01-07T17:24:58+00:00
[INFO] Final Memory: 34M/244M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-jobclient
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
4 tests failed.
FAILED:  
org.apache.hadoop.mapred.TestLineRecordReaderJobs.testDefaultRecordDelimiters

Error Message:
org/apache/hadoop/yarn/util/Apps

Stack Trace:
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

[jira] [Reopened] (MAPREDUCE-6508) TestNetworkedJob fails consistently due to delegation token changes on RM.

2016-01-07 Thread Akihiro Suda (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akihiro Suda reopened MAPREDUCE-6508:
-

I suggest reopening this issue.

The test fails on both of my local machine  
(@6702e7d66a4949a40bcdbd9f51ca1e746c8991e1, Jan 7 2016) and on Jenkins:
{code}
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 65.041 sec <<< 
FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time elapsed: 
30.412 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[AM container is launched, waiting for 
AM container to Register with RM]> but was:<[]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
{code}
https://www.mail-archive.com/mapreduce-dev@hadoop.apache.org/msg15252.html

https://github.com/apache/hadoop/blob/eb6379ca25e1bb6d3978bd3a021723c38c95bec9/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestNetworkedJob.java#L174

Also, the error message is wrong: HADOOP-12693


> TestNetworkedJob fails consistently due to delegation token changes on RM.
> --
>
> Key: MAPREDUCE-6508
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6508
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6508.00.patch, MAPREDUCE-6508.01.patch
>
>
> {noformat}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 84.215 sec 
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time elapsed: 
> 31.537 sec  <<< ERROR!
> java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: 
> java.io.IOException: Delegation Token can be issued only with kerberos 
> authentication
>   at 
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getDelegationToken(ClientRMService.java:1044)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getDelegationToken(ApplicationClientProtocolPBServiceImpl.java:325)
>   at 
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2236)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2232)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2230)
> Caused by: java.io.IOException: Delegation Token can be issued only with 
> kerberos authentication
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getDelegationToken(ClientRMService.java:1017)
>   ... 10 more
>   at org.apache.hadoop.ipc.Client.call(Client.java:1448)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1379)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>   at com.sun.proxy.$Proxy84.getDelegationToken(Unknown Source)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getDelegationToken(ApplicationClientProtocolPBClientImpl.java:339)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:251)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy85.getDelegationToken(Unknown Source)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getRMDelegationToken(YarnClientImpl.java:541)
>   at 
>

Re: [DISCUSSION] Release Plan for Apache Hadoop 2.6.4

2016-01-07 Thread Akira AJISAKA


+1 for releasing 2.6.4 after release 2.7.2.
I'll work on the list [2].

Thanks,
Akira

On 1/7/16 08:59, Junping Du wrote:

Hello folks,
Hope everyone had a wonderful holiday and good rest in passed weeks. We 
just had a 2.6.3 release on Dec.17 and now it is a good time to look forward to 
the next branch-2.6 maintenance release - 2.6.4 in the beginning of new year. 
As usual, this release will be based on latest release 2.6.3 but include a 
couple of critical/blocker bug fixes that identified in community recently.

Things Done/TODO:

   + Schedule
 -- According to previous email discussion thread in community, we are 
prefering fast-moving way (roughly as monthly) of releasing fixes on branch-2.6. So 
I would expect this release could happen around end of Jan. or early of Feb. 
dependens on if RC & VOTE process going on smoothly. Also, it should be after 
release 2.7.2 to get rid of any possible disturb to that release.

   + Branch
 -- 2.6.4 will be based on branch-2.6, which has been open to 2.6.4 commits 
for a while. I plan to branch out branch-2.6.4 sometime next week according to 
status of target/fixed fixes.
 -- Before 2.6.4 branch is cut off, new commits coming as fix to 2.6.4 is 
good to commit/merge to trunk, branch-2 and branch-2.6.

   + Patches
 -- In last couple of days, I went through all critial/blocker bug fixes in 
the coming 2.7.2 release and ping authors/committers on JIRA to ask for the 
same effort on branch-2.6.4. I could do more rounds of the same practice in 
case I could miss something helpful. Also, a kindly reminder from original 
author and committer is also appreciated.
 -- So far, 2.6.4 already has 18 fixes as list of [1] shows. However, there 
are totally 55 fixes target for 2.6.4 for now [2], so there is still a large 
gap and we need to work harder to resolve these tickets as many as we can and 
push out some non-critical ones to meet our promise of date.

   + Release
 -- Still too early to do anything directly related to RC. Will work on 
resolve fixes in list of [2] first, and appreciate any help on this effort.

Any thoughts? Especially on schedule and list of patches that should be 
included?


Cheers,

Junping

[1] Tickets fixed in 2.6.4:
http://s.apache.org/vJB

[2] Tickets targets for 2.6.4:
http://s.apache.org/JpE



From: Junping Du
Sent: Thursday, December 17, 2015 1:36 PM
To: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
junping...@apache.org
Subject: [RESULT] [VOTE] Release Apache Hadoop 2.6.3 RC0

Hi all in community,
  I give my binding +1 to close the vote for 2.6.3 RC0. With 18 +1s (10 
binding), one +0 (non-binding) and no -1s, the vote passes.

Thanks for everyone who tried the release candidate and voted. Thanks Vinod, 
Sangjin and Steve to provide me advices on release process.

I'll push the release bits and send out an announcement for 2.6.3 soon.

Cheers,

Junping

From: Xuan Gong 
Sent: Thursday, December 17, 2015 12:29 AM
To: common-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; junping...@apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

+1 (binding),

Build and deploy the cluster from source code.
Ran a few example jobs and passed successfully.

Xuan Gong


On Dec 16, 2015, at 4:07 PM, Arpit Agarwal  wrote:

+1 (binding)

- Verified signatures for source and binary distributions
- Built jars from source with java 1.7.0_79
- Deployed single-node pseudo-cluster
- Ran example map reduce jobs
- Ran hdfs admin commands, verified NN web UI shows expected usages



On 12/11/15, 4:16 PM, "Junping Du"  wrote:



Hi all developers in hadoop community,
  I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the next 
maintenance release to follow up 2.6.2.) according to email thread of release 
plan 2.6.3 [1]. Sorry for this RC coming a bit late as several blocker issues 
were getting committed until yesterday. Below is the details:

The RC is available for validation at:
*http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
*

The RC tag in git is: release-2.6.3-RC0

The maven artifacts are staged via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
*

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the usual 5 days.

Thanks and happy weekend!


Cheers,

Junping


[1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y

[jira] [Reopened] (MAPREDUCE-6566) Add retry support to mapreduce CLI tool

Hadoop-Mapreduce-trunk-Java8 - Build # 863 - Still Failing

Hadoop-Mapreduce-trunk - Build # 2798 - Still Failing

Hadoop-Mapreduce-trunk - Build # 2799 - Still Failing

Hadoop-Mapreduce-trunk - Build # 2802 - Still Failing

[jira] [Resolved] (MAPREDUCE-6508) TestNetworkedJob fails consistently due to delegation token changes on RM.

Hadoop-Mapreduce-trunk-Java8 - Build # 861 - Still Failing

[jira] [Reopened] (MAPREDUCE-6508) TestNetworkedJob fails consistently due to delegation token changes on RM.

Re: [DISCUSSION] Release Plan for Apache Hadoop 2.6.4

9 matches

Site Navigation

Mail list logo

Footer information