Jeff Zhang created TEZ-2260: ------------------------------- Summary: AM been shutdown due to NoSuchMethodError in DAGProtos Key: TEZ-2260 URL: https://issues.apache.org/jira/browse/TEZ-2260 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang
Not sure why this happens, maybe due to environment issue. {code} 2015-04-01 09:08:49,757 INFO [Dispatcher thread: Central] history.HistoryEventHandler: [HISTORY][DAG:dag_1427850436467_0007_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=datagen, taskAttemptId=attempt_1427850436467_0007_1_00_000000_0, startTime=1427850527981, finishTime=1427850529750, timeTaken=1769, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 8, File System Counters, HDFS_BYTES_READ=0, HDFS_BYTES_WRITTEN=953030, HDFS_READ_OPS=9, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=6, org.apache.tez.common.counters.TaskCounter, GC_TIME_MILLIS=46, COMMITTED_HEAP_BYTES=257425408, OUTPUT_RECORDS=44195 2015-04-01 09:08:49,757 FATAL [RecoveryEventHandlingThread] yarn.YarnUncaughtExceptionHandler: Thread Thread[RecoveryEventHandlingThread,5,main] threw an Error. Shutting down now... java.lang.NoSuchMethodError: org.apache.tez.dag.api.records.DAGProtos$TezCountersProto$Builder.access$26000()Lorg/apache/tez/dag/api/records/DAGProtos$TezCountersProto$Builder; at org.apache.tez.dag.api.records.DAGProtos$TezCountersProto.newBuilder(DAGProtos.java:24581) at org.apache.tez.dag.api.DagTypeConverters.convertTezCountersToProto(DagTypeConverters.java:544) at org.apache.tez.dag.history.events.TaskAttemptFinishedEvent.toProto(TaskAttemptFinishedEvent.java:97) at org.apache.tez.dag.history.events.TaskAttemptFinishedEvent.toProtoStream(TaskAttemptFinishedEvent.java:120) at org.apache.tez.dag.history.recovery.RecoveryService.handleRecoveryEvent(RecoveryService.java:403) at org.apache.tez.dag.history.recovery.RecoveryService.access$700(RecoveryService.java:50) at org.apache.tez.dag.history.recovery.RecoveryService$1.run(RecoveryService.java:158) at java.lang.Thread.run(Thread.java:745) 2015-04-01 09:08:49,757 INFO [Dispatcher thread: Central] impl.TaskAttemptImpl: attempt_1427850436467_0007_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE {code} This issue result in several consequent issues. Because this error cause the AM to recovery in the next attempt. But in the next attempt it meet the following issue, looks like data node crashed. {code} 2015-04-01 09:09:00,093 WARN [Thread-82] hdfs.DFSClient: DataStreamer Exception java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[127.0.0.1:56238, 127.0.0.1:56234], original=[127.0.0.1:56238, 127.0.0.1:56234]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1040) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1106) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1253) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594) 2015-04-01 09:09:00,093 WARN [Dispatcher thread: Central] hdfs.DFSClient: Error while syncing java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[127.0.0.1:56238, 127.0.0.1:56234], original=[127.0.0.1:56238, 127.0.0.1:56234]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1040) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1106) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1253) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594) 2015-04-01 09:09:00,094 ERROR [Dispatcher thread: Central] recovery.RecoveryService: Error handling summary event, eventType=VERTEX_FINISHED java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[127.0.0.1:56238, 127.0.0.1:56234], original=[127.0.0.1:56238, 127.0.0.1:56234]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1040) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1106) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1253) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594) {code} Because of the above issue (summary recovery log error), it cause the AM shutdown, and in the client side, it throw SessionNotRunning Exception without any diagnostic info. -- This message was sent by Atlassian JIRA (v6.3.4#6332)