[
https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613460#action_12613460
]
jimhuang edited comment on HADOOP-3758 at 7/14/08 3:01 PM:
------------------------------------------------------------
In the span of 11 seconds, there were 103,367 exceptions generated for only 10
unique nodes that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.
{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler
19 on 8020, call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256,
2150426691690, 0, 0) from A.B.C.D:43226: error:
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data
node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data
node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler
27 on 8020, call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888,
2151120783483, 0, 0) from A.B.C.E:56503: error:
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data
node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data
node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}
was (Author: jimhuang):
In the span of 11 seconds, there were 103,367 exceptions generated for only
10 unique nodes that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.
{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler
19 on 8020, call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256,
2150426691690, 0, 0) from 76.1
3.169.226:43226: error: org.apache.hadoop.dfs.IncorrectVersionException:
Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data
node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler
27 on 8020, call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888,
2151120783483, 0, 0) from 76.1
3.169.218:56503: error: org.apache.hadoop.dfs.IncorrectVersionException:
Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data
node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate
> the entire partition space, in this case close to 700GB log file was
> generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
