[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224617#comment-14224617
]
Kihwal Lee edited comment on HDFS-7443 at 11/25/14 2:44 PM:
------------------------------------------------------------
This is the first error seen.
{noformat}
ERROR datanode.DataNode: Initialization failed for Block pool <registering>
(Datanode Uuid unassigned)
service to some.host:8020 EEXIST: File exists
{noformat}
This was after successful upgrade of several volumes. Since the hard link
summary was not printed and it was multiple seconds after starting upgrade of
this volume (did not fail right away), the error must have come from
{{DataStorage.linkBlocks()}} when it was checking the result with
{{Futures.get()}}.
Then it was retried and failed the same way.
{noformat}
INFO common.Storage: Analyzing storage directories for bpid BP-xxxx
INFO common.Storage: Recovering storage directory
/a/b/hadoop/var/hdfs/data/current/BP-xxxx
from previous upgrade
INFO common.Storage: Upgrading block pool storage directory
/a/b/hadoop/var/hdfs/data/current/BP-xxxx
old LV = -55; old CTime = 12345678.
new LV = -56; new CTime = 45678989
ERROR datanode.DataNode: Initialization failed for Block pool <registering>
(Datanode Uuid unassigned)
service to some.host:8020 EEXIST: File exists
{noformat}
This indicates {{Storage.analyzeStorage()}} correctly returning
{{RECOVER_UPGRADE}} and the partial upgrade is undone before retrying. This
repeated hundreds of times before termination of datanode, which logged the
stack trace.
{noformat}
FATAL datanode.DataNode: Initialization failed for Block pool <registering>
(Datanode Uuid unassigned)
service to some.host:8020. Exiting.
java.io.IOException: EEXIST: File exists
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at
com.google.common.util.concurrent.Futures.newFromConstructor(Futures.java:1258)
at
com.google.common.util.concurrent.Futures.newWithCause(Futures.java:1218)
at
com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1131)
at com.google.common.util.concurrent.Futures.get(Futures.java:1048)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(DataStorage.java:999)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.linkAllBlocks(BlockPoolSliceStorage.java:594)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doUpgrade(BlockPoolSliceStorage.java:403)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doTransition(BlockPoolSliceStorage.java:337)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:197)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:438)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1312)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1277)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:314)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:221)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
at java.lang.Thread.run(Thread.java:722)
Caused by: EEXIST: File exists
at org.apache.hadoop.io.nativeio.NativeIO.link0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO.link(NativeIO.java:836)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage$2.call(DataStorage.java:991)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage$2.call(DataStorage.java:984)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
{noformat}
At this point, {{previous.tmp}} contained the new directory structure with
blocks and meta files placed in the ID-based directory. Some orphaned meta and
block files were observed. Restarting datanode does not reproduce the issue,
but I suspect data loss based on the missing files and the number of missing
blocks.
was (Author: kihwal):
This is the first error seen.
{noformat}
ERROR datanode.DataNode: Initialization failed for Block pool <registering>
(Datanode Uuid unassigned) service to some.host:8020 EEXIST: File exists
{noformat}
This was after successful upgrade of several volumes. Since the hard link
summary was not printed and it was multiple seconds after starting upgrade of
this volume (did not fail right away), the error must have come from
{{DataStorage.linkBlocks()}} when it was checking the result with
{{Futures.get()}}.
Then it was retried and failed the same way.
{noformat}
INFO common.Storage: Analyzing storage directories for bpid BP-xxxx
INFO common.Storage: Recovering storage directory
/a/b/hadoop/var/hdfs/data/current/BP-xxxx from previous upgrade
INFO common.Storage: Upgrading block pool storage directory
/a/b/hadoop/var/hdfs/data/current/BP-xxxx
old LV = -55; old CTime = 12345678.
new LV = -56; new CTime = 45678989
ERROR datanode.DataNode: Initialization failed for Block pool <registering>
(Datanode Uuid unassigned) service to some.host:8020 EEXIST: File exists
{noformat}
This indicates {{Storage.analyzeStorage()}} correctly returning
{{RECOVER_UPGRADE}} and the partial upgrade is undone before retrying. This
repeated hundreds of times before termination of datanode, which logged the
stack trace.
{noformat}
FATAL datanode.DataNode: Initialization failed for Block pool <registering>
(Datanode Uuid unassigned) service to some.host:8020. Exiting.
java.io.IOException: EEXIST: File exists
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at
com.google.common.util.concurrent.Futures.newFromConstructor(Futures.java:1258)
at
com.google.common.util.concurrent.Futures.newWithCause(Futures.java:1218)
at
com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1131)
at com.google.common.util.concurrent.Futures.get(Futures.java:1048)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(DataStorage.java:999)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.linkAllBlocks(BlockPoolSliceStorage.java:594)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doUpgrade(BlockPoolSliceStorage.java:403)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doTransition(BlockPoolSliceStorage.java:337)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:197)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:438)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1312)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1277)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:314)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:221)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
at java.lang.Thread.run(Thread.java:722)
Caused by: EEXIST: File exists
at org.apache.hadoop.io.nativeio.NativeIO.link0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO.link(NativeIO.java:836)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage$2.call(DataStorage.java:991)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage$2.call(DataStorage.java:984)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
{noformat}
At this point, {{previous.tmp}} contained the new directory structure with
blocks and meta files placed in the ID-based directory. Some orphaned meta and
block files were observed. Restarting datanode does not reproduce the issue,
but I suspect data loss based on the missing files and the number of missing
blocks.
> Datanode upgrade to BLOCKID_BASED_LAYOUT sometimes fails
> --------------------------------------------------------
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Kihwal Lee
> Priority: Blocker
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
