[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828469#comment-15828469 ] Arpit Agarwal commented on HDFS-9574: - [~kihwal], my apologies. That was a mistake, thanks for catching it. > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1 > > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, > HDFS-9574.v3.br26.patch, HDFS-9574.v3.br27.patch, HDFS-9574.v3.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828340#comment-15828340 ] Kihwal Lee commented on HDFS-9574: -- [~arpitagarwal], please elaborate the reason for marking it incompatible. If valid, we need to update the release note as well. > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1 > > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, > HDFS-9574.v3.br26.patch, HDFS-9574.v3.br27.patch, HDFS-9574.v3.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089645#comment-15089645 ] Kihwal Lee commented on HDFS-9574: -- This is an important improvement for rolling upgrades. Committed to branch-2.7 and branch-2.6. > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 3.0.0, 2.7.3, 2.6.4 > > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, > HDFS-9574.v3.br26.patch, HDFS-9574.v3.br27.patch, HDFS-9574.v3.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089560#comment-15089560 ] Hudson commented on HDFS-9574: -- FAILURE: Integrated in Hadoop-trunk-Commit #9073 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9073/]) HDFS-9574. Reduce client failures during datanode restart. Contributed (kihwal: rev 38c4c14472996562eb3d610649246770c2888c6b) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestDatanodeRestart.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataXceiverLazyPersistHint.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeFaultInjector.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, HDFS-9574.v3.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089549#comment-15089549 ] Kihwal Lee commented on HDFS-9574: -- Committed to trunk, branch-2 and branch-2.8. branch-2.7 and 2.6 need a separate patch since files have been moved and the hdfs client has been separated out. > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, HDFS-9574.v3.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089536#comment-15089536 ] Kihwal Lee commented on HDFS-9574: -- bq. DFSInputStream: Instead of tracking the retryList separately, would it be easier to just add it back to the nodeList and set the isRetry boolean? I thought about doing that. But with one list, it is hard to tell whether it has any good candidate left to try or it has only retriable nodes. The code ended up being more complicated than necessary so I settled with two separate lists. bq. DataXceiver: Might consider changing checkAccess to not require the stream and just have it call getBufferedOutputStream. It looks like that might be okay for the current usages. I didn't try to improve all inconsistencies in there. It might be better to be done in a separate clean-up jira. > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, HDFS-9574.v3.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089509#comment-15089509
]
Daryn Sharp commented on HDFS-9574:
---
+1 Looks good, few suggestions if you think they would add value, up to you.
{{DFSInputStream}}: Instead of tracking the {{retryList}} separately, would it
be easier to just add it back to the {{nodeList}} and set the {{isRetry}}
boolean?
{{DataXceiver}}: Might consider changing {{checkAccess}} to not require the
stream and just have it call {{getBufferedOutputStream}}.
> Reduce client failures during datanode restart
> --
>
> Key: HDFS-9574
> URL: https://issues.apache.org/jira/browse/HDFS-9574
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, HDFS-9574.v3.patch
>
>
> Since DataXceiverServer is initialized before BP is fully up, client requests
> will fail until the datanode registers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089277#comment-15089277
]
Kihwal Lee commented on HDFS-9574:
--
The failed test cases all pass when run locally.
{noformat}
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.947 sec -
in org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32 sec - in
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.namenode.TestFsck
Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 135.155 sec -
in org.apache.hadoop.hdfs.server.namenode.TestFsck
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.697 sec - in
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.252 sec
- in org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestSafeMode
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.149 sec - in
org.apache.hadoop.hdfs.TestSafeMode
Results :
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
{noformat}
The whitespace warning is from the context, not my change. Nothing to be done
for the checkstyle warnings.
> Reduce client failures during datanode restart
> --
>
> Key: HDFS-9574
> URL: https://issues.apache.org/jira/browse/HDFS-9574
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, HDFS-9574.v3.patch
>
>
> Since DataXceiverServer is initialized before BP is fully up, client requests
> will fail until the datanode registers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088673#comment-15088673
]
Hadoop QA commented on HDFS-9574:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-hdfs-project (total was 692, now 690). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 5s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 6s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 183m 57s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.T
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088438#comment-15088438
]
Kihwal Lee commented on HDFS-9574:
--
The new patch addresses the review comments.
- All relevant DataTransfer methods are now calling {{checkAccess()}} and the
registration is checked from there.
- The elapsed time is now tracked using {{StopWatch}}.
- {{getReplicaVisibleLength()}} now throws {{RetriableException}}.
- {{DFSInpuStream}} retries those nodes that threw {{RetriableException}} on
{{getReplicaVisibleLength()}}, with a limit. The client read timeout is used
for the retry timeout.
- The test case was expanded to cover the {{getReplicaVisibleLength()}} case.
> Reduce client failures during datanode restart
> --
>
> Key: HDFS-9574
> URL: https://issues.apache.org/jira/browse/HDFS-9574
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9574.patch, HDFS-9574.v2.patch, HDFS-9574.v3.patch
>
>
> Since DataXceiverServer is initialized before BP is fully up, client requests
> will fail until the datanode registers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083463#comment-15083463
]
Daryn Sharp commented on HDFS-9574:
---
Might consider checking if the bp is registered in {{checkAccess}} to avoid
every caller explicitly checking the bp before calling {{checkAccess}}.
Sleeping for 1s and and incrementing a counter until it reaches the number of
configured seconds is fragile - assumes the sleep really was for 1s which may
not be true if there was a long GC, etc. I'd suggest using a {{StopWatch}} for
correctness.
I think something similar needs to be done for the RPC service. Block tokens
cannot be authenticated until after registration when it gets the block secret.
The dfs client checks {{getReplicaVisibleLength}} for the last block if not
complete and the rpc client doesn't appear to have any retry proxy. This is
likely to affect users that frequently read while writing or appending to a
file (ex. logging into hdfs, perhaps hbase?).
Blocking in the RPC layer, unlike the data xceiver threads, is not desirable.
Once the readers jam due to one unregistered bp, admin calls or calls for other
block pools will be stalled too. Ideally the DN secret manager should throw a
{{RetriableException}} if the bp has no secrets. The client can handle the
retries. Appears it would be backwards compat.
> Reduce client failures during datanode restart
> --
>
> Key: HDFS-9574
> URL: https://issues.apache.org/jira/browse/HDFS-9574
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9574.patch, HDFS-9574.v2.patch
>
>
> Since DataXceiverServer is initialized before BP is fully up, client requests
> will fail until the datanode registers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068260#comment-15068260
]
Kihwal Lee commented on HDFS-9574:
--
{noformat}
mvn test -Dtest=TestDFSStripedOutputStreamWithFailure*
...
Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.532 sec - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
...
Results :
Tests run: 224, Failures: 0, Errors: 0, Skipped: 0
{noformat}
All reported test failures are not reproduced when run locally.
> Reduce client failures during datanode restart
> --
>
> Key: HDFS-9574
> URL: https://issues.apache.org/jira/browse/HDFS-9574
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9574.patch, HDFS-9574.v2.patch
>
>
> Since DataXceiverServer is initialized before BP is fully up, client requests
> will fail until the datanode registers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068237#comment-15068237
]
Kihwal Lee commented on HDFS-9574:
--
{noformat}
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.643 sec - in
org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.741 sec -
in org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.namenode.TestBackupNode
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.891 sec - in
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.523 sec - in
org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
Tests run: 11, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 1.102 sec - in
org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestDatanodeRegistration
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.489 sec - in
org.apache.hadoop.hdfs.TestDatanodeRegistration
Results :
Tests run: 39, Failures: 0, Errors: 0, Skipped: 6
{noformat}
> Reduce client failures during datanode restart
> --
>
> Key: HDFS-9574
> URL: https://issues.apache.org/jira/browse/HDFS-9574
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9574.patch, HDFS-9574.v2.patch
>
>
> Since DataXceiverServer is initialized before BP is fully up, client requests
> will fail until the datanode registers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067299#comment-15067299
]
Hadoop QA commented on HDFS-9574:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 613, now 613). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 30s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 14s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 186m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.server.namenode.TestBackupNode |
| | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailur
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066517#comment-15066517 ] Allen Wittenauer commented on HDFS-9574: bq. do you think we can configure jenkins to avoid this? No. bq. Since HDFS tests are run in parallel, concurrent hdfs builds are worse than before. Maybe this will finally be the push folks need to clean up the tests. > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066489#comment-15066489 ] Kihwal Lee commented on HDFS-9574: -- This precommit (#13934) and #13935 for HDFS-7163 ran on H9 at the same time, causing both to run for over 5 hours and failing many tests due to internal timeout. [~aw], do you think we can configure jenkins to avoid this? Since HDFS tests are run in parallel, concurrent hdfs builds are worse than before. > Reduce client failures during datanode restart > -- > > Key: HDFS-9574 > URL: https://issues.apache.org/jira/browse/HDFS-9574 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-9574.patch, HDFS-9574.v2.patch > > > Since DataXceiverServer is initialized before BP is fully up, client requests > will fail until the datanode registers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065020#comment-15065020
]
Hadoop QA commented on HDFS-9574:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 2s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 612, now 612). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 59s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 214m 11s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 216m 6s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
41s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 504m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFai
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064026#comment-15064026
]
Kihwal Lee commented on HDFS-9574:
--
All failed test cases pass when run locally, except this one.
{noformat}
Running org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint
Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 48.968 sec <<<
FAILURE!
- in org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint
testWithRemoteClient(org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint)
Time elapsed: 16.784 sec <<< ERROR!
java.io.IOException: Not ready to serve the block pool, Dummy-pool.
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAndWaitForBP(DataXceiver.java:218)
at
org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint.issueWriteBlockCall(TestDataXceiverLazyPersistHint.java:118)
at
org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint.testWithRemoteClient(TestDataXceiverLazyPersistHint.java:90)
testOverrideWithRemoteClient(org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint)
Time elapsed: 16.047 sec <<< ERROR!
java.io.IOException: Not ready to serve the block pool, Dummy-pool.
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAndWaitForBP(DataXceiver.java:218)
at
org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint.issueWriteBlockCall(TestDataXceiverLazyPersistHint.java:118)
at
org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint.testOverrideWithRemoteClient(TestDataXceiverLazyPersistHint.java:107)
testWithLocalClient(org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint)
Time elapsed: 16.049 sec <<< ERROR!
java.io.IOException: Not ready to serve the block pool, Dummy-pool.
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAndWaitForBP(DataXceiver.java:218)
at
org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint.issueWriteBlockCall(TestDataXceiverLazyPersistHint.java:118)
at
org.apache.hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint.testWithLocalClient(TestDataXceiverLazyPersistHint.java:75)
{noformat}
> Reduce client failures during datanode restart
> --
>
> Key: HDFS-9574
> URL: https://issues.apache.org/jira/browse/HDFS-9574
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9574.patch
>
>
> Since DataXceiverServer is initialized before BP is fully up, client requests
> will fail until the datanode registers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063349#comment-15063349
]
Hadoop QA commented on HDFS-9574:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 4 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 612, now 613). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 26s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 14s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 30s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 185m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.T
[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart
[
https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063037#comment-15063037
]
Kihwal Lee commented on HDFS-9574:
--
Read fails like this:
{noformat}
java.io.IOException: cannot find BPOfferService for bpid=BP-xxx
at
org.apache.hadoop.hdfs.server.datanode.DataNode.getDNRegistrationForBP(DataNode.java:1434)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:525)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Writes fail like this:
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:200)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:679)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> Reduce client failures during datanode restart
> --
>
> Key: HDFS-9574
> URL: https://issues.apache.org/jira/browse/HDFS-9574
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>
> Since DataXceiverServer is initialized before BP is fully up, client requests
> will fail until the datanode registers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
