[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621033#comment-16621033
]
Hudson commented on HDDS-460:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15019 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15019/])
HDDS-460. Replication manager failed to import container data. (nanda: rev
042bf74d5eb3dc627658b8c4c027628569169513)
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java
* (add)
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/package-info.java
* (add)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisor.java
* (delete)
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestReplicateContainerCommandHandler.java
* (add)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ContainerReplicator.java
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java
* (add)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationTask.java
* (add)
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationSupervisor.java
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/ReplicateContainerCommandHandler.java
* (add)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/DownloadAndImportReplicator.java
> Replication manager failed to import container data
> ---
>
> Key: HDDS-460
> URL: https://issues.apache.org/jira/browse/HDDS-460
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1, 0.3.0
>
> Attachments: HDDS-460-ozone-0.2.001.patch,
> all-node-ozone-logs-1536920345.tar.gz
>
>
> replication manager is not able to import downloaded container data on
> datanode failure.
>
> snippet of ozone.log
> -
> {noformat}
> 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO
> (GrpcReplicationClient.java:161) - Container is downloaded to
> /tmp/container-copy/container-14.tar.gz
> 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR
> (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded
> container data id=8
> java.io.EOFException
> at
> org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
> at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
> at
> org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCall
[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621001#comment-16621001
]
Nanda kumar commented on HDDS-460:
--
Thanks [~nilotpalnandi] for reporting the issue and thanks to [~elek] for the
contribution. I have committed this to trunk and ozone-0.2.
> Replication manager failed to import container data
> ---
>
> Key: HDDS-460
> URL: https://issues.apache.org/jira/browse/HDDS-460
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1, 0.3.0
>
> Attachments: HDDS-460-ozone-0.2.001.patch,
> all-node-ozone-logs-1536920345.tar.gz
>
>
> replication manager is not able to import downloaded container data on
> datanode failure.
>
> snippet of ozone.log
> -
> {noformat}
> 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO
> (GrpcReplicationClient.java:161) - Container is downloaded to
> /tmp/container-copy/container-14.tar.gz
> 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR
> (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded
> container data id=8
> java.io.EOFException
> at
> org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
> at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
> at
> org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> at
> org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.shad
[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620968#comment-16620968
]
Nanda kumar commented on HDDS-460:
--
Created HDDS-510 to address the review comments above. I'm +1 on the patch,
will commit it shortly.
> Replication manager failed to import container data
> ---
>
> Key: HDDS-460
> URL: https://issues.apache.org/jira/browse/HDDS-460
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-460-ozone-0.2.001.patch,
> all-node-ozone-logs-1536920345.tar.gz
>
>
> replication manager is not able to import downloaded container data on
> datanode failure.
>
> snippet of ozone.log
> -
> {noformat}
> 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO
> (GrpcReplicationClient.java:161) - Container is downloaded to
> /tmp/container-copy/container-14.tar.gz
> 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR
> (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded
> container data id=8
> java.io.EOFException
> at
> org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
> at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
> at
> org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> at
> org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
>
[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620951#comment-16620951
]
Nanda kumar commented on HDDS-460:
--
Thanks [~elek] for working on the patch. Overall it looks pretty good.
In ReplicateContainerCommandHandler, we should not mark the command as success
[{{updateCommandStatus(context, command, true, LOG)]}} as soon as we queue
ReplicationTask. We have to update it only when the task status is marked as
DONE or FAILED.
ReplicationSupervisor pool size can be made configurable, this can be done in a
follow-up jira.
> Replication manager failed to import container data
> ---
>
> Key: HDDS-460
> URL: https://issues.apache.org/jira/browse/HDDS-460
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-460-ozone-0.2.001.patch,
> all-node-ozone-logs-1536920345.tar.gz
>
>
> replication manager is not able to import downloaded container data on
> datanode failure.
>
> snippet of ozone.log
> -
> {noformat}
> 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO
> (GrpcReplicationClient.java:161) - Container is downloaded to
> /tmp/container-copy/container-14.tar.gz
> 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR
> (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded
> container data id=8
> java.io.EOFException
> at
> org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
> at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
> at
> org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> at
> org.apache.ratis
[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615171#comment-16615171
]
Hadoop QA commented on HDDS-460:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} ozone-0.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
18s{color} | {color:green} ozone-0.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} ozone-0.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} ozone-0.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} ozone-0.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} ozone-0.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} ozone-0.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red} hadoop-hdds/container-service generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdds/container-service |
| | Unconditional wait in
org.apache.hadoop.ozone.container.replication.ReplicationSupervisor$Worker.run()
At ReplicationSupervisor.java:At ReplicationSupervisor.java:[line 122] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-460 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939756/HDDS-460-ozone-0.2.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 238f127af540 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | ozone-0.2 / 2ba438a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDDS-Build/1068/artifact/out/new-findbugs-hadoop-hdds_container-service.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1068/testReport/ |
| Max. process
[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615084#comment-16615084
]
Elek, Marton commented on HDDS-460:
---
thanks the information and the report [~nilotpalnandi]
I think there are two problems here: it seems that the replication manager
sends multiple replication request to the datanode. This is a smaller problem.
The bigger one is that the parallel processing in the datanode side is bad.
There is a lock but it's just for the import not for the download.
I created a patch which introduces a new class to handle the download queue. It
could help to monitor the process and also we can add more metrics to here (not
part of the patch).
> Replication manager failed to import container data
> ---
>
> Key: HDDS-460
> URL: https://issues.apache.org/jira/browse/HDDS-460
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Assignee: Elek, Marton
>Priority: Major
> Attachments: all-node-ozone-logs-1536920345.tar.gz
>
>
> replication manager is not able to import downloaded container data on
> datanode failure.
>
> snippet of ozone.log
> -
> {noformat}
> 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO
> (GrpcReplicationClient.java:161) - Container is downloaded to
> /tmp/container-copy/container-14.tar.gz
> 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR
> (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded
> container data id=8
> java.io.EOFException
> at
> org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
> at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
> at
> org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallI
[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614680#comment-16614680
]
Nilotpal Nandi commented on HDDS-460:
-
[~elek] : Its a non-zero size file :
{noformat}
Host::172.27.13.74
-rw-r--r-- 1 root root 1.4G Sep 14 08:58 /tmp/container-copy/container-38.tar.gz
-rw-r--r-- 1 root root 1.6G Sep 14 08:58 /tmp/container-copy/container-40.tar.gz
-rw-r--r-- 1 root root 2.1G Sep 14 08:59 /tmp/container-copy/container-34.tar.gz
-rw-r--r-- 1 root root 4.6G Sep 14 09:06 /tmp/container-copy/container-10.tar.gz
-rw-r--r-- 1 root root 4.6G Sep 14 09:06 /tmp/container-copy/container-6.tar.gz
-rw-r--r-- 1 root root 4.6G Sep 14 09:06 /tmp/container-copy/container-16.tar.gz
-rw-r--r-- 1 root root 4.6G Sep 14 09:31 /tmp/container-copy/container-14.tar.gz
-rw-r--r-- 1 root root 4.6G Sep 14 09:38
/tmp/container-copy/container-4.tar.gz{noformat}
> Replication manager failed to import container data
> ---
>
> Key: HDDS-460
> URL: https://issues.apache.org/jira/browse/HDDS-460
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Priority: Major
> Attachments: all-node-ozone-logs-1536920345.tar.gz
>
>
> replication manager is not able to import downloaded container data on
> datanode failure.
>
> snippet of ozone.log
> -
> {noformat}
> 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO
> (GrpcReplicationClient.java:161) - Container is downloaded to
> /tmp/container-copy/container-14.tar.gz
> 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR
> (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded
> container data id=8
> java.io.EOFException
> at
> org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
> at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
> at
> org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614647#comment-16614647
]
Elek, Marton commented on HDDS-460:
---
What is the size of /tmp/container-copy/container-14.tar.gz
I guess it is a zero size file. Is it true?
> Replication manager failed to import container data
> ---
>
> Key: HDDS-460
> URL: https://issues.apache.org/jira/browse/HDDS-460
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Priority: Major
> Attachments: all-node-ozone-logs-1536920345.tar.gz
>
>
> replication manager is not able to import downloaded container data on
> datanode failure.
>
> snippet of ozone.log
> -
> {noformat}
> 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO
> (GrpcReplicationClient.java:161) - Container is downloaded to
> /tmp/container-copy/container-14.tar.gz
> 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR
> (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded
> container data id=8
> java.io.EOFException
> at
> org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
> at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
> at
> org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> at
> org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(Threa
[jira] [Commented] (HDDS-460) Replication manager failed to import container data
[
https://issues.apache.org/jira/browse/HDDS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614635#comment-16614635
]
Nilotpal Nandi commented on HDDS-460:
-
logs from all nodes present here :
[^all-node-ozone-logs-1536920345.tar.gz]
> Replication manager failed to import container data
> ---
>
> Key: HDDS-460
> URL: https://issues.apache.org/jira/browse/HDDS-460
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Priority: Major
> Attachments: all-node-ozone-logs-1536920345.tar.gz
>
>
> replication manager is not able to import downloaded container data on
> datanode failure.
>
> snippet of ozone.log
> -
> {noformat}
> 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO
> (GrpcReplicationClient.java:161) - Container is downloaded to
> /tmp/container-copy/container-14.tar.gz
> 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR
> (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded
> container data id=8
> java.io.EOFException
> at
> org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
> at
> org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
> at
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
> at
> org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
> at
> org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> at
> org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> at
> org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
