[
https://issues.apache.org/jira/browse/HDDS-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Sarin updated HDDS-14964:
------------------------------
Description:
During a large-scale deletion test, the cluster was being filled with 100GB
files nested across various directory structures. Writes were progressing
successfully until ~90% disk utilization was reached on approximately 80% of
the DataNodes. At that point, all large file writes started failing with
{{{}ContainerNotOpenException{}}}.
Key observation: Small file writes on the same cluster succeeded without issue
during this same time window. Only concurrent parallel writes of large files
(100GB) were failing. This suggests the issue is not that there are zero
writable containers on the cluster, but rather that the client - while in the
middle of writing a large file - encounters a container that has transitioned
to {{CLOSED}} state and is unable to recover or allocate a new one successfully
in that context.
*Root cause hypothesis:* When writing a 100GB file, many chunks are sent
sequentially to the same container. At high cluster utilization, a container
can transition to {{CLOSED}} state mid-write (e.g., due to being full or the
SCM closing it). The client hits {{ContainerNotOpenException}} on a
{{WriteChunk}} call. While retry/failover logic exists, under high disk
pressure the client is unable to get a new OPEN container allocated in time,
causing the write to fail entirely.
Error Stack :
{code:java}
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:105)
at
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:66)
at
org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:147)
at
org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:351)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequestWithRetry$5(OrderedAsync.java:210)
at
java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:325)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:381)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:325)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
at
org.apache.ratis.server.impl.RaftServerImpl.writeAsyncImpl(RaftServerImpl.java:987)
at
org.apache.ratis.server.impl.RaftServerImpl.writeAsync(RaftServerImpl.java:960)
at
org.apache.ratis.server.impl.RaftServerImpl.replyFuture(RaftServerImpl.java:953)
at
org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:930)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$11(RaftServerImpl.java:919)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$12(RaftServerImpl.java:919)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
... 3 more
Caused by:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
Container 20677 in CLOSED state
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at
org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:265)
at
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:455)
at
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:441)
at
org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:408)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:313)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
... 3 more
26/03/02 23:09:43 ERROR impl.OrderedAsync: Failed to send request,
message=cmdType: WriteChunk
traceID: ""
containerID: 20677
datanodeUuid: "c15cc65e-f4eb-43f0-b27c-6b6a3fbc5a86"
writeChunk {
blockID {
containerID: 20677
localID: 117883640217932227
blockCommitSequenceId: 686720
replicaIndex: 0
}
chunkData {
chunkName: "117883640217932227_chunk_53"
offset: 218103808
len: 4194304
checksumData {
type: CRC32
bytesPerChecksum: 1048576
checksums: "\216\301\375\321"
checksums: "\300\2022\n"
checksums: "\334\002L0"
checksums: "\335m=\322"
}
}
} {code}
*Steps to Reproduce:*
# Set up an Apache Ozone cluster with sufficient DataNodes (e.g., 10+).
# Write 100GB files concurrently across various directory structures using
Freon (or similar tool) targeting petabyte-scale data.
# Continue writing until approximately 80–90% disk utilization is reached on
the majority of DataNodes.
# Observe that large file writes begin failing with
{{{}ContainerNotOpenException{}}}, while small file writes on the same cluster
continue to succeed.
*Expected Behavior:*
Large file writes should either:
* Successfully obtain a new OPEN container when the current container
transitions to CLOSED mid-write and continue writing, OR
* Fail early with a clear, user-facing error (e.g., "Insufficient cluster
space") before the write is initiated, rather than failing mid-stream after
chunks have already been sent.
Actual Behavior:
Large file (100GB) writes fail mid-stream with:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
...
Caused by:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
Container 20677 in CLOSED state
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
...
Failed {{WriteChunk}} request details:
* Container ID: 20677
* Block local ID: 117883640217932227
* Chunk: {{{}117883640217932227_chunk_53{}}}, offset: 218103808, len: 4194304
*Proposed Improvement:*
Similar to how quota validation is performed before a key {{PUT}} (checking
namespace/space quota against available capacity), a pre-write cluster-level
space check should be introduced before initiating a large file write.
Specifically:
* Before allocating block pipelines for a write, check whether the cluster has
sufficient OPEN/allocable containers to accommodate the expected write size.
* If the cluster is approaching full capacity and cannot allocate the required
containers, fail fast with a clear error (e.g., {{ClusterStorageFullException}}
or similar) rather than allowing the write to proceed and fail mid-stream with
a confusing {{{}ContainerNotOpenException{}}}.
This would provide better UX, avoid partial writes/orphaned blocks, and align
with the existing quota-check pattern already present at the OM layer.
was:
During a large-scale deletion test, the cluster was being filled with 100GB
files nested across various directory structures. Writes were progressing
successfully until ~90% disk utilization was reached on approximately 80% of
the DataNodes. At that point, all large file writes started failing with
{{{}ContainerNotOpenException{}}}.
Key observation: Small file writes on the same cluster succeeded without issue
during this same time window. Only concurrent parallel writes of large files
(100GB) were failing. This suggests the issue is not that there are zero
writable containers on the cluster, but rather that the client — while in the
middle of writing a large file — encounters a container that has transitioned
to {{CLOSED}} state and is unable to recover or allocate a new one successfully
in that context.
*Root cause hypothesis:* When writing a 100GB file, many chunks are sent
sequentially to the same container. At high cluster utilization, a container
can transition to {{CLOSED}} state mid-write (e.g., due to being full or the
SCM closing it). The client hits {{ContainerNotOpenException}} on a
{{WriteChunk}} call. While retry/failover logic exists, under high disk
pressure the client is unable to get a new OPEN container allocated in time,
causing the write to fail entirely.
Error Stack :
{code:java}
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:105)
at
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:66)
at
org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:147)
at
org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:351)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequestWithRetry$5(OrderedAsync.java:210)
at
java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:325)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:381)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:325)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
at
org.apache.ratis.server.impl.RaftServerImpl.writeAsyncImpl(RaftServerImpl.java:987)
at
org.apache.ratis.server.impl.RaftServerImpl.writeAsync(RaftServerImpl.java:960)
at
org.apache.ratis.server.impl.RaftServerImpl.replyFuture(RaftServerImpl.java:953)
at
org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:930)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$11(RaftServerImpl.java:919)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$12(RaftServerImpl.java:919)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
... 3 more
Caused by:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
Container 20677 in CLOSED state
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at
org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:265)
at
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:455)
at
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:441)
at
org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:408)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:313)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
... 3 more
26/03/02 23:09:43 ERROR impl.OrderedAsync: Failed to send request,
message=cmdType: WriteChunk
traceID: ""
containerID: 20677
datanodeUuid: "c15cc65e-f4eb-43f0-b27c-6b6a3fbc5a86"
writeChunk {
blockID {
containerID: 20677
localID: 117883640217932227
blockCommitSequenceId: 686720
replicaIndex: 0
}
chunkData {
chunkName: "117883640217932227_chunk_53"
offset: 218103808
len: 4194304
checksumData {
type: CRC32
bytesPerChecksum: 1048576
checksums: "\216\301\375\321"
checksums: "\300\2022\n"
checksums: "\334\002L0"
checksums: "\335m=\322"
}
}
} {code}
*Steps to Reproduce:*
# Set up an Apache Ozone cluster with sufficient DataNodes (e.g., 10+).
# Write 100GB files concurrently across various directory structures using
Freon (or similar tool) targeting petabyte-scale data.
# Continue writing until approximately 80–90% disk utilization is reached on
the majority of DataNodes.
# Observe that large file writes begin failing with
{{{}ContainerNotOpenException{}}}, while small file writes on the same cluster
continue to succeed.
*Expected Behavior:*
Large file writes should either:
* Successfully obtain a new OPEN container when the current container
transitions to CLOSED mid-write and continue writing, OR
* Fail early with a clear, user-facing error (e.g., "Insufficient cluster
space") before the write is initiated, rather than failing mid-stream after
chunks have already been sent.
Actual Behavior:
Large file (100GB) writes fail mid-stream with:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
...
Caused by:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
Container 20677 in CLOSED state
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
...
Failed {{WriteChunk}} request details:
* Container ID: 20677
* Block local ID: 117883640217932227
* Chunk: {{{}117883640217932227_chunk_53{}}}, offset: 218103808, len: 4194304
*Proposed Improvement:*
Similar to how quota validation is performed before a key {{PUT}} (checking
namespace/space quota against available capacity), a pre-write cluster-level
space check should be introduced before initiating a large file write.
Specifically:
* Before allocating block pipelines for a write, check whether the cluster has
sufficient OPEN/allocable containers to accommodate the expected write size.
* If the cluster is approaching full capacity and cannot allocate the required
containers, fail fast with a clear error (e.g., {{ClusterStorageFullException}}
or similar) rather than allowing the write to proceed and fail mid-stream with
a confusing {{{}ContainerNotOpenException{}}}.
This would provide better UX, avoid partial writes/orphaned blocks, and align
with the existing quota-check pattern already present at the OM layer.
> Write failures with ContainerNotOpenException during large file (100GB)
> concurrent parallel writes at high cluster utilization (~90%)
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-14964
> URL: https://issues.apache.org/jira/browse/HDDS-14964
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Arun Sarin
> Priority: Major
>
> During a large-scale deletion test, the cluster was being filled with 100GB
> files nested across various directory structures. Writes were progressing
> successfully until ~90% disk utilization was reached on approximately 80% of
> the DataNodes. At that point, all large file writes started failing with
> {{{}ContainerNotOpenException{}}}.
> Key observation: Small file writes on the same cluster succeeded without
> issue during this same time window. Only concurrent parallel writes of large
> files (100GB) were failing. This suggests the issue is not that there are
> zero writable containers on the cluster, but rather that the client - while
> in the middle of writing a large file - encounters a container that has
> transitioned to {{CLOSED}} state and is unable to recover or allocate a new
> one successfully in that context.
> *Root cause hypothesis:* When writing a 100GB file, many chunks are sent
> sequentially to the same container. At high cluster utilization, a container
> can transition to {{CLOSED}} state mid-write (e.g., due to being full or the
> SCM closing it). The client hits {{ContainerNotOpenException}} on a
> {{WriteChunk}} call. While retry/failover logic exists, under high disk
> pressure the client is unable to get a new OPEN container allocated in time,
> causing the write to fail entirely.
> Error Stack :
> {code:java}
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.exceptions.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
> from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68:
> Container 20677 in CLOSED state
> at
> org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
> at
> org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
> at
> org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:105)
> at
> org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:66)
> at
> org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:147)
> at
> org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:351)
> at
> org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequestWithRetry$5(OrderedAsync.java:210)
> at
> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:325)
> at java.base/java.util.Optional.ifPresent(Optional.java:178)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:381)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:325)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.ratis.protocol.exceptions.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
> from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68:
> Container 20677 in CLOSED state
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
> at
> org.apache.ratis.server.impl.RaftServerImpl.writeAsyncImpl(RaftServerImpl.java:987)
> at
> org.apache.ratis.server.impl.RaftServerImpl.writeAsync(RaftServerImpl.java:960)
> at
> org.apache.ratis.server.impl.RaftServerImpl.replyFuture(RaftServerImpl.java:953)
> at
> org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:930)
> at
> org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$11(RaftServerImpl.java:919)
> at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
> at
> org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$12(RaftServerImpl.java:919)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
> ... 3 more
> Caused by:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
> Container 20677 in CLOSED state
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
> at
> org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:265)
> at
> org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:455)
> at
> org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:441)
> at
> org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:408)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:313)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> ... 3 more
> 26/03/02 23:09:43 ERROR impl.OrderedAsync: Failed to send request,
> message=cmdType: WriteChunk
> traceID: ""
> containerID: 20677
> datanodeUuid: "c15cc65e-f4eb-43f0-b27c-6b6a3fbc5a86"
> writeChunk {
> blockID {
> containerID: 20677
> localID: 117883640217932227
> blockCommitSequenceId: 686720
> replicaIndex: 0
> }
> chunkData {
> chunkName: "117883640217932227_chunk_53"
> offset: 218103808
> len: 4194304
> checksumData {
> type: CRC32
> bytesPerChecksum: 1048576
> checksums: "\216\301\375\321"
> checksums: "\300\2022\n"
> checksums: "\334\002L0"
> checksums: "\335m=\322"
> }
> }
> } {code}
> *Steps to Reproduce:*
> # Set up an Apache Ozone cluster with sufficient DataNodes (e.g., 10+).
> # Write 100GB files concurrently across various directory structures using
> Freon (or similar tool) targeting petabyte-scale data.
> # Continue writing until approximately 80–90% disk utilization is reached on
> the majority of DataNodes.
> # Observe that large file writes begin failing with
> {{{}ContainerNotOpenException{}}}, while small file writes on the same
> cluster continue to succeed.
> *Expected Behavior:*
> Large file writes should either:
> * Successfully obtain a new OPEN container when the current container
> transitions to CLOSED mid-write and continue writing, OR
> * Fail early with a clear, user-facing error (e.g., "Insufficient cluster
> space") before the write is initiated, rather than failing mid-stream after
> chunks have already been sent.
> Actual Behavior:
> Large file (100GB) writes fail mid-stream with:
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.exceptions.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
> from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68:
> Container 20677 in CLOSED state
> at
> org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
> at
> org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
> ...
> Caused by:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
> Container 20677 in CLOSED state
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
> ...
>
> Failed {{WriteChunk}} request details:
> * Container ID: 20677
> * Block local ID: 117883640217932227
> * Chunk: {{{}117883640217932227_chunk_53{}}}, offset: 218103808, len: 4194304
> *Proposed Improvement:*
> Similar to how quota validation is performed before a key {{PUT}} (checking
> namespace/space quota against available capacity), a pre-write cluster-level
> space check should be introduced before initiating a large file write.
> Specifically:
> * Before allocating block pipelines for a write, check whether the cluster
> has sufficient OPEN/allocable containers to accommodate the expected write
> size.
> * If the cluster is approaching full capacity and cannot allocate the
> required containers, fail fast with a clear error (e.g.,
> {{ClusterStorageFullException}} or similar) rather than allowing the write to
> proceed and fail mid-stream with a confusing
> {{{}ContainerNotOpenException{}}}.
> This would provide better UX, avoid partial writes/orphaned blocks, and align
> with the existing quota-check pattern already present at the OM layer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]