Arun Sarin created HDDS-14964:
---------------------------------
Summary: Write failures with ContainerNotOpenException during
large file (100GB) concurrent parallel writes at high cluster utilization (~90%)
Key: HDDS-14964
URL: https://issues.apache.org/jira/browse/HDDS-14964
Project: Apache Ozone
Issue Type: Improvement
Reporter: Arun Sarin
During a large-scale deletion test, the cluster was being filled with 100GB
files nested across various directory structures. Writes were progressing
successfully until ~90% disk utilization was reached on approximately 80% of
the DataNodes. At that point, all large file writes started failing with
{{{}ContainerNotOpenException{}}}.
Key observation: Small file writes on the same cluster succeeded without issue
during this same time window. Only concurrent parallel writes of large files
(100GB) were failing. This suggests the issue is not that there are zero
writable containers on the cluster, but rather that the client — while in the
middle of writing a large file — encounters a container that has transitioned
to {{CLOSED}} state and is unable to recover or allocate a new one successfully
in that context.
*Root cause hypothesis:* When writing a 100GB file, many chunks are sent
sequentially to the same container. At high cluster utilization, a container
can transition to {{CLOSED}} state mid-write (e.g., due to being full or the
SCM closing it). The client hits {{ContainerNotOpenException}} on a
{{WriteChunk}} call. While retry/failover logic exists, under high disk
pressure the client is unable to get a new OPEN container allocated in time,
causing the write to fail entirely.
Error Stack :
{code:java}
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:105)
at
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:66)
at
org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:147)
at
org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:351)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequestWithRetry$5(OrderedAsync.java:210)
at
java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:325)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:381)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:325)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
at
org.apache.ratis.server.impl.RaftServerImpl.writeAsyncImpl(RaftServerImpl.java:987)
at
org.apache.ratis.server.impl.RaftServerImpl.writeAsync(RaftServerImpl.java:960)
at
org.apache.ratis.server.impl.RaftServerImpl.replyFuture(RaftServerImpl.java:953)
at
org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:930)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$11(RaftServerImpl.java:919)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$12(RaftServerImpl.java:919)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
... 3 more
Caused by:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
Container 20677 in CLOSED state
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at
org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:265)
at
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:455)
at
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:441)
at
org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:408)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:313)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
... 3 more
26/03/02 23:09:43 ERROR impl.OrderedAsync: Failed to send request,
message=cmdType: WriteChunk
traceID: ""
containerID: 20677
datanodeUuid: "c15cc65e-f4eb-43f0-b27c-6b6a3fbc5a86"
writeChunk {
blockID {
containerID: 20677
localID: 117883640217932227
blockCommitSequenceId: 686720
replicaIndex: 0
}
chunkData {
chunkName: "117883640217932227_chunk_53"
offset: 218103808
len: 4194304
checksumData {
type: CRC32
bytesPerChecksum: 1048576
checksums: "\216\301\375\321"
checksums: "\300\2022\n"
checksums: "\334\002L0"
checksums: "\335m=\322"
}
}
} {code}
*Steps to Reproduce:*
# Set up an Apache Ozone cluster with sufficient DataNodes (e.g., 10+).
# Write 100GB files concurrently across various directory structures using
Freon (or similar tool) targeting petabyte-scale data.
# Continue writing until approximately 80–90% disk utilization is reached on
the majority of DataNodes.
# Observe that large file writes begin failing with
{{{}ContainerNotOpenException{}}}, while small file writes on the same cluster
continue to succeed.
*Expected Behavior:*
Large file writes should either:
* Successfully obtain a new OPEN container when the current container
transitions to CLOSED mid-write and continue writing, OR
* Fail early with a clear, user-facing error (e.g., "Insufficient cluster
space") before the write is initiated, rather than failing mid-stream after
chunks have already been sent.
Actual Behavior:
Large file (100GB) writes fail mid-stream with:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container
20677 in CLOSED state
at
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
...
Caused by:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
Container 20677 in CLOSED state
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
...
Failed {{WriteChunk}} request details:
* Container ID: 20677
* Block local ID: 117883640217932227
* Chunk: {{{}117883640217932227_chunk_53{}}}, offset: 218103808, len: 4194304
*Proposed Improvement:*
Similar to how quota validation is performed before a key {{PUT}} (checking
namespace/space quota against available capacity), a pre-write cluster-level
space check should be introduced before initiating a large file write.
Specifically:
* Before allocating block pipelines for a write, check whether the cluster has
sufficient OPEN/allocable containers to accommodate the expected write size.
* If the cluster is approaching full capacity and cannot allocate the required
containers, fail fast with a clear error (e.g., {{ClusterStorageFullException}}
or similar) rather than allowing the write to proceed and fail mid-stream with
a confusing {{{}ContainerNotOpenException{}}}.
This would provide better UX, avoid partial writes/orphaned blocks, and align
with the existing quota-check pattern already present at the OM layer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]