[ 
https://issues.apache.org/jira/browse/HDDS-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Sarin updated HDDS-14964:
------------------------------
    Description: 
During a large-scale deletion test, the cluster was being filled with 100GB 
files nested across various directory structures. Writes were progressing 
successfully until ~90% disk utilization was reached on approximately 80% of 
the DataNodes. At that point, all large file writes started failing with 
{{{}ContainerNotOpenException{}}}.

Key observation: Small file writes on the same cluster succeeded without issue 
during this same time window. Only concurrent parallel writes of large files 
(100GB) were failing. This suggests the issue is not that there are zero 
writable containers on the cluster, but rather that the client - while in the 
middle of writing a large file - encounters a container that has transitioned 
to {{CLOSED}} state and is unable to recover or allocate a new one successfully 
in that context.

*Root cause hypothesis:* When writing a 100GB file, many chunks are sent 
sequentially to the same container. At high cluster utilization, a container 
can transition to {{CLOSED}} state mid-write (e.g., due to being full or the 
SCM closing it). The client hits {{ContainerNotOpenException}} on a 
{{WriteChunk}} call. While retry/failover logic exists, under high disk 
pressure the client is unable to get a new OPEN container allocated in time, 
causing the write to fail entirely.

Error Stack :
{code:java}
java.util.concurrent.CompletionException: 
org.apache.ratis.protocol.exceptions.StateMachineException: 
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 
20677 in CLOSED state
        at 
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
        at 
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
        at 
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
        at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at 
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at 
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:105)
        at 
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:66)
        at 
org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:147)
        at 
org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:351)
        at 
org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequestWithRetry$5(OrderedAsync.java:210)
        at 
java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
        at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at 
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:325)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:381)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:325)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
        at 
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
        at 
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.ratis.protocol.exceptions.StateMachineException: 
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 
20677 in CLOSED state
        at 
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
        at 
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
        at 
org.apache.ratis.server.impl.RaftServerImpl.writeAsyncImpl(RaftServerImpl.java:987)
        at 
org.apache.ratis.server.impl.RaftServerImpl.writeAsync(RaftServerImpl.java:960)
        at 
org.apache.ratis.server.impl.RaftServerImpl.replyFuture(RaftServerImpl.java:953)
        at 
org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:930)
        at 
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$11(RaftServerImpl.java:919)
        at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
        at 
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$12(RaftServerImpl.java:919)
        at 
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
        ... 3 more
Caused by: 
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException: 
Container 20677 in CLOSED state
        at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
 Method)
        at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at 
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at 
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at 
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
        at 
org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:265)
        at 
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:455)
        at 
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:441)
        at 
org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:408)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:313)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
        at 
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
        at 
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        ... 3 more
26/03/02 23:09:43 ERROR impl.OrderedAsync: Failed to send request, 
message=cmdType: WriteChunk
traceID: ""
containerID: 20677
datanodeUuid: "c15cc65e-f4eb-43f0-b27c-6b6a3fbc5a86"
writeChunk {
  blockID {
    containerID: 20677
    localID: 117883640217932227
    blockCommitSequenceId: 686720
    replicaIndex: 0
  }
  chunkData {
    chunkName: "117883640217932227_chunk_53"
    offset: 218103808
    len: 4194304
    checksumData {
      type: CRC32
      bytesPerChecksum: 1048576
      checksums: "\216\301\375\321"
      checksums: "\300\2022\n"
      checksums: "\334\002L0"
      checksums: "\335m=\322"
    }
  }
} {code}
*Steps to Reproduce:*
 # Set up an Apache Ozone cluster with sufficient DataNodes (e.g., 10+).
 # Write 100GB files concurrently across various directory structures using 
Freon (or similar tool) targeting petabyte-scale data.
 # Continue writing until approximately 80–90% disk utilization is reached on 
the majority of DataNodes.
 # Observe that large file writes begin failing with 
{{{}ContainerNotOpenException{}}}, while small file writes on the same cluster 
continue to succeed.

*Expected Behavior:*

Large file writes should either:
 * Successfully obtain a new OPEN container when the current container 
transitions to CLOSED mid-write and continue writing, OR
 * Fail early with a clear, user-facing error (e.g., "Insufficient cluster 
space") before the write is initiated, rather than failing mid-stream after 
chunks have already been sent.

Actual Behavior:

Large file (100GB) writes fail mid-stream with:
java.util.concurrent.CompletionException: 
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 
20677 in CLOSED state
at 
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
at 
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
...
Caused by: 
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException: 
Container 20677 in CLOSED state
at 
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
at 
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
...
 
Failed {{WriteChunk}} request details:
 * Container ID: 20677
 * Block local ID: 117883640217932227
 * Chunk: {{{}117883640217932227_chunk_53{}}}, offset: 218103808, len: 4194304

*Proposed Improvement:*

Similar to how quota validation is performed before a key {{PUT}} (checking 
namespace/space quota against available capacity), a pre-write cluster-level 
space check should be introduced before initiating a large file write. 
Specifically:
 * Before allocating block pipelines for a write, check whether the cluster has 
sufficient OPEN/allocable containers to accommodate the expected write size.
 * If the cluster is approaching full capacity and cannot allocate the required 
containers, fail fast with a clear error (e.g., {{ClusterStorageFullException}} 
or similar) rather than allowing the write to proceed and fail mid-stream with 
a confusing {{{}ContainerNotOpenException{}}}.

This would provide better UX, avoid partial writes/orphaned blocks, and align 
with the existing quota-check pattern already present at the OM layer.

  was:
During a large-scale deletion test, the cluster was being filled with 100GB 
files nested across various directory structures. Writes were progressing 
successfully until ~90% disk utilization was reached on approximately 80% of 
the DataNodes. At that point, all large file writes started failing with 
{{{}ContainerNotOpenException{}}}.

Key observation: Small file writes on the same cluster succeeded without issue 
during this same time window. Only concurrent parallel writes of large files 
(100GB) were failing. This suggests the issue is not that there are zero 
writable containers on the cluster, but rather that the client — while in the 
middle of writing a large file — encounters a container that has transitioned 
to {{CLOSED}} state and is unable to recover or allocate a new one successfully 
in that context.

*Root cause hypothesis:* When writing a 100GB file, many chunks are sent 
sequentially to the same container. At high cluster utilization, a container 
can transition to {{CLOSED}} state mid-write (e.g., due to being full or the 
SCM closing it). The client hits {{ContainerNotOpenException}} on a 
{{WriteChunk}} call. While retry/failover logic exists, under high disk 
pressure the client is unable to get a new OPEN container allocated in time, 
causing the write to fail entirely.

Error Stack :
{code:java}
java.util.concurrent.CompletionException: 
org.apache.ratis.protocol.exceptions.StateMachineException: 
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 
20677 in CLOSED state
        at 
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
        at 
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
        at 
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
        at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at 
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at 
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:105)
        at 
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:66)
        at 
org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:147)
        at 
org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:351)
        at 
org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequestWithRetry$5(OrderedAsync.java:210)
        at 
java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
        at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at 
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:325)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:381)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:325)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
        at 
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
        at 
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.ratis.protocol.exceptions.StateMachineException: 
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 
20677 in CLOSED state
        at 
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
        at 
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
        at 
org.apache.ratis.server.impl.RaftServerImpl.writeAsyncImpl(RaftServerImpl.java:987)
        at 
org.apache.ratis.server.impl.RaftServerImpl.writeAsync(RaftServerImpl.java:960)
        at 
org.apache.ratis.server.impl.RaftServerImpl.replyFuture(RaftServerImpl.java:953)
        at 
org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:930)
        at 
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$11(RaftServerImpl.java:919)
        at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
        at 
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$12(RaftServerImpl.java:919)
        at 
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
        ... 3 more
Caused by: 
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException: 
Container 20677 in CLOSED state
        at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
 Method)
        at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at 
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at 
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at 
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
        at 
org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:265)
        at 
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:455)
        at 
org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:441)
        at 
org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:408)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:313)
        at 
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
        at 
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
        at 
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at 
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        ... 3 more
26/03/02 23:09:43 ERROR impl.OrderedAsync: Failed to send request, 
message=cmdType: WriteChunk
traceID: ""
containerID: 20677
datanodeUuid: "c15cc65e-f4eb-43f0-b27c-6b6a3fbc5a86"
writeChunk {
  blockID {
    containerID: 20677
    localID: 117883640217932227
    blockCommitSequenceId: 686720
    replicaIndex: 0
  }
  chunkData {
    chunkName: "117883640217932227_chunk_53"
    offset: 218103808
    len: 4194304
    checksumData {
      type: CRC32
      bytesPerChecksum: 1048576
      checksums: "\216\301\375\321"
      checksums: "\300\2022\n"
      checksums: "\334\002L0"
      checksums: "\335m=\322"
    }
  }
} {code}
*Steps to Reproduce:*
 # Set up an Apache Ozone cluster with sufficient DataNodes (e.g., 10+).
 # Write 100GB files concurrently across various directory structures using 
Freon (or similar tool) targeting petabyte-scale data.
 # Continue writing until approximately 80–90% disk utilization is reached on 
the majority of DataNodes.
 # Observe that large file writes begin failing with 
{{{}ContainerNotOpenException{}}}, while small file writes on the same cluster 
continue to succeed.

*Expected Behavior:*

Large file writes should either:
 * Successfully obtain a new OPEN container when the current container 
transitions to CLOSED mid-write and continue writing, OR
 * Fail early with a clear, user-facing error (e.g., "Insufficient cluster 
space") before the write is initiated, rather than failing mid-stream after 
chunks have already been sent.

Actual Behavior:

Large file (100GB) writes fail mid-stream with:
java.util.concurrent.CompletionException: 
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 
20677 in CLOSED state
at 
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
at 
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
...
Caused by: 
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException: 
Container 20677 in CLOSED state
at 
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
at 
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
...
 
Failed {{WriteChunk}} request details:
 * Container ID: 20677
 * Block local ID: 117883640217932227
 * Chunk: {{{}117883640217932227_chunk_53{}}}, offset: 218103808, len: 4194304

*Proposed Improvement:*

Similar to how quota validation is performed before a key {{PUT}} (checking 
namespace/space quota against available capacity), a pre-write cluster-level 
space check should be introduced before initiating a large file write. 
Specifically:
 * Before allocating block pipelines for a write, check whether the cluster has 
sufficient OPEN/allocable containers to accommodate the expected write size.
 * If the cluster is approaching full capacity and cannot allocate the required 
containers, fail fast with a clear error (e.g., {{ClusterStorageFullException}} 
or similar) rather than allowing the write to proceed and fail mid-stream with 
a confusing {{{}ContainerNotOpenException{}}}.

This would provide better UX, avoid partial writes/orphaned blocks, and align 
with the existing quota-check pattern already present at the OM layer.


> Write failures with ContainerNotOpenException during large file (100GB) 
> concurrent parallel writes at high cluster utilization (~90%)
> -------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-14964
>                 URL: https://issues.apache.org/jira/browse/HDDS-14964
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Arun Sarin
>            Priority: Major
>
> During a large-scale deletion test, the cluster was being filled with 100GB 
> files nested across various directory structures. Writes were progressing 
> successfully until ~90% disk utilization was reached on approximately 80% of 
> the DataNodes. At that point, all large file writes started failing with 
> {{{}ContainerNotOpenException{}}}.
> Key observation: Small file writes on the same cluster succeeded without 
> issue during this same time window. Only concurrent parallel writes of large 
> files (100GB) were failing. This suggests the issue is not that there are 
> zero writable containers on the cluster, but rather that the client - while 
> in the middle of writing a large file - encounters a container that has 
> transitioned to {{CLOSED}} state and is unable to recover or allocate a new 
> one successfully in that context.
> *Root cause hypothesis:* When writing a 100GB file, many chunks are sent 
> sequentially to the same container. At high cluster utilization, a container 
> can transition to {{CLOSED}} state mid-write (e.g., due to being full or the 
> SCM closing it). The client hits {{ContainerNotOpenException}} on a 
> {{WriteChunk}} call. While retry/failover logic exists, under high disk 
> pressure the client is unable to get a new OPEN container allocated in time, 
> causing the write to fail entirely.
> Error Stack :
> {code:java}
> java.util.concurrent.CompletionException: 
> org.apache.ratis.protocol.exceptions.StateMachineException: 
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
> from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: 
> Container 20677 in CLOSED state
>       at 
> org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
>       at 
> org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
>       at 
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
>       at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>       at 
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
>       at 
> org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:105)
>       at 
> org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:66)
>       at 
> org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:147)
>       at 
> org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:351)
>       at 
> org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequestWithRetry$5(OrderedAsync.java:210)
>       at 
> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
>       at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>       at 
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
>       at 
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:325)
>       at java.base/java.util.Optional.ifPresent(Optional.java:178)
>       at 
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:381)
>       at 
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:325)
>       at 
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
>       at 
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
>       at 
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
>       at 
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
>       at 
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
>       at 
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
>       at 
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
>       at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>       at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>       at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.ratis.protocol.exceptions.StateMachineException: 
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
> from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: 
> Container 20677 in CLOSED state
>       at 
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
>       at 
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
>       at 
> org.apache.ratis.server.impl.RaftServerImpl.writeAsyncImpl(RaftServerImpl.java:987)
>       at 
> org.apache.ratis.server.impl.RaftServerImpl.writeAsync(RaftServerImpl.java:960)
>       at 
> org.apache.ratis.server.impl.RaftServerImpl.replyFuture(RaftServerImpl.java:953)
>       at 
> org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:930)
>       at 
> org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$11(RaftServerImpl.java:919)
>       at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
>       at 
> org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$12(RaftServerImpl.java:919)
>       at 
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
>       ... 3 more
> Caused by: 
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
>  Container 20677 in CLOSED state
>       at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method)
>       at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
>       at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>       at 
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
>       at 
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
>       at 
> org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:265)
>       at 
> org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:455)
>       at 
> org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:441)
>       at 
> org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:408)
>       at 
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:313)
>       at 
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
>       at 
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
>       at 
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
>       at 
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
>       at 
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
>       at 
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
>       at 
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
>       ... 3 more
> 26/03/02 23:09:43 ERROR impl.OrderedAsync: Failed to send request, 
> message=cmdType: WriteChunk
> traceID: ""
> containerID: 20677
> datanodeUuid: "c15cc65e-f4eb-43f0-b27c-6b6a3fbc5a86"
> writeChunk {
>   blockID {
>     containerID: 20677
>     localID: 117883640217932227
>     blockCommitSequenceId: 686720
>     replicaIndex: 0
>   }
>   chunkData {
>     chunkName: "117883640217932227_chunk_53"
>     offset: 218103808
>     len: 4194304
>     checksumData {
>       type: CRC32
>       bytesPerChecksum: 1048576
>       checksums: "\216\301\375\321"
>       checksums: "\300\2022\n"
>       checksums: "\334\002L0"
>       checksums: "\335m=\322"
>     }
>   }
> } {code}
> *Steps to Reproduce:*
>  # Set up an Apache Ozone cluster with sufficient DataNodes (e.g., 10+).
>  # Write 100GB files concurrently across various directory structures using 
> Freon (or similar tool) targeting petabyte-scale data.
>  # Continue writing until approximately 80–90% disk utilization is reached on 
> the majority of DataNodes.
>  # Observe that large file writes begin failing with 
> {{{}ContainerNotOpenException{}}}, while small file writes on the same 
> cluster continue to succeed.
> *Expected Behavior:*
> Large file writes should either:
>  * Successfully obtain a new OPEN container when the current container 
> transitions to CLOSED mid-write and continue writing, OR
>  * Fail early with a clear, user-facing error (e.g., "Insufficient cluster 
> space") before the write is initiated, rather than failing mid-stream after 
> chunks have already been sent.
> Actual Behavior:
> Large file (100GB) writes fail mid-stream with:
> java.util.concurrent.CompletionException: 
> org.apache.ratis.protocol.exceptions.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
> from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: 
> Container 20677 in CLOSED state
> at 
> org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
> at 
> org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
> ...
> Caused by: 
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
>  Container 20677 in CLOSED state
> at 
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
> at 
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
> ...
>  
> Failed {{WriteChunk}} request details:
>  * Container ID: 20677
>  * Block local ID: 117883640217932227
>  * Chunk: {{{}117883640217932227_chunk_53{}}}, offset: 218103808, len: 4194304
> *Proposed Improvement:*
> Similar to how quota validation is performed before a key {{PUT}} (checking 
> namespace/space quota against available capacity), a pre-write cluster-level 
> space check should be introduced before initiating a large file write. 
> Specifically:
>  * Before allocating block pipelines for a write, check whether the cluster 
> has sufficient OPEN/allocable containers to accommodate the expected write 
> size.
>  * If the cluster is approaching full capacity and cannot allocate the 
> required containers, fail fast with a clear error (e.g., 
> {{ClusterStorageFullException}} or similar) rather than allowing the write to 
> proceed and fail mid-stream with a confusing 
> {{{}ContainerNotOpenException{}}}.
> This would provide better UX, avoid partial writes/orphaned blocks, and align 
> with the existing quota-check pattern already present at the OM layer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]


Reply via email to