Thanks Stephen for explaining,
> I have a few thoughts on this, but my knowledge may be out-dated.
>
> 1. During putBlock, the DN notices the usage has gone beyond 90%, so it
> sends a close command to SCM via its heartbeat.
>
> 2. SCM closes the container on the SCM side. At this point, SCM will not
> allocate any more blocks to it but there may be some currently being
> written, previously allocated.
Even if this works correctly, it is possible for too many blocks being allocated
between 2 heartbeats.
> 3. The 5GB container limit is a soft-limit - its ok for a container to go
> beyond this size.
We observed all closed containers are less than 5GB on disk.
> 4. It was my understanding, although I cannot find the code right now, that
> there is some "grace period" for inflight blocks to complete writing when a
> container starts to close. If we stop allocating blocks in SCM because the
> close process has been triggered, then the grace period should allow most
> inflight blocks to complete writing.
>
> Does the grace period still exist, and if so, it is not helping with this
> problem?
I'm not sure, but we can see a lot of errors like this in the client log.
Please see the attachment for more details.
2022-09-06 15:43:57,044 [pool-2-thread-63] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-09-06 15:43:57,058 [pool-2-thread-55] WARN io.KeyOutputStream: EC stripe
write failed: S S S S S S S S S S S S F S
2022-09-06 15:43:57,058 [pool-2-thread-55] WARN io.KeyOutputStream: Failure
for replica index: 13, DatanodeDetails: ...
java.io.IOException: Unexpected Storage Container Exception:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException:
Requested operation not allowed as ContainerState is CLOSED
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.setIoException(BlockOutputStream.java:629)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
