xichen01 commented on PR #10566:
URL: https://github.com/apache/ozone/pull/10566#issuecomment-4819171174
> > ... SequenceIdGenerator#getNextId waits for SCMStateMachine to finish
applying DeleteScmKeyBlocks's
DeletedBlockLogStateManagerImpl#addTransactionsToDB
>
> Quick question: why getNextId will wait for addTransactionsToDB? Is it
happen only if a new batch of ids is needed?
Yes it is happen only if a new batch of ids is needed, to request the next
batch ID, you must go through Ratis. `SCMStateMachine's applyTransaction` is
also single-threaded; any slow transaction will block subsequent transactions.
This is a typical stack.
```
"IPC Server handler 23 on default port 9863" #289 [3407343] daemon prio=5
os_prio=0 cpu=581314.26ms elapsed=287858.01s tid=0x00007feb27451810 nid=3407343
waiting on condition [0x00007feb8e0b9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21-sdi/Native Method)
- parking to wait for <0x000004161650b580> (a
java.util.concurrent.CompletableFuture$Signaller)
at
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21-sdi/LockSupport.java:269)
at
java.util.concurrent.CompletableFuture$Signaller.block(java.base@21-sdi/CompletableFuture.java:1866)
at
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21-sdi/ForkJoinPool.java:3780)
at
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21-sdi/ForkJoinPool.java:3725)
at
java.util.concurrent.CompletableFuture.timedGet(java.base@21-sdi/CompletableFuture.java:1939)
at
java.util.concurrent.CompletableFuture.get(java.base@21-sdi/CompletableFuture.java:2095)
at
org.apache.hadoop.hdds.scm.ha.SCMRatisServerImpl.submitRequest(SCMRatisServerImpl.java:227)
at
org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeRatisServer(SCMHAInvocationHandler.java:123)
at
org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeRatis(SCMHAInvocationHandler.java:112)
at
org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:74)
at org.apache.hadoop.hdds.scm.ha.$Proxy14.allocateBatch(Unknown Source)
at
org.apache.hadoop.hdds.scm.ha.SequenceIdGenerator.getNextId(SequenceIdGenerator.java:142)
at
org.apache.hadoop.hdds.scm.block.BlockManagerImpl.newBlock(BlockManagerImpl.java:201)
at
org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:177)
at
org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:243)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:222)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:150)
at
//...
```
If the cluster's write and delete QPS are high, the end result will be a
large number of long-tail allocateSCMBlocks.
<img width="877" height="212" alt="image"
src="https://github.com/user-attachments/assets/4b4e6593-82c9-4b3a-974d-f5acdcc3b04f";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
