ankitsultana opened a new issue, #11145: URL: https://github.com/apache/pinot/issues/11145
We are seeing an issue with one of our high ingestion throughput tables where ingestion lag continues to increase because of a table-level lock in Pinot controller. https://github.com/apache/pinot/blob/723b764bc91275c0b8361d3f9135f151b6404c39/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L594 @Jackie-Jiang I think we briefly discussed this yesterday. I don't have enough context about this, but one potential solution could be to add some jitter in the number of docs in a segment at partition level so these events arrive at slightly different times. But I don't think that is tenable and we may need a more proper fix. Also, there is a lock being taken in logback which is causing a lot of contention (there were 10 threads blocked in a sample thread dump I took). That seems like a simpler fix which we may wanna do anyways. ``` ... "grizzly-http-server-435" #625 prio=5 os_prio=0 cpu=16853811.42ms elapsed=7356619.65s tid=0x00007f0283f80000 nid=0x2d1 waiting for monitor entry [0x00007edb0c584000] java.lang.Thread.State: BLOCKED (on object monitor) at ch.qos.logback.classic.LoggerContext.getLogger(LoggerContext.java:151) - waiting to lock <0x00007ee26209cf80> (a ch.qos.logback.classic.Logger) at ch.qos.logback.classic.LoggerContext.getLogger(LoggerContext.java:53) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363) at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.<init>(SegmentCompletionManager.java:437) at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.fsmInHolding(SegmentCompletionManager.java:399) at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.lookupOrCreateFsm(SegmentCompletionManager.java:162) at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentConsumed(SegmentCompletionManager.java:197) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
