Hi Vignesh, I think you can check the following things: 1. Check the cpu usage of the workers. Are they close to zero or almost full? 2. Any back pressure happened in downstream tasks? 3. Is the fullGC significant serious?
Best, Yuan > On 7 Apr 2022, at 12:33 PM, Vignesh Ramesh <vicki.ram...@gmail.com> wrote: > > Hi Team, > > We are using flink 1.14.4 and facing issues in production where our threads > are blocked waiting for > LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:346). > > "Legacy Source Thread - Source: ReindexBatchProcess Source (61/64)#0" Id=376 > WAITING on java.util.concurrent.CompletableFuture$Signaller@6ddd0af > at sun.misc.Unsafe.park(Native Method) > - waiting on java.util.concurrent.CompletableFuture$Signaller@6ddd0af > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707) > at > java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) > at > java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) > at org.apache.flink.runtime.io > <http://org.apache.flink.runtime.io/>.network.buffer.LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:346) > at org.apache.flink.runtime.io > <http://org.apache.flink.runtime.io/>.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:318) > > Attaching the thread dump with this mail. We also don't see any issues in > downstream processing . The network buffer memory and heap memory is also > used very less please refer below pic. > > <image.png> > > Kindly let us know if we can do anything to solve this issue.. We are also > open for commercial support. > <threaddump-latest.txt>
