Hi Li, The error suggests that Job is not able to acquire the required TaskManager TaskSlots within the configured time duration of 5 minutes. Job Runs on the TaskManagers (Worker Nodes). Helpful Link - https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/flink-architecture/#anatomy-of-a-flink-cluster TaskSlot information is available in the Flink GUI as below. If you see available as 0, and the job is not running, it means it is not able to acquire the required resources (CPU/MEM). For example - 1 taskslot = 1 compute resource which could be 1 CPU and 4GB of RAM, this is the config that you define and you get possible Task Slots for your node type.
[image: image.png] Thanks Saurabh On Mon, Jul 22, 2024 at 11:01 AM Li Shao <lsgreat12...@gmail.com> wrote: > Hi All, > We are using flink batch mode to process s3 files. However, recently we > are seeing the errors like: > > > Caused by: java.util.concurrent.CompletionException: > org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: > Slot request bulk is not fulfillable! Could not allocate the required slot > within slot request timeout > > at > java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown > Source) > > at > java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown > Source) > > at > java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown > Source) > > ... 36 more > > Caused by: > org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: > Slot request bulk is not fulfillable! Could not allocate the required slot > within slot request timeout > > at > org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86) > > ... 29 more > > Caused by: java.util.concurrent.TimeoutException: Timeout has occurred: > 300000 ms > > ... 30 more > > > I can see the job manager & task manager still got memory to run jobs. > Anyone can help me on this? > > Job manager: JVM: 5.02 GB / 6.00 GB > jobmanager.heap.size 6144m > jobmanager.memory.heap.size 6442450944b > jobmanager.memory.jvm-metaspace.size 4294967296b > jobmanager.memory.jvm-overhead.max 1073741824b > jobmanager.memory.jvm-overhead.min 1073741824b > jobmanager.memory.off-heap.size 134217728b >
