When trying to savepoint our job, we are getting the stack trace below. Is there a way to know more about this failure? Like which function in the job graph is associated with the problematic state and which key (assuming it is keyed state)?
Or is there a fix for this exception? The only mention of this exception that I can find is in [1] and [2]. [1] has a message at the bottom saying that the issue was fixed in RocksDb in 2018. And while we do have a part of the job graph that matches the pattern discussed in these two links, our attempts to reproduce the problem by pumping messages through at a rate millions of times higher than normal have not worked. We are using Flink version 1.13.5. Thanks, Mike [1] https://issues.apache.org/jira/browse/FLINK-9268 [2] https://www.mail-archive.com/user@flink.apache.org/msg34915.html Caused by: java.lang.Exception: Could not materialize checkpoint 49768 for operator KeyedProcess -> KeyedProcess -> re-operator-output -> Sink: Kafka sink to ec.platform.braid.responses-rtw (9/15)#0. at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:257) ... 4 more Caused by: java.util.concurrent.ExecutionException: java.lang.NegativeArraySizeException: -785722504 at java.base/java.util.concurrent.FutureTask.report(Unknown Source) at java.base/java.util.concurrent.FutureTask.get(Unknown Source) at org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:636) at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(OperatorSnapshotFinalizer.java:54) at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:128) ... 3 more Caused by: java.lang.NegativeArraySizeException: -785722504 at org.rocksdb.RocksIterator.$$YJP$$value0(Native Method) at org.rocksdb.RocksIterator.value0(RocksIterator.java) at org.rocksdb.RocksIterator.value(RocksIterator.java:50) at org.apache.flink.contrib.streaming.state.RocksIteratorWrapper.value(RocksIteratorWrapper.java:103) at org.apache.flink.contrib.streaming.state.iterator.RocksSingleStateIterator.value(RocksSingleStateIterator.java:66) at org.apache.flink.contrib.streaming.state.iterator.RocksStatesPerKeyGroupMergeIterator.value(RocksStatesPerKeyGroupMergeIterator.java:202) at org.apache.flink.runtime.state.FullSnapshotAsyncWriter.writeKVStateData(FullSnapshotAsyncWriter.java:210) at org.apache.flink.runtime.state.FullSnapshotAsyncWriter.writeSnapshotToOutputStream(FullSnapshotAsyncWriter.java:107) at org.apache.flink.runtime.state.FullSnapshotAsyncWriter.get(FullSnapshotAsyncWriter.java:77) at org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:91) at org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:88) at org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:78) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:633)
