[jira] [Commented] (FLINK-22962) Key group is not in KeyGroupRange error while checkpointing

2021-06-10 Thread Prateek Kohli (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17360956#comment-17360956
 ] 

Prateek Kohli commented on FLINK-22962:
---

We have designed our keyby logic in such a way that we are using 
KeySelector i.e. our getKey() method is returning 
Serializable.
Also, our KeyedProcessFunction uses Object type for 
key(KeyedProcessFunction) instead of any concrete class.
Can this be the cause of any error?

> Key group is not in KeyGroupRange error while checkpointing
> ---
>
> Key: FLINK-22962
> URL: https://issues.apache.org/jira/browse/FLINK-22962
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Affects Versions: 1.12.1
> Environment: Linux
>Reporter: Prateek Kohli
>Priority: Major
>
> Hi,
>  
> We are getting the below exception while using rocksdb as state backend at 
> the time of checkpointing:
> 2021-06-10 12:05:13,933 INFO 
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - 
> Aggregator (3/4)#0 - asynchronous part of checkpoint 2 could not be completed.
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Key group 0 is not in KeyGroupRange\{startKeyGroup=5, endKeyGroup=7}.
>  at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_261]
>  at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_261]
>  at 
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:621)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:54)
>  ~[flink-streaming-java_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:122)
>  [flink-streaming-java_2.11-1.12.1.jar:1.12.1]
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  [?:1.8.0_261]
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [?:1.8.0_261]
>  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
> Caused by: java.lang.IllegalArgumentException: Key group 0 is not in 
> KeyGroupRange\{startKeyGroup=5, endKeyGroup=7}.
>  at 
> org.apache.flink.runtime.state.KeyGroupRangeOffsets.computeKeyGroupIndex(KeyGroupRangeOffsets.java:144)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.runtime.state.KeyGroupRangeOffsets.setKeyGroupOffset(KeyGroupRangeOffsets.java:106)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.writeKVStateData(RocksFullSnapshotStrategy.java:333)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.writeSnapshotToOutputStream(RocksFullSnapshotStrategy.java:264)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.callInternal(RocksFullSnapshotStrategy.java:227)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.callInternal(RocksFullSnapshotStrategy.java:180)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:78)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_261]
>  at 
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:618)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  ... 5 more
>  
> When we change the state backend to file or heap we do not get this error.
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-22962) Key group is not in KeyGroupRange error while checkpointing

2021-12-07 Thread Yun Tang (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454991#comment-17454991
 ] 

Yun Tang commented on FLINK-22962:
--

[~prateekkohli2112] After flink-1.13, we unify the format of savepoint of 
RocksDB and heap keyed state backends. Will you still face the problem?



> Key group is not in KeyGroupRange error while checkpointing
> ---
>
> Key: FLINK-22962
> URL: https://issues.apache.org/jira/browse/FLINK-22962
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Affects Versions: 1.12.1
> Environment: Linux
>Reporter: Prateek Kohli
>Priority: Major
>
> Hi,
>  
> We are getting the below exception while using rocksdb as state backend at 
> the time of checkpointing:
> 2021-06-10 12:05:13,933 INFO 
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - 
> Aggregator (3/4)#0 - asynchronous part of checkpoint 2 could not be completed.
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Key group 0 is not in KeyGroupRange\{startKeyGroup=5, endKeyGroup=7}.
>  at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_261]
>  at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_261]
>  at 
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:621)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:54)
>  ~[flink-streaming-java_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:122)
>  [flink-streaming-java_2.11-1.12.1.jar:1.12.1]
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  [?:1.8.0_261]
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [?:1.8.0_261]
>  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
> Caused by: java.lang.IllegalArgumentException: Key group 0 is not in 
> KeyGroupRange\{startKeyGroup=5, endKeyGroup=7}.
>  at 
> org.apache.flink.runtime.state.KeyGroupRangeOffsets.computeKeyGroupIndex(KeyGroupRangeOffsets.java:144)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.runtime.state.KeyGroupRangeOffsets.setKeyGroupOffset(KeyGroupRangeOffsets.java:106)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.writeKVStateData(RocksFullSnapshotStrategy.java:333)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.writeSnapshotToOutputStream(RocksFullSnapshotStrategy.java:264)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.callInternal(RocksFullSnapshotStrategy.java:227)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.callInternal(RocksFullSnapshotStrategy.java:180)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at 
> org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:78)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_261]
>  at 
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:618)
>  ~[flink-dist_2.11-1.12.1.jar:1.12.1]
>  ... 5 more
>  
> When we change the state backend to file or heap we do not get this error.
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)