Re: failed when job graph change

2023-12-03 Thread nick toker
Hi restart the job it's ok and i do that , but i must cancel the job and submit a new one and i dont want the data from the state forget to mention that i use the parameter "-allowNonRestoredState" my steps: 1. stop the job with savepoint 2. run the updated job ( update job graph) from

Re:Re: Exploring Data Storage Mechanisms in Flink with RocksDB: A Focus on Windowing and Aggregation

2023-12-03 Thread Xuyang
Hi, IIUC, yes. -- Best! Xuyang 在 2023-12-04 15:13:56,"arjun s" 写道: Thank you for providing the details. Can it be confirmed that the Hashmap within the accumulator stores the map in RocksDB as a binary object and undergoes deserialization/serialization during the execution of

Re: Exploring Data Storage Mechanisms in Flink with RocksDB: A Focus on Windowing and Aggregation

2023-12-03 Thread arjun s
Thank you for providing the details. Can it be confirmed that the Hashmap within the accumulator stores the map in RocksDB as a binary object and undergoes deserialization/serialization during the execution of the aggregate function? Thanks, Arjun On Mon, 4 Dec 2023 at 12:24, Xuyang wrote: >

Re:Exploring Data Storage Mechanisms in Flink with RocksDB: A Focus on Windowing and Aggregation

2023-12-03 Thread Xuyang
Hi, Arjun. > I'm using a HashMap to aggregate the results. Do you means the you define a hashMap in the accumulator? If yes, I think it restores a binary object about map in RocksDB and deserialize it like this[1]. If you are using flink sql, you can try to debug the class 'WindowOperator' or

Re:failed when job graph change

2023-12-03 Thread Xuyang
Hi, nick. > using savepoint i must cancel the job to be able run the new graph Do you mean that you need cancel and start the job using the new flink job graph in 1.17.1, and in the past, it was able to make the changes to the new operator effective without restarting the job? I think in

Re:Re:Flink SQL作业配置'table.exec.sink.upsert-materialize'参数会影响TIMESTAMP类型精度?

2023-12-03 Thread Xuyang
Hi, 可以提供一下最小能复现的query么? 我在本地尝试了下貌似并未复现,source数据为: 1,1,1970-01-01 00:00:00.001001 2,2,1970-01-01 00:00:00.002002 3,3,1970-01-01 00:00:00.003003 query为: //tEnv.getConfig.getConfiguration // .set(ExecutionConfigOptions.TABLE_EXEC_SINK_UPSERT_MATERIALIZE,

failed when job graph change

2023-12-03 Thread nick toker
Hi when i add or remove an operator in the job graph , using savepoint i must cancel the job to be able run the new graph e.g. by adding or removing operator (like new sink target) it was working in the past i using flink 1.17.1 1. is it a known bug? if so when planned to be fix 2. do i need

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

2023-12-03 Thread Barak Ben-Nathan
Hi guys, Forking in sbt solved the issue (Test / fork := true). On Sun, Dec 3, 2023 at 7:48 AM Barak Ben-Nathan wrote: > By the way, I also upgraded to flink-connector-kafka ver. 3.0.2-1.18, to > no avail. > > On Sun, Dec 3, 2023 at 7:45 AM Barak Ben-Nathan > wrote: > >> Thank's Jim, >> >>