Re:Re: Clear the State Backends in Flink

2023-11-06 Thread Xuyang
Hi, Arjun. Are you using DataStream api? Maybe you can refer this doc[1] to set an operator-level state TTL to let the state cleared automatically. Back to your scene, do you use state explicitl in some operators to store file names? If not and using a DataStream api, and I'm not mistaken, Flin

Re: Clear the State Backends in Flink

2023-11-06 Thread arjun s
Thanks for your response. I have shared my scenario below. In the context of the Flink job use case, our data source is files, with three new files arriving in the source directory every second. The Flink job is responsible for reading and processing these files. To the best of my knowledge, the S

Re: Clear the State Backends in Flink

2023-11-05 Thread Hangxiang Yu
Hi, Arjun. Do you mean clearing all states stored in a user-defined state ? IIUC, It could be done for Operator state. But it cannot be done for Keyed state for users because every operation for it is binded with a specific key currently. BTW, Could you also share your business scenario ? It could

Clear the State Backends in Flink

2023-10-30 Thread arjun s
Hi team, I'm interested in understanding if there is a method available for clearing the State Backends in Flink. If so, could you please provide guidance on how to accomplish this particular use case? Thanks and regards, Arjun S