Hi, Arjun. Are you using DataStream api? Maybe you can refer this doc[1] to set an operator-level state TTL to let the state cleared automatically.
Back to your scene, do you use state explicitl in some operators to store file names? If not and using a DataStream api, and I'm not mistaken, Flink will not store the state actively. [1]https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#state-time-to-live-ttl -- Best！ Xuyang 在 2023-11-06 21:04:11，"arjun s" <arjunjoice...@gmail.com> 写道： Thanks for your response. I have shared my scenario below. In the context of the Flink job use case, our data source is files, with three new files arriving in the source directory every second. The Flink job is responsible for reading and processing these files. To the best of my knowledge, the State Backend maintains a record of the file names that have been processed by the Flink job. Please correct me if I'm mistaken. If the State Backend does indeed store the file names, I'm concerned about potential memory-related issues during long-term operation of the Flink job. If such issues may arise, what are the recommended best practices for managing this scenario? Thanks and regards, Arjun On Mon, 6 Nov 2023 at 09:24, Hangxiang Yu <master...@gmail.com> wrote: Hi, Arjun. Do you mean clearing all states stored in a user-defined state ? IIUC, It could be done for Operator state. But it cannot be done for Keyed state for users because every operation for it is binded with a specific key currently. BTW, Could you also share your business scenario ? It could help us to rethink the interface. Thanks! On Tue, Oct 31, 2023 at 12:02 AM arjun s <arjunjoice...@gmail.com> wrote: Hi team, I'm interested in understanding if there is a method available for clearing the State Backends in Flink. If so, could you please provide guidance on how to accomplish this particular use case? Thanks and regards, Arjun S -- Best, Hangxiang.
