Hi amenreet, I think there are two ways to clean up state data in the flink job automatically:
1. State TTL. You can configure the ttl [1] for state according to your requirements, and flink job will clean the data when they are out of date. For flink SQL jobs you can set a global ttl for all operators and set each operator ttl like [2] 2. If you use rocksdb statebacked in your job, flink will delete the sst files if they are not referenced by checkpoint any more. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#state-time-to-live-ttl [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-280%3A+Introduce+EXPLAIN+PLAN_ADVICE+to+provide+SQL+advice Best, Shammon FY On Tue, Jul 11, 2023 at 3:20 PM amenreet sodhi <amenso...@gmail.com> wrote: > Hi Team, > > I wanted to confirm, the local state which TM stores in the directory > either we provide through config or default i.e. /tmp folder, does it clear > itself from time to time or the size just keeps on increasing? > > Thanks > Regards > Amenreet Singh Sodhi > >
