Hi, Banu.
In your description, the only stateful node in the entire job is the sliding
window. The fluctuations in the size of the state will be affected by the
window size (2 seconds). Theoretically, after the window is triggered, its
state can be immediately cleared, thus its size will be reduced. As for whether
the state cleanup in RocksDB is lazy or eager, I'm not too familiar with that.
--
Best！
Xuyang
在 2024-08-01 14:56:51，"banu priya" <[email protected]> 写道：
Hi Xuyang,
Thanks a lot for the reply.
I am using standard alone flink. Now I have found a working solution.
I have one more query regarding checkpoints,
I have a flink streaming job with RMQ Source,keyby, filters, tumbling window
(fires every 2s, uses processing time), RMQ Sink.
Rocks db backend with incremental check point is enabled for every 10s.
Check point size is growing continuously and through put is also increasing.
10 unique events I will get from my source per second. In one hour I can get,
600 unique events and then same keys are continuing.
I have not tuned any of rocks db parameters. What I see is , after sometime
check point size is decreasing to some point and then continuously increasing.
I am not able to understand if it is expected or not.
Could you please help me on this by sharing your thoughts?
Thanks
Banu
On Wed, 31 Jul, 2024, 7:29 am Xuyang, <[email protected]> wrote:
Hi, Banu.
Could you check whether the "Configuration" icon under the "Task Managers" and
"Job Manager" buttons on the left side of the Flink-UI
shows that the currently effective flink conf includes these JVM changes? I
suspect that you are using a session cluster mode, where
changes to the yaml file only affect the job, not the cluster.
--
Best！
Xuyang
At 2024-07-30 16:23:24, "banu priya" <[email protected]> wrote:
Hi All,
I am trying to change GC settings of Taskmanager. I am using flink 1.18 and
Java 8. Jave 8 uses parallel GC by default. I need to change it to use G1GC.
In flink-conf.yaml, I passed the property ( refer attached image line no. 357).
But it is not reflecting. What am I doing wrong?
Thanks
Banu
