Hi Nico, Thanks for getting back to us. We are using Flink 1.14.0 and we are using RocksDB. We currently are using the default memory settings. We'll look into increasing our managed memory fraction to 0.6 and see what happens.
Do writes to ValueStates/MapStates have a direct on churn of the Flink State or is the data buffered in between? Thanks. Jai On Wed, Feb 23, 2022 at 12:55 AM Nicolaus Weidner < nicolaus.weid...@ververica.com> wrote: > Hi Jai, > > On Tue, Feb 22, 2022 at 9:19 PM Jai Patel <jai.pa...@cloudkitchens.com> > wrote: > >> It seems like the errors are similar to those discussed here: >> - https://issues.apache.org/jira/browse/FLINK-14316 >> - https://cdmana.com/2020/11/20201116104527255b.html >> > > I couldn't find any other existing issue apart from the one you already > linked. Just to be sure: Which Flink version are you using? Is it one where > the reported issue is fixed? > > As for the issue itself, it looks like the connection between JobManager > and TaskManager was lost, though I can't tell why. Do you have full logs > from JobManager and TaskManager surrounding such an incident? > > >> When looking at the memory structure it looks like all memory is below >> 100% except for managed memory. We have 9.1GB of managed memory for each >> of our 6 task managers and I estimate that our total Flink State is 600GB. >> Is it okay for run with that little memory for that much State? >> > > Are you using RocksDB or HashMap state backend [1]? I assume it's RocksDB, > since with HashMapStateBackend, state size is limited by memory size (and > you are way above that). Did you check out the memory configuration > recommendations in the docs [2, 3]? > In principle (assuming RocksDB is used), I don't think the amount of > memory should be an issue (at least it shouldn't cause crashes). The logs > would help to understand what's happening. > > Best, > Nico > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/ > [2] > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_tuning/#configure-memory-for-state-backends > [3] > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/large_state_tuning/#tuning-rocksdb-memory >
