Hi Nico, Wanted to close the loop here. We did end up find a number of problems in our code: 1. Our operator was slow. It was iterating over several large Protobufs in a MapState then filtering it down to 1. We were able to identify that one up-front and significantly improve the runtime of the operator. 2. We did increase the total memory and raised the managed memory fraction from 40% to 70%.
Both solutions were needed in order to bring us the stability we were looking for. Since then, everything has been running great. Thanks. Jai On Thu, Feb 24, 2022 at 2:42 AM Nicolaus Weidner < nicolaus.weid...@ververica.com> wrote: > Hi Jai, > > Do writes to ValueStates/MapStates have a direct on churn of the Flink >> State or is the data buffered in between? >> > > Writes to keyed state go directly to RocksDB. So there shouldn't be any > memory issues with buffers overflowing or similar. In general, more memory > should increase performance (larger cache sizes before having to write to > disk), but less memory shouldn't cause crashes. > > Since the errors you encountered are not that specific, can you provide > full logs surrounding such incidents? There is not much to go on without > further info. > > Best, > Nico > >>
