Unfortunately your information wouldn't provide any hint that rows in the
state are evicted correctly on watermark advance or there's an unknown bug
which some of the rows in state are silently dropped. I haven't heard of
the case for the latter - probably you'd like to double check it with
We're using Stateful Structured Streaming in Spark 2.4. We are noticing
that when the load on the system is heavy & LOTs of messages are coming in
some of the states disappear with no error message. Any suggestions on how
we can debug this? Any tips for fixing this?
Thanks in advance.
