Unfortunately your information wouldn't provide any hint that rows in the state are evicted correctly on watermark advance or there's an unknown bug which some of the rows in state are silently dropped. I haven't heard of the case for the latter - probably you'd like to double check it with focusing on watermark advance. If the case is turned out to be the latter, you'll probably need to deal with Spark code to inject the debug log.
On Fri, Oct 23, 2020 at 3:12 PM Eric Beabes <mailinglist...@gmail.com> wrote: > We're using Stateful Structured Streaming in Spark 2.4. We are noticing > that when the load on the system is heavy & LOTs of messages are coming in > some of the states disappear with no error message. Any suggestions on how > we can debug this? Any tips for fixing this? > > Thanks in advance. >
