Hi all, I would like to start the discussion for a SPIP, Asynchronous Offset Management in Structured Streaming. The high level summary of the SPIP is that currently in Structured Streaming we perform a couple of offset management operations for progress tracking purposes synchronously on the critical path which can contribute significantly to processing latency. If we were to make these operations asynchronous and less frequent we can dramatically improve latency for certain types of workloads.
I have put together a SPIP to implement such a mechanism. Please take a look! SPIP Jira: https://issues.apache.org/jira/browse/SPARK-39591 SPIP doc: https://docs.google.com/document/d/1iPiI4YoGCM0i61pBjkxcggU57gHKf2jVwD7HWMHgH-Y/edit?usp=sharing Best, Jerry
