Hi dev, I would like to start a discussion on "Structured Streaming - Arbitrary State API v2". This proposal aims to address a bunch of limitations we see today using mapGroupsWithState/flatMapGroupsWithState operator. The detailed set of limitations is described in the SPIP doc.
We propose to support various features such as multiple state variables (flexible data modeling), composite types, enhanced timer functionality, support for chaining operators after new operator, handling initial state along with state data source, schema evolution etc This will allow users to write more powerful streaming state management logic primarily used in operational use-cases. Other built-in stateful operators could also benefit from such changes in the future. JIRA: https://issues.apache.org/jira/browse/SPARK-45939 SPIP: https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig/edit?usp=sharing Design Doc: https://docs.google.com/document/d/1QjZmNZ-fHBeeCYKninySDIoOEWfX6EmqXs2lK097u9o/edit?usp=sharing cc - @Jungtaek Lim <kabhwan.opensou...@gmail.com> who has graciously agreed to be the shepherd for this project Looking forward to your feedback ! Thanks, Anish
