Hi devs, It was Spark 2.3 in Feb 2018 which introduced continuous mode in Structured Streaming as "experimental".
Now we are here at 2.5 years after its release - I feel it would be a good time to evaluate the mode, whether the mode has been widely used or not, and the mode has been making progress, as the mode is "experimental". At least from the surface I don't see any active effort for continuous mode around the community - the last major effort was stateful operation which was incomplete and I removed that. There were some couples of bug reports as well as fixes more than a year ago and almost nothing has been handled. (A trivial bugfix PR has been merged recently but that's all.) The new features introduced to the Structured Streaming (at least observable metrics, SS UI) don't apply to continuous mode, and no one made "support continuous mode" as a hard requirement on passing review in these PRs. I have no idea how many companies are using the mode in production (please add the voice if someone has statistics about this) but I don't see any bug reports recently, and see only a few questions in SO, which makes me think about cost on maintenance. I know there's a mood to avoid discontinue support as possible, but it sounds weird to keep something as "unmaintained", especially it's still "experimental" and main authors are no more active enough to promise maintenance/improvement on the module. Thoughts? Thanks, Jungtaek Lim (HeartSaVioR)
