Probably it would depend on the meaning of "experimental". My understanding of "experimental" is more likely "incubation", which may be graduated finally, or may be retired.
To be clear, I'm evaluating the continuous mode as "candidate to retire", unless there are actual use cases in production and at least a couple of community members volunteer to maintain it. As far as I see the activity in a year, there's no interest for the continuous mode in community members. I can refer to at least three PRs which suffered to find reviewers (around 1 year) and closed on inactivity. No improvements/bug fixes except trivials. It doesn't seem to get some traction - few questions in SO, a few posts in google search results which were all posted around the date when continuous mode was introduced. Though I would be convinced if someone could provide meaningful numbers of actual use cases. If the answer really has to be taken between un-experimental or not (which says retirement is not an option), I'd rather vote to leave as experimental, so I just keep forgetting about it. Actually it bothers sometimes even if the change is done in micro-batch side (so that's not a zero cost to maintain), but still better than officially supporting it. On Tue, Sep 15, 2020 at 9:08 PM Sean Owen <sro...@gmail.com> wrote: > If you're suggesting making it un-Experimental, probably yes, as it is > de facto not going to change much I expect. > If you're saying remove it, probably not? I don't see that it's > anywhere near deprecated, and not sure it's unmaintained - obviously > tests etc still have to keep passing. > > On Mon, Sep 14, 2020 at 11:34 PM Jungtaek Lim > <kabhwan.opensou...@gmail.com> wrote: > > > > Hi devs, > > > > It was Spark 2.3 in Feb 2018 which introduced continuous mode in > Structured Streaming as "experimental". > > > > Now we are here at 2.5 years after its release - I feel it would be a > good time to evaluate the mode, whether the mode has been widely used or > not, and the mode has been making progress, as the mode is "experimental". > > > > At least from the surface I don't see any active effort for continuous > mode around the community - the last major effort was stateful operation > which was incomplete and I removed that. There were some couples of bug > reports as well as fixes more than a year ago and almost nothing has been > handled. (A trivial bugfix PR has been merged recently but that's all.) The > new features introduced to the Structured Streaming (at least observable > metrics, SS UI) don't apply to continuous mode, and no one made "support > continuous mode" as a hard requirement on passing review in these PRs. > > > > I have no idea how many companies are using the mode in production > (please add the voice if someone has statistics about this) but I don't see > any bug reports recently, and see only a few questions in SO, which makes > me think about cost on maintenance. > > > > I know there's a mood to avoid discontinue support as possible, but it > sounds weird to keep something as "unmaintained", especially it's still > "experimental" and main authors are no more active enough to promise > maintenance/improvement on the module. Thoughts? > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) >
