Hi Jungtaek, All I see at the moment is that most of the users choose Flink over Spark when continues processing is needed. Unless there is a revolution in this area there is no point to keep maintenance. 2.5 years is lot in bigdata industry. If there will be efforts in this area then happy to join to push this forward...
BR, G On Tue, Sep 15, 2020 at 6:34 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Hi devs, > > It was Spark 2.3 in Feb 2018 which introduced continuous mode in > Structured Streaming as "experimental". > > Now we are here at 2.5 years after its release - I feel it would be a good > time to evaluate the mode, whether the mode has been widely used or not, > and the mode has been making progress, as the mode is "experimental". > > At least from the surface I don't see any active effort for continuous > mode around the community - the last major effort was stateful operation > which was incomplete and I removed that. There were some couples of bug > reports as well as fixes more than a year ago and almost nothing has been > handled. (A trivial bugfix PR has been merged recently but that's all.) The > new features introduced to the Structured Streaming (at least observable > metrics, SS UI) don't apply to continuous mode, and no one made "support > continuous mode" as a hard requirement on passing review in these PRs. > > I have no idea how many companies are using the mode in production (please > add the voice if someone has statistics about this) but I don't see any bug > reports recently, and see only a few questions in SO, which makes me think > about cost on maintenance. > > I know there's a mood to avoid discontinue support as possible, but it > sounds weird to keep something as "unmaintained", especially it's still > "experimental" and main authors are no more active enough to promise > maintenance/improvement on the module. Thoughts? > > Thanks, > Jungtaek Lim (HeartSaVioR) >
