Yeah, the main intention of this thread is to collect interest on possible feature list for structured streaming. From what I can see in Spark community, most of the discussions as well as contributions are for SQL, and I'd wish to see similar activeness / efforts on structured streaming. (Unfortunately there's less effort to review others' works - design doc as well as pull request - most of efforts looks like being spent to their own works.)
I respect the role of PMC member, so the final decision would be up to PMC members, but contributors as well as end users could show the interest as well as discuss about requirements on SPIP, which could be a good background to persuade PMC members. Before going into the deep I guess we could use this thread to discuss about possible use cases, and if we would like to move forward to individual thread we could initiate (or resurrect) its discussion thread. For queryable state, at least there seems no workaround in Spark to provide similar thing, especially state is getting bigger. I may have some concerns on the details, but I'll add my thought on the discussion thread. - Jungtaek Lim (HeartSaVioR) 2018년 10월 22일 (월) 오전 1:15, Stavros Kontopoulos < stavros.kontopou...@lightbend.com>님이 작성: > Hi Jungtaek, > > I just tried to start the discussion in the dev list along time ago. > I enumerated some uses cases as Michael proposed here > <http://mail-archives.apache.org/mod_mbox/spark-dev/201712.mbox/%3CCACTd3c_snT=y4r9vod+ebty1fdgtqsxzgjgubox-k8araur...@mail.gmail.com%3E>. > The discussion didn't go further. > > If people find it useful we should start discussing it in detail again. > > Stavros > > On Sun, Oct 21, 2018 at 4:54 PM, Jungtaek Lim <kabh...@gmail.com> wrote: > >> Stavros, if my memory is right, you were trying to drive queryable state, >> right? >> >> Could you summary the progress and the reason why the progress got >> stopped? >> >> 2018년 10월 21일 (일) 오후 10:27, Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com>님이 작성: >> >>> That is a very interesting list thanks. I could create a design doc as >>> a starting pointing for discussion if this is a feature we would like to >>> have. >>> >>> Regards, >>> Stavros >>> >>> On Sun, Oct 21, 2018 at 3:04 PM, JackyLee <qcsd2...@163.com> wrote: >>> >>>> Thanks for raising them. >>>> >>>> FYI, I believe this open issues could also be considered: >>>> >>>> https://issues.apache.org/jira/browse/SPARK-24630 >>>> <https://issues.apache.org/jira/browse/SPARK-24630> >>>> >>>> An new ability to express Struct Streaming on pure SQL. >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> >>> >>> >>> >>> > >
