OK, good to know, and that all makes sense. Thanks for clearing up my concern.
One of great things about Spark is, as you pointed out, that improvements to core components benefit multiple features at once. On Mon, Jan 14, 2019 at 8:36 PM Reynold Xin <r...@databricks.com> wrote: > BTW the largest change to SS right now is probably the entire data source > API v2 effort, which aims to unify streaming and batch from data source > perspective, and provide a reliable, expressive source/sink API. > > > On Mon, Jan 14, 2019 at 5:34 PM, Reynold Xin <r...@databricks.com> wrote: > >> There are a few things to keep in mind: >> >> 1. Structured Streaming isn't an independent project. It actually (by >> design) depends on all the rest of Spark SQL, and virtually all >> improvements to Spark SQL benefit Structured Streaming. >> >> 2. The project as far as I can tell is relatively mature for core ETL and >> incremental processing purpose. I interact with a lot of users using it >> everyday. We can always expand the use cases and add more, but that also >> adds maintenance burden. In any case, it'd be good to get some activity >> here. >> >> >> >> >> On Mon, Jan 14, 2019 at 5:11 PM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >>> As an observer, this thread is interesting and concerning. Is there an >>> emerging consensus that Structured Streaming is somehow not relevant >>> anymore? Or is it just that folks consider it "complete enough"? >>> >>> Structured Streaming was billed as the replacement to DStreams. If >>> committers, generally speaking, have lost interest in Structured Streaming, >>> does that mean the Apache Spark project is somehow no longer aiming to >>> provide a "first-class" solution to the problem of stream processing? >>> >>> On Mon, Jan 14, 2019 at 3:43 PM Jungtaek Lim <kabh...@gmail.com> wrote: >>> >>>> Cody, I guess I already addressed your comments in the PR (#22138). The >>>> approach was changed to address your concern, and after that Gabor helped >>>> to review the PR. Please take a look again when you have time to get into. >>>> >>>> 2019년 1월 15일 (화) 오전 1:01, Cody Koeninger <c...@koeninger.org>님이 작성: >>>> >>>>> I feel like I've already said my piece on >>>>> https://github.com/apache/spark/pull/22138 let me know if you have >>>>> more questions. >>>>> >>>>> As for SS in general, I don't have a production SS deployment, so I'm >>>>> less comfortable with reviewing large changes to it. But if no other >>>>> committers are working on it... >>>>> >>>>> On Sun, Jan 13, 2019 at 5:19 PM Sean Owen <sro...@gmail.com> wrote: >>>>> > >>>>> > Yes you're preaching to the choir here. SS does seem somewhat >>>>> > abandoned by those that have worked on it. I have also been at times >>>>> > frustrated that some areas fall into this pattern. >>>>> > >>>>> > There isn't a way to make people work on it, and I personally am not >>>>> > interested in it nor have a background in SS. >>>>> > >>>>> > I did leave some comments on your PR and will see if we can get >>>>> > comfortable with merging it, as I presume you are pretty >>>>> knowledgeable >>>>> > about the change. >>>>> > >>>>> > On Sun, Jan 13, 2019 at 4:55 PM Jungtaek Lim <kabh...@gmail.com> >>>>> wrote: >>>>> > > >>>>> > > Sean, this is actually a fail-back on pinging committers. I know >>>>> who can review and merge in SS area, and pinged to them, didn't work. Even >>>>> there's a PR which approach was encouraged by committer and reviewed the >>>>> first phase, and no review. >>>>> > > >>>>> > > That's not the first time I have faced the situation, and I used >>>>> the fail-back approach at that time. (You can see there was no response >>>>> even in the mail thread.) Not sure which approach worked. >>>>> > > >>>>> https://lists.apache.org/thread.html/c61f32249949b1ff1b265c1a7148c2ea7eda08891e3016fb24008561@%3Cdev.spark.apache.org%3E >>>>> > > >>>>> > > I've observed that only (critical) bugfixes are being reviewed and >>>>> merged in time for SS area. For other stuffs like new features and >>>>> improvements, both discussions and PRs were pretty less popular from >>>>> committers: though there was even participation/approve from non-committer >>>>> community. I don't think SS is the thing to be turned into maintenance. >>>>> > > >>>>> > > I guess PMC members should try to resolve such situation, as it >>>>> will (slowly and quietly) make some issues like contributors leaving, >>>>> module stopped growing up, etc.. The problem will grow up like a snowball: >>>>> getting bigger and bigger. I don't mind if there's no interest on both >>>>> contributors and committers for such module, but SS is not. Maybe either >>>>> other committers who weren't familiar with should try to get familiar and >>>>> cover the area, or the area needs more committers. >>>>> > > >>>>> > > -Jungtaek Lim (HeartSaVioR) >>>>> > > >>>>> > > 2019년 1월 13일 (일) 오후 11:37, Sean Owen <sro...@gmail.com>님이 작성: >>>>> > >> >>>>> > >> Jungtaek, the best strategy is to find who wrote the code you are >>>>> > >> modifying (use Github history or git blame) and ping them >>>>> directly on >>>>> > >> the PR. I don't know this code well myself. >>>>> > >> It also helps if you can address why the functionality is >>>>> important, >>>>> > >> and describe compatibility implications. >>>>> > >> >>>>> > >> Most PRs are not merged, note. Not commenting on this particular >>>>> one, >>>>> > >> but it's not a 'bug' if it's not being merged. >>>>> > >> >>>>> > >> On Sun, Jan 13, 2019 at 12:29 AM Jungtaek Lim <kabh...@gmail.com> >>>>> wrote: >>>>> > >> > >>>>> > >> > I'm sorry but let me remind this, as non-SS PRs are being >>>>> reviewed accordingly, whereas many of SS PRs (regardless of who create) >>>>> are >>>>> still not reviewed and merged in time. >>>>> > >> > >>>>> > >> > 2019년 1월 3일 (목) 오전 7:57, Jungtaek Lim <kabh...@gmail.com>님이 작성: >>>>> > >> >> >>>>> > >> >> Spark devs, happy new year! >>>>> > >> >> >>>>> > >> >> I would like to remind this kindly, since there was actually >>>>> no review after initiating the thread. >>>>> > >> >> >>>>> > >> >> Thanks, >>>>> > >> >> Jungtaek Lim (HeartSaVioR) >>>>> > >> >> >>>>> > >> >> 2018년 12월 12일 (수) 오후 11:12, Vaclav Kosar < >>>>> ad...@vaclavkosar.com>님이 작성: >>>>> > >> >>> >>>>> > >> >>> I am also waiting for any finalization of my PR [3]. I seems >>>>> that SS PRs are not being reviewed much these days. >>>>> > >> >>> >>>>> > >> >>> [3] https://github.com/apache/spark/pull/21919 >>>>> > >> >>> >>>>> > >> >>> >>>>> > >> >>> On 12. 12. 18 14:37, Dongjin Lee wrote: >>>>> > >> >>> >>>>> > >> >>> If it is possible, could you review my PR on Kafka's header >>>>> functionality[^1] also? It was added in Kafka 0.11.0.0 but still not >>>>> supported in Spark. >>>>> > >> >>> >>>>> > >> >>> Thanks, >>>>> > >> >>> Dongjin >>>>> > >> >>> >>>>> > >> >>> [^1]: https://github.com/apache/spark/pull/22282 >>>>> > >> >>> [^2]: https://issues.apache.org/jira/browse/KAFKA-4208 >>>>> > >> >>> >>>>> > >> >>> On Wed, Dec 12, 2018 at 6:43 PM Jungtaek Lim < >>>>> kabh...@gmail.com> wrote: >>>>> > >> >>>> >>>>> > >> >>>> Hi devs, >>>>> > >> >>>> >>>>> > >> >>>> Would I kindly ask for reviewing on PRs for Structured >>>>> Streaming? I have 5 open pull requests on SS side [1] (earliest PR was >>>>> opened around 4 months so far), and there looks like couple of PR for >>>>> others [2] which looks good to be reviewed, too. >>>>> > >> >>>> >>>>> > >> >>>> Thanks in advance, >>>>> > >> >>>> Jungtaek Lim (HeartSaVioR) >>>>> > >> >>>> >>>>> > >> >>>> 1. >>>>> https://github.com/apache/spark/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+author%3AHeartSaVioR+%5BSS%5D >>>>> > >> >>>> 2. >>>>> https://github.com/apache/spark/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+%5BSS%5D+ >>>>> > >> >>>> >>>>> > >> >>> >>>>> > >> >>> >>>>> > >> >>> -- >>>>> > >> >>> Dongjin Lee >>>>> > >> >>> >>>>> > >> >>> A hitchhiker in the mathematical world. >>>>> > >> >>> >>>>> > >> >>> github: github.com/dongjinleekr >>>>> > >> >>> linkedin: kr.linkedin.com/in/dongjinleekr >>>>> > >> >>> speakerdeck: speakerdeck.com/dongjin >>>>> > >>>>> > --------------------------------------------------------------------- >>>>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>> > >>>> >>>> >
