Just want to say thank you to all the active SS contributors. I saw many great features/improvements in Streaming have been merged and will be available in the upcoming 3.1 release.
- Cache fetched list of files beyond maxFilesPerTrigger as unread file (SPARK-32568) - Streamline the logic on file stream source and sink metadata log (SPARK-30462) - Add DataStreamReader.table API (SPARK-32885) - Add DataStreamWriter.saveAsTable API (SPARK-32896) - Left semi stream-stream join (SPARK-32862) - Introduce schema validation for streaming state store (SPARK-31894) - Support to use a different compression codec in state store (SPARK-33263) - Kafka connector infinite wait because metadata never updated (SPARK-28367) - Upgrade Kafka to 2.6.0 (SPARK-32568) - Pagination support for Structured Streaming UI pages (SPARK-31642, SPARK-30119) - State information in Structured Streaming UI (SPARK-33223) Structured Streaming UI support in Spark History Server is another great usability feature: https://github.com/apache/spark/pull/28781 Hopefully, this can be part of 3.1 release. Go Spark! Xiao Ryan Blue <rb...@netflix.com.invalid> 于2020年11月30日周一 上午11:35写道： > Jungtaek, > > If there are contributors that you trust for reviews, then please let PMC > members know so they can be considered. I agree that is the best solution. > > If there aren't contributors that the PMC wants to add as committers, then > I suggest agreeing on a temporary exception to help make progress in this > area and give contributors more opportunities to develop. Something like > this: for the next 6 months, contributions from committers to SS can be > committed without a committer +1 if they are reviewed by at least one > contributor (and have no dissent from committers, of course). Then after > the period expires, we would ideally have new people ready to be added as > committers. > > That would need to be voted on, but I think it is a reasonable step to > help resuscitate Spark streaming. > > On Fri, Nov 27, 2020 at 7:15 PM Sean Owen <sro...@gmail.com> wrote: > >> I don't know the code well, but those look minor and straightforward. >> They have reviews from the two most knowledgeable people in this area. I >> don't think you need to block for 6 months after proactively seeking all >> likely reviewers - I'm saying that's the resolution to this type of >> situation (too). >> >> On Fri, Nov 27, 2020 at 8:55 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Btw, there are two more PRs which got LGTM by a SS contributor but fail >>> to get attention from committers. They're 6+ months old. Could you help >>> reviewing this as well, or do you all think 6 months of time range + LGTM >>> from an SS contributor is enough to go ahead? >>> >>> https://github.com/apache/spark/pull/27649 >>> https://github.com/apache/spark/pull/28363 >>> >>> These are under 100 lines of changes per each, and not invasive. >>> >> > > -- > Ryan Blue > Software Engineer > Netflix >
