Basically, it would be great to get the latest code in the flink-connector-files (FLIP-27).
On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <stevenz...@gmail.com> wrote: > Stephan, it will be great if we can also backport the DataStreamUtils > related commits that help with collecting output from unbounded streams. > e.g. > > > https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc > > I tried to copy and paste the code to unblock myself. but it quickly got > into the rabbit hole of more and more code. > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org> wrote: > >> I have started with backporting the source API changes. Some minor >> conflicts to solve, will need a bit more to finish this. >> >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> wrote: >> >> > @Stephan Ewen <se...@apache.org> >> > Are there already plans or ongoing efforts for backporting the list of >> > FLIP-27 changes that you posted? >> > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <tonysong...@gmail.com> >> > wrote: >> > >> >> Hi folks, >> >> >> >> Just to provide some updates concerning the status on the >> >> test instabilities. >> >> >> >> Currently, we have 30 unresolved tickets labeled with `Affects Version` >> >> 1.11.x. >> >> >> >> >> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC >> >> >> >> Among the 30 tickets, 11 of them are: >> >> - Have occured in the recent 3 months >> >> - Not confirmed to be pure testability issues >> >> - Not confirmed to be rare condition cases >> >> >> >> It would be nice if someone familiar with these components can take a >> look >> >> into these issues. >> >> >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6) >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka) >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka) >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka) >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka) >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka) >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka) >> >> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest) >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS) >> >> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet) >> >> - https://issues.apache.org/jira/browse/FLINK-19775 >> >> (SystemProcessingTimeServiceTest) >> >> >> >> Thank you~ >> >> >> >> Xintong Song >> >> >> >> >> >> >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <jingsongl...@gmail.com> >> >> wrote: >> >> >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x. >> >> > >> >> > If possible, that would be great. Many people are looking forward to >> the >> >> > FLIP-27 interface, but they don't want to take the risk to upgrade to >> >> 1.12 >> >> > (And wait 1.12). After all, 1.11 is a relatively stable version. >> >> > >> >> > Best, >> >> > Jingsong >> >> > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org> >> wrote: >> >> > >> >> > > Thanks for starting this. >> >> > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x. >> >> > > >> >> > > There were quite a few changes, and I think we need to cherry-pick >> >> them >> >> > all >> >> > > to not get some inconsistent mix of changes and many merge >> conflicts. >> >> > > I made a list below of what we need to add to "release-1.11". >> >> > > >> >> > > *#### Core Source API Changes to backport (in REVERSE order)* >> >> > > >> >> > > (Use: "git log >> >> > > flink-core/src/main/java/org/apache/flink/api/connector/source") >> >> > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e >> >> > > [FLINK-19492][core] Consolidate Source Events between Source API >> and >> >> > Split >> >> > > Reader API >> >> > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436 >> >> > > [FLINK-19205][core] Add access to configuration and hostname in the >> >> > > SourceReaderContext >> >> > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33 >> >> > > [hotfix][core] Add to Source Enumerator convenience methods to >> assign >> >> > > single split >> >> > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE order)* >> >> > > >> >> > > (Use: "git log flink-connectors/flink-connector-base") >> >> > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()' >> >> > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1 >> >> > > [FLINK-19448][connector base] Explicitly check for un-expected >> >> condition >> >> > > that would leave an inconsistent state >> >> > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e >> >> > > [FLINK-19492][core] Consolidate Source Events between Source API >> and >> >> > Split >> >> > > Reader API >> >> > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c >> >> > > [refactor][tests] Move some source test utils from >> >> flink-connector-base >> >> > to >> >> > > flink-core >> >> > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977 >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in >> >> > > "SplitReader.handleSplitsChanges()" >> >> > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176 >> >> > > [FLINK-19250][connectors] Fix error propagation in connector base >> >> > > (SplitFetcherManager). >> >> > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb >> >> > > [hotfix][connectors] Improve JavaDocs for >> SingleThreadFetcherManager >> >> > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01 >> >> > > [FLINK-19225][connectors] Various small improvements to >> >> SourceReaderBase >> >> > > (part 2) >> >> > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7 >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more >> >> > > elegantly. >> >> > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b >> >> > > [hotfix][tests] Extend test coverage for >> FutureCompletingBlockingQueue >> >> > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631 >> >> > > [FLINK-19245][connectors] Set default capacity for >> >> > > FutureCompletingBlockingQueue. >> >> > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb >> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in >> Base >> >> > > Connector >> >> > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to >> >> availability >> >> > > notifications. >> >> > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92 >> >> > > [refactor][core] Eagerly initialize the FetchTask to support proper >> >> unit >> >> > > testing >> >> > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only >> >> one >> >> > > test into test method >> >> > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e >> >> > > [FLINK-19225][connectors] Various small improvements to >> >> SourceReaderBase >> >> > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1 >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly. >> >> > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903 >> >> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds >> more >> >> > > lightweight. >> >> > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350 >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the >> RecordsWithSplitIds >> >> to >> >> > > support reuse of heavy objects. >> >> > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0 >> >> > > [hotfix][testing] Add a set of parameterizable testing mocks for >> the >> >> > Split >> >> > > Reader API >> >> > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378 >> >> > > [hotfix][connectors] Add RequestSplitEvent to >> 'flink-connector-base' >> >> > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159 >> >> > > [refactor][connectors] Backport of the connector-base exception >> >> handling >> >> > > from the Kafka Connector Pull Request >> >> > > >> >> > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <stevenz...@gmail.com> >> >> wrote: >> >> > > >> >> > > > I would love to see this FLIP-27 source interface improvement [1] >> >> made >> >> > to >> >> > > > 1.11.3. >> >> > > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698 >> >> > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai < >> >> > > tzuli...@apache.org> >> >> > > > wrote: >> >> > > > >> >> > > > > Thanks for the replies so far! >> >> > > > > >> >> > > > > Just to provide a brief update on the status of blockers for >> >> 1.11.3 >> >> > so >> >> > > > far: >> >> > > > > >> >> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717] >> >> > SourceReaderBase.pollNext >> >> > > > may >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws ( >> >> > > > > https://github.com/apache/flink/pull/13776) >> >> > > > > >> >> > > > > *PR opened, reviewed + close to being merged* >> >> > > > > - [FLINK-19741] Timer service should skip restoring from raw >> keyed >> >> > > stream >> >> > > > > if it isn't the writer ( >> >> https://github.com/apache/flink/pull/13761) >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be >> >> > skipping >> >> > > > > unwritten key groups ( >> https://github.com/apache/flink/pull/13772) >> >> > > > > >> >> > > > > *Merged* >> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of >> >> suspended >> >> > > > > ZooKeeper connection >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+ >> >> > > > > >> >> > > > > Right now as it seems, progress is mainly blocked on a reviewer >> >> for >> >> > > > > FLINK-19717. >> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1] >> to >> >> see >> >> > > if >> >> > > > > there are any fixes that should be applied to `release-1.11`. >> >> > > > > >> >> > > > > This is also a reminder, that if there are other blockers that >> we >> >> > need >> >> > > to >> >> > > > > be aware of, or a need to re-establish estimated time for >> getting >> >> > fixes >> >> > > > in >> >> > > > > and delay the RC for 1.11.3, please do let us know! >> >> > > > > >> >> > > > > Cheers, >> >> > > > > Gordon >> >> > > > > >> >> > > > > [1] >> >> > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability >> >> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas < >> >> kklou...@apache.org> >> >> > > > > wrote: >> >> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of >> >> important >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering. >> >> > > > > > >> >> > > > > > Cheers, >> >> > > > > > Kostas >> >> > > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <car...@gmail.com> >> wrote: >> >> > > > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and >> Xintong >> >> for >> >> > > > > > > volunteering as our release managers. >> >> > > > > > > >> >> > > > > > > Best Regards, >> >> > > > > > > Yu >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song < >> >> > tonysong...@gmail.com> >> >> > > > > > wrote: >> >> > > > > > > >> >> > > > > > > > Thanks Gordan for starting this discussion. >> >> > > > > > > > My pleasure to help with the release process. >> >> > > > > > > > >> >> > > > > > > > Thank you~ >> >> > > > > > > > >> >> > > > > > > > Xintong Song >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann < >> >> > > > trohrm...@apache.org >> >> > > > > > >> >> > > > > > > > wrote: >> >> > > > > > > > >> >> > > > > > > > > Thanks for starting this discussion Gordon. There are >> over >> >> > 100 >> >> > > > > issues >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish >> 1.11.3 >> >> > > > release. >> >> > > > > > Thanks >> >> > > > > > > > > for volunteering as our release managers Gordon and >> >> Xintong! >> >> > > > > > > > > >> >> > > > > > > > > Cheers, >> >> > > > > > > > > Till >> >> > > > > > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai < >> >> > > > > > tzuli...@apache.org >> >> > > > > > > > > >> >> > > > > > > > > wrote: >> >> > > > > > > > > >> >> > > > > > > > >> Hi, >> >> > > > > > > > >> >> >> > > > > > > > >> Xintong and I would like to start a discussion for >> >> releasing >> >> > > > Flink >> >> > > > > > > > 1.11.3 >> >> > > > > > > > >> soon. >> >> > > > > > > > >> >> >> > > > > > > > >> It seems like we already have a few pressing issues >> that >> >> > needs >> >> > > > to >> >> > > > > be >> >> > > > > > > > >> included in a new hotfix release: >> >> > > > > > > > >> >> >> > > > > > > > >> - Heap-based timers’ restore behaviour is causing a >> >> > > critical >> >> > > > > > recovery >> >> > > > > > > > >> issue for StateFun [1] [2] [3]. >> >> > > > > > > > >> - There are several robustness issues for the >> FLIP-27 >> >> new >> >> > > > > source >> >> > > > > > API, >> >> > > > > > > > >> such as [4]. We already have some users using the >> >> FLIP-27 >> >> > > API >> >> > > > > > with >> >> > > > > > > > >> 1.11.x, >> >> > > > > > > > >> so it would be important to get those fixes in for >> >> 1.11.x >> >> > > as >> >> > > > > > well. >> >> > > > > > > > >> >> >> > > > > > > > >> Apart from the issues that are already marked as >> blocker >> >> for >> >> > > > > 1.11.3 >> >> > > > > > in >> >> > > > > > > > our >> >> > > > > > > > >> JIRA [5], please let us know in this thread if there >> is >> >> > > already >> >> > > > > > ongoing >> >> > > > > > > > >> work for other important fixes that we should try to >> >> > include. >> >> > > > > > > > >> >> >> > > > > > > > >> Xintong and I would like to volunteer for managing >> this >> >> > > release, >> >> > > > > and >> >> > > > > > > > will >> >> > > > > > > > >> try to communicate the priority of pending blockers >> over >> >> the >> >> > > > next >> >> > > > > > few >> >> > > > > > > > >> days. >> >> > > > > > > > >> Since the aforementioned issues are quite critical, >> we’d >> >> > like >> >> > > to >> >> > > > > aim >> >> > > > > > > > >> for a *feature >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start >> the >> >> > > > release >> >> > > > > > voting >> >> > > > > > > > >> process the week after. >> >> > > > > > > > >> If that is too short of a notice and you might need >> more >> >> > time, >> >> > > > > > please >> >> > > > > > > > let >> >> > > > > > > > >> us know! >> >> > > > > > > > >> >> >> > > > > > > > >> Cheers, >> >> > > > > > > > >> Gordon >> >> > > > > > > > >> >> >> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692 >> >> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741 >> >> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748 >> >> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717 >> >> > > > > > > > >> [5] >> >> > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3 >> >> > > > > > > > >> >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> > >> >> > -- >> >> > Best, Jingsong Lee >> >> > >> >> >> > >> >
