Thanks for starting this. +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
There were quite a few changes, and I think we need to cherry-pick them all to not get some inconsistent mix of changes and many merge conflicts. I made a list below of what we need to add to "release-1.11". *#### Core Source API Changes to backport (in REVERSE order)* (Use: "git log flink-core/src/main/java/org/apache/flink/api/connector/source") commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e [FLINK-19492][core] Consolidate Source Events between Source API and Split Reader API commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436 [FLINK-19205][core] Add access to configuration and hostname in the SourceReaderContext commit 723e1790744ddc78d19c7c978442af1383f38d33 [hotfix][core] Add to Source Enumerator convenience methods to assign single split *#### Connector Base Changes to Backport (in REVERSE order)* (Use: "git log flink-connectors/flink-connector-base") commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d [FLINK-19427][FLINK-19489][tests] Fix test conditions for 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()' commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1 [FLINK-19448][connector base] Explicitly check for un-expected condition that would leave an inconsistent state commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e [FLINK-19492][core] Consolidate Source Events between Source API and Split Reader API commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c [refactor][tests] Move some source test utils from flink-connector-base to flink-core commit ee5c4c211c35c70d28252363bbc8400453609977 [FLINK-19251][connectors] Avoid confusing queue handling in "SplitReader.handleSplitsChanges()" commit 5abef56b2bf85bcac786f6b16b6899b6cced7176 [FLINK-19250][connectors] Fix error propagation in connector base (SplitFetcherManager). commit 8fcca837c55a9216595ee4c03038b52747098dbb [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager commit 4700bb5dde3303cbe98882f6beb7379425717b01 [FLINK-19225][connectors] Various small improvements to SourceReaderBase (part 2) commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7 [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more elegantly. commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue commit cef8a587d7fd2fe64cc644da5ed095d82e46f631 [FLINK-19245][connectors] Set default capacity for FutureCompletingBlockingQueue. commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb [FLINK-19223][connectors] Simplify Availability Future Model in Base Connector commit 511857049ba30c8ff0ee56da551fa4a479dc583e [FLINK-18128][connectors] Ensure idle split fetchers lead to availability notifications. commit a8206467af0830dcb89623ea068b5ca3b3450c92 [refactor][core] Eagerly initialize the FetchTask to support proper unit testing commit 3b2f54bcb437f98e6137c904045cc51072b5c06b [hotfix][tests] Move constants in SplitFetcherTest relevant to only one test into test method commit d7625760a75a508bf05bcddc380bb4d62ee1743e [FLINK-19225][connectors] Various small improvements to SourceReaderBase commit a5b0d3297748c1be47ad579a88f24df2255a8df1 [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly. commit f42a3ebc3e81a034b7221a803c153636fef34903 [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more lightweight. commit e3d273de822b085183d09b275a445879ff94b350 [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to support reuse of heavy objects. commit 8ebc464c2520453a70001cd712abc8dee6ee89e0 [hotfix][testing] Add a set of parameterizable testing mocks for the Split Reader API commit 930a07438be1185388d7150640f294dfe2a5d378 [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base' commit e945ce8a933bc378844782f784ca473c767ca159 [refactor][connectors] Backport of the connector-base exception handling from the Kafka Connector Pull Request On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <stevenz...@gmail.com> wrote: > I would love to see this FLIP-27 source interface improvement [1] made to > 1.11.3. > > [1] https://issues.apache.org/jira/browse/FLINK-19698 > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > > > Thanks for the replies so far! > > > > Just to provide a brief update on the status of blockers for 1.11.3 so > far: > > > > > > *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext > may > > return END_OF_INPUT if SplitReader.fetch throws ( > > https://github.com/apache/flink/pull/13776) > > > > *PR opened, reviewed + close to being merged* > > - [FLINK-19741] Timer service should skip restoring from raw keyed stream > > if it isn't the writer (https://github.com/apache/flink/pull/13761) > > - [FLINK-19748] Raw keyed stream key group iterator should be skipping > > unwritten key groups (https://github.com/apache/flink/pull/13772) > > > > *Merged* > > - [FLINK-19154] Application mode deletes HA data in case of suspended > > ZooKeeper connection > > - [FLINK-19569] Upgrade ICU4J to 67.1+ > > > > Right now as it seems, progress is mainly blocked on a reviewer for > > FLINK-19717. > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if > > there are any fixes that should be applied to `release-1.11`. > > > > This is also a reminder, that if there are other blockers that we need to > > be aware of, or a need to re-establish estimated time for getting fixes > in > > and delay the RC for 1.11.3, please do let us know! > > > > Cheers, > > Gordon > > > > [1] > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <kklou...@apache.org> > > wrote: > > > > > +1 for releasing Flink 1.11.3 as it contains a number of important > > > fixes and thanks Gordon and Xintong for volunteering. > > > > > > Cheers, > > > Kostas > > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <car...@gmail.com> wrote: > > > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for > > > > volunteering as our release managers. > > > > > > > > Best Regards, > > > > Yu > > > > > > > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <tonysong...@gmail.com> > > > wrote: > > > > > > > > > Thanks Gordan for starting this discussion. > > > > > My pleasure to help with the release process. > > > > > > > > > > Thank you~ > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann < > trohrm...@apache.org > > > > > > > > wrote: > > > > > > > > > > > Thanks for starting this discussion Gordon. There are over 100 > > issues > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 > release. > > > Thanks > > > > > > for volunteering as our release managers Gordon and Xintong! > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai < > > > tzuli...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > >> Hi, > > > > > >> > > > > > >> Xintong and I would like to start a discussion for releasing > Flink > > > > > 1.11.3 > > > > > >> soon. > > > > > >> > > > > > >> It seems like we already have a few pressing issues that needs > to > > be > > > > > >> included in a new hotfix release: > > > > > >> > > > > > >> - Heap-based timers’ restore behaviour is causing a critical > > > recovery > > > > > >> issue for StateFun [1] [2] [3]. > > > > > >> - There are several robustness issues for the FLIP-27 new > > source > > > API, > > > > > >> such as [4]. We already have some users using the FLIP-27 API > > > with > > > > > >> 1.11.x, > > > > > >> so it would be important to get those fixes in for 1.11.x as > > > well. > > > > > >> > > > > > >> Apart from the issues that are already marked as blocker for > > 1.11.3 > > > in > > > > > our > > > > > >> JIRA [5], please let us know in this thread if there is already > > > ongoing > > > > > >> work for other important fixes that we should try to include. > > > > > >> > > > > > >> Xintong and I would like to volunteer for managing this release, > > and > > > > > will > > > > > >> try to communicate the priority of pending blockers over the > next > > > few > > > > > >> days. > > > > > >> Since the aforementioned issues are quite critical, we’d like to > > aim > > > > > >> for a *feature > > > > > >> freeze by the end of next week (Oct. 30th)* and start the > release > > > voting > > > > > >> process the week after. > > > > > >> If that is too short of a notice and you might need more time, > > > please > > > > > let > > > > > >> us know! > > > > > >> > > > > > >> Cheers, > > > > > >> Gordon > > > > > >> > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692 > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741 > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748 > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717 > > > > > >> [5] > > > > > >> > > > > > >> > > > > > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3 > > > > > >> > > > > > > > > > > > > > > > > >
