Hi Zhu Zhu, Replenish its[1] influence: For HiveStreamingSink & FileSystemSink in Table/SQL, partition commit make partition visible for downstream Hive/Spark engines. But due to FLINK-19166, will lose some partitions to commit after Job failover in some cases, especially for short partitions. In the user's opinion, the data is lost, which does not conform to exactly-once.
[1]https://issues.apache.org/jira/browse/FLINK-19166 [2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/filesystem.html#partition-commit Best, Jingsong On Wed, Sep 9, 2020 at 11:08 AM Jingsong Li <jingsongl...@gmail.com> wrote: > Hi Zhu Zhu, > > Add a new blocker: https://issues.apache.org/jira/browse/FLINK-19166 > > Will fix it soon. > > Best, > Jingsong > > On Tue, Sep 8, 2020 at 12:26 AM Zhu Zhu <reed...@gmail.com> wrote: > >> Hi All, >> >> Since there are still two 1.11.2 blockers items in progress, RC1 creation >> will be postponed to tomorrow. >> >> Thanks, >> Zhu >> >> Zhu Zhu <reed...@gmail.com> 于2020年9月4日周五 下午3:50写道： >> >> > @Dawid >> > Thanks for the information and preparing a fix for FLINK-19133! >> > I have made it a blocker for 1.11.2 and will keep tracking its status. >> > >> > @Till >> > Thanks for the updates and efforts for FLINK-18959! >> > >> > Thanks, >> > Zhu >> > >> > Till Rohrmann <trohrm...@apache.org> 于2020年9月4日周五 下午3:41写道： >> > >> >> Fyi, FLINK-18959 has been merged into the release-1.10 branch. >> >> >> >> Cheers, >> >> Till >> >> >> >> On Thu, Sep 3, 2020 at 2:38 PM Dawid Wysakowicz < >> dwysakow...@apache.org> >> >> wrote: >> >> >> >>> User has just reported another issue FLINK-19133 which I think should >> be >> >>> a blocker for the 1.11.2 release. I'll try to prepare a fix as soon as >> >>> possible. >> >>> >> >>> On 03/09/2020 09:36, Zhu Zhu wrote: >> >>> > Thanks for the inputs! >> >>> > I have made FLINK-14942 and FLINK-18641 blockers for 1.11.2. >> >>> > >> >>> > And thanks a lot for offering help, zhijiang! >> >>> > >> >>> > Thanks, >> >>> > Zhu >> >>> > >> >>> > Congxian Qiu <qcx978132...@gmail.com> 于2020年9月3日周四 下午3:18写道： >> >>> > >> >>> >> Hi >> >>> >> I'd like to include FLINK-14942 into 1.11.2. FLINK-14942 (this >> >>> fixes a >> >>> >> bug introduce in 1.11.0), there is a pr for it already. >> >>> >> Best, >> >>> >> Congxian >> >>> >> >> >>> >> >> >>> >> Zhou, Brian <b.z...@dell.com> 于2020年9月3日周四 上午11:21写道： >> >>> >> >> >>> >>> Hi, >> >>> >>> >> >>> >>> Thanks Becket for addressing the issue. FLINK-18641 is now a >> blocker >> >>> for >> >>> >>> Pravega connector integration, hope we can have it included in >> 1.11.2 >> >>> >>> release. >> >>> >>> >> >>> >>> Best Regards, >> >>> >>> Brian >> >>> >>> >> >>> >>> -----Original Message----- >> >>> >>> From: Becket Qin <becket....@gmail.com> >> >>> >>> Sent: Thursday, September 3, 2020 11:18 >> >>> >>> To: dev >> >>> >>> Cc: khachatryan.ro...@gmail.com; Till Rohrmann; david; Jingsong >> Li >> >>> >>> Subject: Re: [DISCUSS] Releasing Flink 1.11.2 >> >>> >>> >> >>> >>> >> >>> >>> [EXTERNAL EMAIL] >> >>> >>> >> >>> >>> Hi Zhuzhu, >> >>> >>> >> >>> >>> Thanks for starting the discussion. >> >>> >>> >> >>> >>> I'd like to include FLINK-18641 into 1.11.2 as well. It is a >> >>> regression >> >>> >>> from previous versions and is currently blocking the development >> of >> >>> >> Pravega >> >>> >>> source on top of FLIP-27. >> >>> >>> >> >>> >>> Thanks, >> >>> >>> >> >>> >>> Jiangjie (Becket) Qin >> >>> >>> >> >>> >>> On Wed, Sep 2, 2020 at 11:13 PM Zhu Zhu <reed...@gmail.com> >> wrote: >> >>> >>> >> >>> >>>> Thank you all for the inputs! >> >>> >>>> >> >>> >>>> I agree with Till that we should set a soft deadline first. >> >>> >>>> I'd like to propose next Monday if no new blocker issue pops up. >> >>> >>>> But feel free to raise your concerns if you feel next Monday as a >> >>> >>>> deadline may not work for fixes which should be a blocker for >> >>> 1.11.2. >> >>> >>>> >> >>> >>>> Here's a summary of the wanted/blocker but still open fixes: >> >>> >>>> - FLINK-19121 Avoid accessing HDFS frequently in >> >>> HiveBulkWriterFactory >> >>> >>>> - FLINK-19109 Split Reader eats chained periodic watermarks >> >>> >>>> - (not a strict blocker) FLINK-18959 Fail to >> archiveExecutionGraph >> >>> >>>> because job is not finished when dispatcher close >> >>> >>>> - FLINK-18934 Idle stream does not advance watermark in connected >> >>> >>>> stream >> >>> >>>> >> >>> >>>> Thanks, >> >>> >>>> Zhu >> >>> >>>> >> >>> >>>> Konstantin Knauf <kna...@apache.org> 于2020年9月2日周三 下午9:00写道： >> >>> >>>> >> >>> >>>>> I think it would be nice to include a fix for >> >>> >>>>> https://issues.apache.org/jira/browse/FLINK-18934, too, as it >> >>> >>>>> affects a highly requested feature of Flink 1.11 quite severely. >> >>> >>>>> >> >>> >>>>> On Wed, Sep 2, 2020 at 2:51 PM Till Rohrmann < >> trohrm...@apache.org >> >>> > >> >>> >>>> wrote: >> >>> >>>>>> Thanks a lot for starting this discussion Zhu Zhu and for >> >>> >>>>>> volunteering >> >>> >>>> as >> >>> >>>>>> the release manager. Big +1 for creating the next 1.11 bug fix >> >>> >>>> release. I >> >>> >>>>>> think we already collected quite a bit of fixes which our users >> >>> >>>>>> will benefit from. >> >>> >>>>>> >> >>> >>>>>> For the pending fixes, I would suggest setting a soft deadline >> >>> >>>>>> (maybe >> >>> >>>>> until >> >>> >>>>>> beginning of next week) and then starting to cut the release >> >>> >>>>>> (given >> >>> >>>> that >> >>> >>>>> no >> >>> >>>>>> other blocker issues pop up). Maybe we are able to resolve the >> >>> >>>>>> issues >> >>> >>>>> even >> >>> >>>>>> earlier which would allow us to cut the release also earlier. >> >>> >>>>>> >> >>> >>>>>> From my side I would like to include FLINK-18959 in the >> release. >> >>> >>>>>> But it >> >>> >>>>> is >> >>> >>>>>> not a strict release blocker. >> >>> >>>>>> >> >>> >>>>>> Cheers, >> >>> >>>>>> Till >> >>> >>>>>> >> >>> >>>>>> On Wed, Sep 2, 2020 at 2:40 PM David Anderson >> >>> >>>>>> <da...@alpinegizmo.com> >> >>> >>>>>> wrote: >> >>> >>>>>> >> >>> >>>>>>> I think it's worth considering whether we can get this bugfix >> >>> >>>> included >> >>> >>>>> in >> >>> >>>>>>> 1.11.2: >> >>> >>>>>>> >> >>> >>>>>>> - FLINK-19109 Split Reader eats chained periodic watermarks >> >>> >>>>>>> >> >>> >>>>>>> There is a PR, but it's still a work in progress. Cc'ing >> Roman, >> >>> >>>>>>> who >> >>> >>>> has >> >>> >>>>>>> been working on this. >> >>> >>>>>>> >> >>> >>>>>>> Regards, >> >>> >>>>>>> David >> >>> >>>>>>> >> >>> >>>>>>> >> >>> >>>>>>> On Wed, Sep 2, 2020 at 2:19 PM Zhu Zhu <reed...@gmail.com> >> >>> >> wrote: >> >>> >>>>>>>> Hi All, >> >>> >>>>>>>> >> >>> >>>>>>>> It has been about 1 month since we released Flink 1.11.1. >> It's >> >>> >>>>>>>> not >> >>> >>>>> too >> >>> >>>>>>> far >> >>> >>>>>>>> but >> >>> >>>>>>>> we already have more than 80 resolved improvements/bugs in >> the >> >>> >>>>>>> release-1.11 >> >>> >>>>>>>> branch. Some of them are quite critical. Therefore, I propose >> >>> >>>>>>>> to >> >>> >>>>> create >> >>> >>>>>>> the >> >>> >>>>>>>> next >> >>> >>>>>>>> bugfix release 1.11.2 for Flink 1.11. >> >>> >>>>>>>> >> >>> >>>>>>>> Most noticeable fixes are: >> >>> >>>>>>>> - FLINK-18769 MiniBatch doesn't work with FLIP-95 source >> >>> >>>>>>>> - FLINK-18902 Cannot serve results of asynchronous REST >> >>> >>>>>>>> operations >> >>> >>>> in >> >>> >>>>>>>> per-job mode >> >>> >>>>>>>> - FLINK-18682 Vector orc reader cannot read Hive 2.0.0 table >> >>> >>>>>>>> - FLINK-18608 CustomizedConvertRule#convertCast drops >> >>> >>>>>>>> nullability >> >>> >>>>>>>> - FLINK-18646 Managed memory released check can block RPC >> >>> >>>>>>>> thread >> >>> >>>>>>>> - FLINK-18993 Invoke sanityCheckTotalFlinkMemory method >> >>> >>>>>>>> incorrectly >> >>> >>>>> in >> >>> >>>>>>>> JobManagerFlinkMemoryUtils.java >> >>> >>>>>>>> - FLINK-18663 RestServerEndpoint may prevent server shutdown >> >>> >>>>>>>> - FLINK-18595 Deadlock during job shutdown >> >>> >>>>>>>> - FLINK-18581 Cannot find GC cleaner with java version >> >>> >>>>>>>> previous >> >>> >>>>>>>> jdk8u72(-b01) >> >>> >>>>>>>> - FLINK-17075 Add task status reconciliation between TM and >> JM >> >>> >>>>>>>> >> >>> >>>>>>>> Furthermore, I think the following blocker issue should be >> >>> >>>>>>>> merged >> >>> >>>>>> before >> >>> >>>>>>>> 1.11.2 release >> >>> >>>>>>>> >> >>> >>>>>>>> - FLINK-19121 Avoid accessing HDFS frequently in >> >>> >>>>> HiveBulkWriterFactory >> >>> >>>>>>>> I would volunteer as the release manager and kick off the >> >>> >>>>>>>> release >> >>> >>>>>>> process. >> >>> >>>>>>>> What do you think? >> >>> >>>>>>>> >> >>> >>>>>>>> Please let me know if there are any concerns or any missing >> >>> >>>>>>>> blocker >> >>> >>>>>>> issues >> >>> >>>>>>>> need to be fixed in 1.11.2. >> >>> >>>>>>>> >> >>> >>>>>>>> Thanks, >> >>> >>>>>>>> Zhu Zhu >> >>> >>>>>>>> >> >>> >>>>> >> >>> >>>>> -- >> >>> >>>>> >> >>> >>>>> Konstantin Knauf >> >>> >>>>> >> >>> >>>>> https://twitter.com/snntrable >> >>> >>>>> >> >>> >>>>> https://github.com/knaufk >> >>> >>>>> >> >>> >> >>> >> > > > -- > Best, Jingsong Lee > -- Best, Jingsong Lee
