Hi Gordon, Thanks for the kind reminder of feature freeze date for 1.9.0. I think the date makes sense on my side.
For FLIP-31, I and Andrey could be done within two weeks or so. And I already finished my side work for FLIP-1. Best, Zhijiang ------------------------------------------------------------------ From:Timo Walther <twal...@apache.org> Send Time:2019年5月28日(星期二) 19:26 To:dev <dev@flink.apache.org> Subject:Re: [DISCUSS] Features for Apache Flink 1.9.0 Thanks for being the release managers, Kurt and Gordon! From the Table & SQL API side, there are still a lot of open issues that need to be solved to decouple the API from a planner and enable the Blink planner. Also we need to make sure that the Blink planner supports at least everything of Flink 1.8 to not introduce a regression. We might need to focus more on the main features which is a runnable Blink planner and might need to postpone other discussions such as DDL, new source/sink interfaces, or proper type inference logic. However, in many cases there are shortcuts that we could take in order to achieve our goals. So I'm confident that we solve the big blockers until the feature freeze :) I will keep you updated. Thanks, Timo Am 28.05.19 um 05:07 schrieb Kurt Young: > Thanks Gordon for bringing this up. > > I'm glad to say that blink planner merge work is almost done, and i will > follow up the work of > integrating blink planner with Table API to co-exist with current flink > planner. > > In addition to this, the following features: > 1. FLIP-32: Restructure flink-table for future contributions [1] > 2. FLIP-37: Rework of the Table API Type System [2] > 3. Hive integration work (including hive meta [3] and connectors) > > are also going well, i will spend some time to keep track of them. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > [3] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > > Best, > Kurt > > > On Mon, May 27, 2019 at 7:18 PM jincheng sun <sunjincheng...@gmail.com> > wrote: > >> Hi Gordon, >> >> Thanks for mention the feature freeze date for 1.9.0, that's very helpful >> for contributors to evaluate their dev plan！ >> >> Regarding FLIP-29, we are glad to do our best to finish the dev of FLIP-29, >> then catch up with the release of 1.9. >> >> Thanks again for push the release of 1.9.0 forward! >> >> Cheers, >> Jincheng >> >> >> >> Tzu-Li (Gordon) Tai <tzuli...@apache.org> 于2019年5月27日周一 下午5:48写道： >> >>> Hi all, >>> >>> I want to kindly remind the community that we're now 5 weeks away from >> the >>> proposed feature freeze date for 1.9.0, which is June 28. >>> >>> This is not yet a final date we have agreed on, so I would like to start >>> collecting feedback on how the mentioned features are going, and in >>> general, whether or not the date sounds reasonable given the current >> status >>> of the ongoing efforts. >>> Please let me know what you think! >>> >>> Cheers, >>> Gordon >>> >>> >>> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org >>> >>> wrote: >>> >>>> @Hequn @Jincheng >>>> >>>> Thanks for bringing up FLIP-29 to attention. >>>> As previously mentioned, the original list is not a fixed feature set, >> so >>>> if FLIP-29 has ongoing efforts and can make it before the feature >> freeze, >>>> then of course it should be included! >>>> >>>> @himansh1306 >>>> >>>> Concerning the ORC format for StreamingFileSink, is there already a >> JIRA >>>> ticket tracking that? If not, I suggest to first open one and see if >>> there >>>> are similar interests from committers in adding that. >>>> >>>> >>>> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <chenghe...@gmail.com> >>> wrote: >>>>> Hi, >>>>> >>>>> Great job, Gordon! Thanks a lot for driving this and wrapping features >>> up >>>>> to a detailed list. +1 on it! >>>>> >>>>> Would be great if we can also add flip29 to the list. @jincheng sun >>>>> <sunjincheng...@gmail.com> and I are focusing on it these days. I >>> think >>>>> these features in flip29 would bring big enhancements to the Table >> API. >>>>> :-) >>>>> >>>>> Best, Hequn >>>>> >>>>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <becket....@gmail.com> >>> wrote: >>>>>> Thanks for driving this release, Gordon. +1 on the feature list. >>>>>> >>>>>> This is a pretty exciting and ambitious release! >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Jiangjie (Becket) Qin >>>>>> >>>>>> On Sun, May 5, 2019 at 4:28 PM jincheng sun < >> sunjincheng...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> Thanks a lot for being our release manager, Great job! >>>>>>> >>>>>>> +1 for the feature list and It's better to add FLIP-29 >>>>>>> < >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739 >>>>>>>> (Support >>>>>>> map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of >>>>> release >>>>>>> 1.9. >>>>>>> >>>>>>> What do you think? >>>>>>> >>>>>>> Best, >>>>>>> Jincheng >>>>>>> >>>>>>> Bowen Li <bowenl...@gmail.com> 于2019年5月5日周日 上午12:47写道： >>>>>>> >>>>>>>> +1, exciting and ambitious goals, the rough timeline looks >>>>> reasonable. >>>>>>>> Let's make it happen! >>>>>>>> >>>>>>>> On Sat, May 4, 2019 at 2:47 AM Jark Wu <imj...@gmail.com> >> wrote: >>>>>>>>> +1 for the 1.9.0 feature list. Excited to see it is happening. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Jark >>>>>>>>> >>>>>>>>> On Thu, 2 May 2019 at 17:07, himansh1...@gmail.com < >>>>>>>> himansh1...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 for Protobuf, Hive Metastore integration & features >> related >>>>> to >>>>>>>>>> savepoint. >>>>>>>>>> >>>>>>>>>> I was hoping if we could support for ORC File format can be >>>>> added >>>>>> in >>>>>>>>>> StreamingFileSink Writer, Currently only Parquet is >> supported >>>>> with >>>>>>>>> respect >>>>>>>>>> to columnar file formats >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" < >>>>> tzuli...@apache.org >>>>>>>>>> wrote: >>>>>>>>>>> Hi community, >>>>>>>>>>> >>>>>>>>>>> Apache Flink 1.8.0 has been released a few weeks ago, so >>>>>> naturally, >>>>>>>>> it’s >>>>>>>>>>> time to start thinking about what we want to aim for >> 1.9.0. >>>>>>>>>>> Kurt and I had collected some features that would be >>>>> reasonable >>>>>> to >>>>>>>>>> consider >>>>>>>>>>> including for the next release, based on talking with >>> various >>>>>>> people >>>>>>>> as >>>>>>>>>>> well as observations from mailing list discussions and >>>>> questions. >>>>>>>>>>> Note that having specific features listed here does not >> mean >>>>> that >>>>>>> no >>>>>>>>>> other >>>>>>>>>>> pull requests or topics will be reviewed. I am sure that >>> there >>>>>> are >>>>>>>>> other >>>>>>>>>>> ongoing efforts that we missed here and will likely make >> it >>>>> as an >>>>>>>>>>> improvement or new feature in the next release. This >>>>> discussion >>>>>> is >>>>>>>>> merely >>>>>>>>>>> for bootstrapping a discussion for 1.9, as well as to give >>>>>>>> contributors >>>>>>>>>> an >>>>>>>>>>> idea of what the community is looking to focus on in the >>> next >>>>>>> couple >>>>>>>> of >>>>>>>>>>> weeks. >>>>>>>>>>> >>>>>>>>>>> *Proposed features and focus* >>>>>>>>>>> >>>>>>>>>>> In the previous major release, Apache Flink 1.8.0, the >>>>> community >>>>>>> had >>>>>>>>>>> prepared for some major Table & SQL additions from the >> Blink >>>>>>> branch. >>>>>>>>> With >>>>>>>>>>> this in mind, for the next release, it would be great to >>> wind >>>>> up >>>>>>>> those >>>>>>>>>>> efforts by merging in the Blink-based Table / SQL planner >>> and >>>>>>> runtime >>>>>>>>> for >>>>>>>>>>> 1.9. >>>>>>>>>>> >>>>>>>>>>> Following Stephan’s previous thread [1] in the mailing >> list >>>>> about >>>>>>>>>> features >>>>>>>>>>> in Blink, we should also start focusing on preparing for >>>>> Blink’s >>>>>>>> other >>>>>>>>>>> several enhancements for batch execution. This includes >>>>> resource >>>>>>>>>>> optimization, fine-grained failover, pluggable shuffle >>>>> service, >>>>>>>>> adapting >>>>>>>>>>> stream operators for batch execution, as well as better >>>>>> integration >>>>>>>>> with >>>>>>>>>>> commonly used systems by batch executions such as Apache >>> Hive. >>>>>>>>>>> Moreover, besides efforts related to the Blink merge, we >>> would >>>>>> also >>>>>>>>> like >>>>>>>>>> us >>>>>>>>>>> to work towards pushing forward some of the most discussed >>> and >>>>>>>>>> anticipated >>>>>>>>>>> features by the community. Most of these had discussions >> in >>>>> the >>>>>>>> mailing >>>>>>>>>>> lists that span multiple releases, and are also frequently >>>>>> brought >>>>>>> up >>>>>>>>> in >>>>>>>>>>> community events such as Flink Forward. This includes >>> features >>>>>> such >>>>>>>> as >>>>>>>>>>> source event-time alignment and the source interface >>> rework, a >>>>>>>>> savepoint >>>>>>>>>>> connector that allows users to manipulate and query state >> in >>>>>>>>> savepoints, >>>>>>>>>>> interactive programming, as well as terminating a job >> with a >>>>>> final >>>>>>>>>>> savepoint. >>>>>>>>>>> >>>>>>>>>>> Last but not least, we have several existing contributions >>> or >>>>>>>>> discussions >>>>>>>>>>> for the ecosystem surrounding Flink, which we think is >> also >>>>> very >>>>>>>>> valuable >>>>>>>>>>> to try to merge in for 1.9. This includes a web UI rework >>>>>> (recently >>>>>>>>>> already >>>>>>>>>>> merged), active K8s integration, Google PubSub connector, >>>>> native >>>>>>>>> support >>>>>>>>>>> for the Protobuf format, Python support in the Table API, >> as >>>>> well >>>>>>> as >>>>>>>>>>> reworking Flink’s support for machine learning. >>>>>>>>>>> >>>>>>>>>>> To wrap this up as a list of items, some of which already >>> have >>>>>>> JIRAs >>>>>>>> or >>>>>>>>>>> mailing list threads to track them: >>>>>>>>>>> >>>>>>>>>>> - Merge Blink runner for Table & SQL [2] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Restructure flink-table to separate API from core >>>>> runtime >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Make table planners pluggable >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Rework Table / SQL type system to integrate better >>> with >>>>> the >>>>>>> SQL >>>>>>>>>>> standard [3] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Merge Blink planner and runtime for Table / SQL >>>>>>>>>>> - Further preparations for more batch execution >>>>>> optimization >>>>>>>> from >>>>>>>>>>> Blink >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Dedicated scheduler component [4] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Fine grained failover for batch [5] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Selectable input stream operator [6] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Pluggable Shuffle Service [7] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> FLIP-30: Unified Catalog API & Hive metastore >>>>> integration >>>>>> [8] >>>>>>>>>>> - Heavily anticipated / discussed features in the >>>>> community >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> FLIP-27: Source interface rework [9] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Savepoint connector [10] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> FLIP-34: Terminate / Suspend job with savepoint [11] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> FLIP-36: Interactive Programming [12] >>>>>>>>>>> - Ecosystem >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Web UI rework [13] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Active K8s integration [14] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Google PubSub connector [15] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> First-class Protobuf support [16] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> FLIP-38: Python support in Table API [17] >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> FLIP-39: Flink ML pipeline and libraries on top of >>> Table >>>>>> API >>>>>>>> [18] >>>>>>>>>>> *Suggested release timeline* >>>>>>>>>>> >>>>>>>>>>> Apache Flink 1.8.0 was released earlier this month, so >> based >>>>> on >>>>>> our >>>>>>>>> usual >>>>>>>>>>> timely release schedule, we should aim for releasing 1.9.0 >>>>> around >>>>>>> mid >>>>>>>>> to >>>>>>>>>>> end July. >>>>>>>>>>> >>>>>>>>>>> Since it seems that this is going to be a fairly large >>>>> release, >>>>>> to >>>>>>>> give >>>>>>>>>> the >>>>>>>>>>> community enough testing time, I propose that the feature >>>>> freeze >>>>>> to >>>>>>>> be >>>>>>>>>> near >>>>>>>>>>> the end of June (8-9 weeks from now, probable June 28). >> This >>>>> is >>>>>> of >>>>>>>>>> course a >>>>>>>>>>> ballpark estimation for now; we should follow-up with a >>>>> separate >>>>>>>> thread >>>>>>>>>>> later in the release cycle to prepare contributors with an >>>>>> official >>>>>>>>>> feature >>>>>>>>>>> freeze date. >>>>>>>>>>> >>>>>>>>>>> I’d also like to use this opportunity to propose myself >> and >>>>> Kurt >>>>>> as >>>>>>>> the >>>>>>>>>>> release managers for 1.9. >>>>>>>>>>> AFAIK, we did not used to have 2 RMs for a single release >> in >>>>> the >>>>>>>> past, >>>>>>>>>> but >>>>>>>>>>> 1.9.0 is definitely quite ambitious so it would not hurt >> to >>>>> have >>>>>>> one >>>>>>>>> more >>>>>>>>>>> on board :) Cheers, Gordon [1] >>>>>>>>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html >>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11439 >>>>>>>>>>> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-12251 >>>>>>>>>>> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-10429 >>>>>>>>>>> >>>>>>>>>>> [5] >>>>>>>>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html >>>>>>>>>>> [6] >>>>>>>>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html >>>>>>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-10653 >>>>>>>>>>> >>>>>>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-11275 >>>>>>>>>>> >>>>>>>>>>> [9] >>>>>>>>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html >>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-12047 >>>>>>>>>>> >>>>>>>>>>> [11] >>>>>>>>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html >>>>>>>>>>> [12] >>>>>>>>>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink >>>>>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10705 >>>>>>>>>>> >>>>>>>>>>> [14] https://issues.apache.org/jira/browse/FLINK-9953 >>>>>>>>>>> >>>>>>>>>>> [15] https://issues.apache.org/jira/browse/FLINK-9311 >>>>>>>>>>> >>>>>>>>>>> [16] https://issues.apache.org/jira/browse/FLINK-11333 >>>>>>>>>>> >>>>>>>>>>> [17] >>>>>>>>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html >>>>>>>>>>> [18] >>>>>>>>>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
