To quickly clarify: I said Friday 3pm for the feature freeze in my Tuesday email, but I decided to actually turn it into a proper *"end of the week" by doing the feature freeze Sunday evening/Monday morning*.
On Fri, May 5, 2017 at 11:46 AM, Radu Tudoran <radu.tudo...@huawei.com> wrote: > Sorry in case I missed something in the discussion of coming up with the > list of open threads, but there is also > > [FLINK-6075] Support Order By for Stream SQL (Support Limit/Top(Sort) for > Stream SQL) > > ...in case you want to consider it (it only needs a review) > > Best regards, > > -----Original Message----- > From: Robert Metzger [mailto:rmetz...@apache.org] > Sent: Friday, May 05, 2017 10:43 AM > To: dev@flink.apache.org > Subject: Re: [DISCUSS] Feature Freeze > > I've checked the release status again, and it seems that some more issues > made it into master, while other very big new features are still being > finished (incremental checkpointing, retractions, serializer upgrades and > automatic buffer management). > > I'll publish the first, non-voting, testing only RC0 on Monday morning. > > > Completed Features for 1.3 > - TODO after feature freeze > > Blockers: > 12 blockers (4 documentation, 1 refcatoring) --> 8 real blockers (can be > fixed after the feature freeze) > > > Really important (open): > - FLINK-6364: Implement incremental checkpointing in RocksDBStateBackend > (pending PR) > - FLINK-6047: Add support for Retraction in Table API / SQL (depends on > FLINK-6093) > - FLINK-4545: Flink automatically manages TM network buffer (pending PR, > under review) > - FLINK-6178: Allow upgrades to state serializers (pending PR, no review ) > > > Mentioned in the thread (open): > - FLINK-6033: Support UNNEST query in the stream SQL API (no PR) > - FLINK-6335: Parse DISTINCT over grouped window in stream SQL (pending PR) > - FLINK-6373: Add runtime support for distinct aggregation over grouped > windows (pending PR, under review) > - FLINK-6281: Create TableSink for JDBC (pending PR, under review) > - FLINK-6225: Support Row Stream for CassandraSink (pending PR, under > review) > - FLINK-6196: Support dynamic schema in Table Function (pending PR, under > review) > - FLINK-4022: Partition and topic discovery for FlinkKafkaConsumer > (pending PR, no review) > - FLINK-4821: Implement rescalable non-partitioned state for Kinesis > Connector (pending PR, under review) > > Closed: > - FLINK-6377: Support map types in the Table / SQL API (merged) > - FLINK-6398: RowSerializer's duplicate should always return a new instance > (merged) > - FLINK-5998: Un-fat Hadoop from Flink fat jar (merged) > - FLINK-6337: Remove the buffer provider from PartitionRequestServerHandler > (merged) > - FLINK-5892: Recover job state at the granularity of operator (merged) > - FLINK-5906: Add support to register user defined aggregates in > TableEnvironment (merged) > - FLINK-6334: Refactoring UDTF interface (merged) > > > > On Wed, May 3, 2017 at 11:42 AM, Aljoscha Krettek <aljos...@apache.org> > wrote: > > > This is the list of blocking issues for Flink 1.3: > > https://issues.apache.org/jira/browse/FLINK-5740?jql=project > > %20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In% > > 20Progress%22%2C%20Reopened)%20AND%20priority%20%3D% > > 20Blocker%20AND%20fixVersion%20%3D%201.3.0 > > <https://issues.apache.org/jir > > a/browse/FLINK-5740?jql=project%20=%20FLINK%20AND%20status% > > 20in%20(Open,%20%22In%20Progress%22,%20Reopened)%20AND%20priority%20=% > > 20Blocker%20AND%20fixVersion%20=%201.3.0> > > > > Could everyone please update the state of the issues mentioned in this > > thread to “blocking” and set the version to “1.3.0” if they feel that > > this is appropriate. Otherwise it is very hard for the release manager > > to get an overview over the progress of the release. With an > > up-to-date list it’s also possible to triage issues and discuss on the > > issue whether it really should be a release blocker. > > > > Best, > > Aljoscha > > > On 2. May 2017, at 22:29, Eron Wright <eronwri...@gmail.com> wrote: > > > > > > Robert, I'd like to see FLINK-5974 > > > <https://issues.apache.org/jira/browse/FLINK-5974> (Mesos DNS > > > support) added to the list of important issues. A PR is ready. > > > > > > On Tue, May 2, 2017 at 4:30 AM, Kostas Kloudas < > > k.klou...@data-artisans.com> > > > wrote: > > > > > >> The only thing that I want to add in the features to be added for > > >> 1.3 is the NOT pattern for the CEP library. > > >> > > >> There is an open PR here: https://github.com/apache/flink/pull/3798 > > >> < https://github.com/apache/flink/pull/3798> > > >> which is not far from getting in. > > >> > > >> Kostas > > >> > > >>> On May 2, 2017, at 12:10 PM, Robert Metzger <rmetz...@apache.org> > > wrote: > > >>> > > >>> Thanks a lot Ufuk for starting the discussion regarding the 1.3 > > >>> feature freeze. > > >>> > > >>> I didn't feature freeze yesterday (Monday) because it was a public > > >> holiday > > >>> here in Germany. > > >>> > > >>> I haven't made up my mind whether to do the feature freeze today > > >>> or > > not. > > >>> Many important features seem to be close to completion. > > >>> I don't think we can pick features and wait with the release until > > >>> they > > >> are > > >>> finished. The only thing I can imagine we could do is extend the > > >> deadline a > > >>> little bit to give people more time to complete. > > >>> But I'm strictly against multiple extensions. > > >>> > > >>> To make the discussion a bit easier, I've collected all the JIRAs > > >> mentioned > > >>> in this thread. I decided to group them into "really important" > > >>> and "important". I hope nobody is offended by my selection and I'm > > >>> very > > open > > >> to > > >>> discuss it. > > >>> It doesn't really matter anyways because we will not block the > > >>> release > > on > > >>> the completion of certain features. > > >>> > > >>> > > >>> Completed Features for 1.3 > > >>> - TODO! > > >>> > > >>> Blockers: > > >>> - TODO! > > >>> > > >>> > > >>> Really important (open): > > >>> - FLINK-6364: Implement incremental checkpointing in > > RocksDBStateBackend > > >>> (pending PR) > > >>> - FLINK-5906: Add support to register user defined aggregates in > > >>> TableEnvironment (no PR) > > >>> - FLINK-6047: Add support for Retraction in Table API / SQL > > >>> (depends on > > >>> FLINK-6093) > > >>> - FLINK-6093: Implement and turn on retraction for table sink > > >>> (pending > > >> PR) > > >>> - FLINK-6334: Refactoring UDTF interface (pending PR, under > > >>> review) > > >>> - FLINK-5998: Un-fat Hadoop from Flink fat jar (pending PR, under > > review) > > >>> - FLINK-4545: Flink automatically manages TM network buffer > > >>> (pending > > PR, > > >>> under review) > > >>> - FLINK-6178: Allow upgrades to state serializers (pending PR, no > > review > > >> ) > > >>> > > >>> > > >>> Really important (closed): > > >>> - FLINK-5892: Recover job state at the granularity of operator > > >>> (merged) > > >>> > > >>> > > >>> > > >>> Important (open): > > >>> - FLINK-6013: Add Datadog HTTP metrics reporter (pending PR, under > > >> review) > > >>> - FLINK-6337: Remove the buffer provider from > > >> PartitionRequestServerHandler > > >>> (pending PR, under review) > > >>> - FLINK-6033: Support UNNEST query in the stream SQL API (no PR) > > >>> - FLINK-6335: Parse DISTINCT over grouped window in stream SQL > > >>> (pending > > >> PR) > > >>> - FLINK-6373: Add runtime support for distinct aggregation over > > >>> grouped windows (pending PR, under review) > > >>> - FLINK-6281: Create TableSink for JDBC (pending PR, under review) > > >>> - FLINK-6225: Support Row Stream for CassandraSink (pending PR, > > >>> under > > >>> review) > > >>> - FLINK-6196: Support dynamic schema in Table Function (pending > > >>> PR, > > under > > >>> review) > > >>> - FLINK-4022: Partition and topic discovery for FlinkKafkaConsumer > > >> (pending > > >>> PR, no review) > > >>> - FLINK-4821: Implement rescalable non-partitioned state for > > >>> Kinesis Connector (pending PR, under review) Important (closed): > > >>> - FLINK-6377: Support map types in the Table / SQL API (merged) > > >>> - FLINK-6398: RowSerializer's duplicate should always return a new > > >> instance > > >>> (merged) > > >>> > > >>> > > >>> I'll now collect the list of finished features in 1.3 and the > > >>> blockers > > >> and > > >>> update above list. > > >>> > > >>> Stephan's email just came in, and I like the idea of freezing the > > feature > > >>> set now but extending the deadline to Friday for the branching. > > >>> (Friday > > >> 3pm > > >>> CEST) > > >>> > > >>> Does everybody agree to that proposal? > > >>> > > >>> > > >>> On Tue, May 2, 2017 at 12:05 PM, Stephan Ewen <se...@apache.org> > > wrote: > > >>> > > >>>> Thanks all for the lively discussion about the feature freeze and > > >>>> how > > to > > >>>> proceed. > > >>>> Because we committed to a time-based release schedule, we should > > >>>> not > > >> break > > >>>> the feature freeze too badly, or we would just disable the > > >>>> time-based-release-policy at the very first time it would trigger. > > >>>> > > >>>> Here are a few thoughts about what we can do: > > >>>> > > >>>> - First of all, please note that feature freeze does not mean bug > > >>>> fix freeze. Important bug fixes can and should go in as part of > > >>>> the 1.3.0 release testing period > > >>>> > > >>>> - We should probably not add brand new features to the list at > > >>>> this > > >> point > > >>>> (that would most likely break the release schedule completely) > > >>>> > > >>>> - I think there is a case to include some features which are > > >>>> partially (but not fully) in master already, or exist basically > > >>>> completely as > > Pull > > >>>> Requests at this time. > > >>>> > > >>>> How about we do *freeze the set of features* for the release > > >>>> today, > > but > > >>>> *leave > > >>>> a grace period to get the complete-but-not-merged features in > > >>>> until > > the > > >> end > > >>>> of this week*? > > >>>> That would mean forking the release-1.3 branch end of this week. > > >>>> > > >>>> > > >>>> We should try and agree on which features fall under that list, > > >>>> and > > make > > >>>> sure each one has a committer committed to merging them in the > > >>>> next > > >> days. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On Tue, May 2, 2017 at 11:01 AM, Tzu-Li (Gordon) Tai < > > >> tzuli...@apache.org> > > >>>> wrote: > > >>>> > > >>>>> Hi, > > >>>>> > > >>>>> I have 2 parts to discuss about delaying the feature freeze for > > >>>>> a > > bit: > > >>>>> > > >>>>> 1. Allow upgrades to state serializers (FLINK-6178) The feature > > >>>>> is split up into 2 PRs, one of which is already open and > > >> the > > >>>>> second one coming up very soon. > > >>>>> PR #1 reconfigurable TypeSerializers: https:// > > >>>>> github.com/apache/flink/pull/3804 PR #2 (will open soon) > > >>>>> integrate serializer reconfiguration into > > state > > >>>>> restore flow (FLINK-6425) > > >>>>> > > >>>>> 2. Major new functionality for the Kafka and Kinesis consumers. > > >>>>> Kafka - partition & topic regex discovery (PR - > > >>>> https://github.com/apache/ > > >>>>> flink/pull/3746) > > >>>>> Kinesis - rescalable Kinesis consumer (PR - > > https://github.com/apache/ > > >>>>> flink/pull/3001) > > >>>>> > > >>>>> All above mentioned features are not yet merged to master. I > > >>>>> think an additional few days to the end of this week for the > > >>>>> feature freeze > > >> should > > >>>>> make this possible. > > >>>>> > > >>>>> Cheers, > > >>>>> Gordon > > >>>>> > > >>>>> On 29 April 2017 at 4:31:17 AM, Chesnay Schepler > > >>>>> (ches...@apache.org > > ) > > >>>>> wrote: > > >>>>> > > >>>>> FLINK-5892 has been merged. > > >>>>> > > >>>>> For FLINK-4545 (replacing numNetworkBuffer parameter) a PR is > > >>>>> also > > >> still > > >>>>> open and could use a second pair of eyes. > > >>>>> > > >>>>> On 28.04.2017 17:03, Kurt Young wrote: > > >>>>>> Hi Flavio, > > >>>>>> > > >>>>>> I have also fix the issue in 1.2 branch, but the next release > > >>>>>> will > > be > > >>>>> 1.2.2 > > >>>>>> > > >>>>>> Best, > > >>>>>> Kurt > > >>>>>> > > >>>>>> On Fri, Apr 28, 2017 at 11:01 PM, Ted Yu <yuzhih...@gmail.com> > > wrote: > > >>>>>> > > >>>>>>> Flavio: > > >>>>>>> Have you seen this (w.r.t. 1.2.1) ? > > >>>>>>> > > >>>>>>> http://search-hadoop.com/m/Flink/VkLeQejxLg24Lk0D1?subj=+ > > >>>>>>> RESULT+VOTE+Release+Apache+Flink+1+2+1+RC2+ > > >>>>>>> > > >>>>>>> On Fri, Apr 28, 2017 at 5:07 AM, Flavio Pompermaier < > > >>>>> pomperma...@okkam.it> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> Any chance to cherry-pick this also into 1.2.1? We're usign > > >>>>>>>> Flink > > >>>> 1.2.0 > > >>>>>>> in > > >>>>>>>> production and maybe an upgrade to 1.2.1 would be a safer > > >>>>>>>> option > > in > > >>>> the > > >>>>>>>> short term.. > > >>>>>>>> > > >>>>>>>> Best, > > >>>>>>>> Flavio > > >>>>>>>> > > >>>>>>>> On Fri, Apr 28, 2017 at 2:00 PM, Aljoscha Krettek < > > >>>> aljos...@apache.org > > >>>>>> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Ah, I see. The fix for that has been merged into master so > > >>>>>>>>> it > > will > > >>>> be > > >>>>>>>>> release in Flink 1.3. > > >>>>>>>>> > > >>>>>>>>>> On 28. Apr 2017, at 13:50, Flavio Pompermaier < > > >>>> pomperma...@okkam.it> > > >>>>>>>>> wrote: > > >>>>>>>>>> Sorry, you're right Aljosha..the issue number is correct, > > >>>>>>>>>> the > > link > > >>>> is > > >>>>>>>>>> wrong! The correct one is https://issues.apache.org/ > > >>>>>>>>> jira/browse/FLINK-6398 > > >>>>>>>>>> On Fri, Apr 28, 2017 at 11:48 AM, Aljoscha Krettek < > > >>>>>>>> aljos...@apache.org> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> I think there might be a typo. We haven’t yet reached > > >>>>>>>>>>> issue > > >> number > > >>>>>>>> 6389, > > >>>>>>>>>>> if I’m not mistaken. The latest as I’m writing this is 6410. > > >>>>>>>>>>> > > >>>>>>>>>>>> On 28. Apr 2017, at 10:00, Flavio Pompermaier < > > >>>>>>> pomperma...@okkam.it> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>> If it's not a problem it will be great for us to include > > >>>>>>>>>>>> also > > >>>>>>>>> FLINK-6398 > > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/FLINK-6938> if > > >>>>>>>>>>>> it's > > not > > >> a > > >>>>>>> big > > >>>>>>>>>>> deal > > >>>>>>>>>>>> Best, > > >>>>>>>>>>>> Flavio > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Fri, Apr 28, 2017 at 3:32 AM, Zhuoluo Yang < > > >>>>>>>>>>> zhuoluo....@alibaba-inc.com> > > >>>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Hi Devs, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks for the release plan. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Could you also please add the feature FLINK-6196 > > >>>>>>>>>>>>> <https://issues.apache.org/jira/browse/FLINK-6196> > > >>>>>>>>>>>>> Support > > >>>>>>> dynamic > > >>>>>>>>>>> schema > > >>>>>>>>>>>>> in Table Function? > > >>>>>>>>>>>>> I’d like to update the code as comments left on PR today. > > >>>>>>>>>>>>> I will try to make sure the code is updated before the > > >>>>>>>>>>>>> Apr > > >> 30th. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Zhuoluo > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> 在 2017年4月28日，上午8:48，Haohui Mai <ricet...@gmail.com> 写道： > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Hello, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks for starting this thread. It would be great to > > >>>>>>>>>>>>> see the > > >>>>>>>>> following > > >>>>>>>>>>>>> features available in Flink 1.3: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> * Support for complex schema: FLINK-6033, FLINK-6377 > > >>>>>>>>>>>>> * Various improvements on SQL over group windows: > > >>>>>>>>>>>>> FLINK-6335, > > >>>>>>>>> FLINK-6373 > > >>>>>>>>>>>>> * StreamTableSink for JDBC and Cassandra: FLINK-6281, > > >> FLINK-6225 > > >>>>>>>>>>>>> * Decoupling Flink and Hadoop: FLINK-5998 > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> All of them have gone through at least one round of > > >>>>>>>>>>>>> review so > > >>>> I'm > > >>>>>>>>>>>>> optimistic that they can make it to 1.3 in a day or two. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Additionally it would be great to see FLINK-6232 go in, > > >>>>>>>>>>>>> but > > it > > >>>>>>>> depends > > >>>>>>>>>>> on > > >>>>>>>>>>>>> FLINK-5884 so it might be a little bit tough. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>> Haohui > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Thu, Apr 27, 2017 at 12:22 PM Chesnay Schepler < > > >>>>>>>> ches...@apache.org > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Hello, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> FLINK-5892 (Restoring state by operator) is also nearing > > >>>>>>> completion, > > >>>>>>>>> but > > >>>>>>>>>>>>> with only 1 day left before the weekend we're cutting it > > really > > >>>>>>>> short. > > >>>>>>>>>>>>> Since this eliminates a major pain point when updating > > >>>>>>>>>>>>> jobs, > > as > > >>>> it > > >>>>>>>>>>>>> allows the modification of chains, another day or 2 > > >>>>>>>>>>>>> would be > > >>>> good > > >>>>>>> i > > >>>>>>>>>>> think. > > >>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>> Chesnay > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On 27.04.2017 18:55, Bowen Li wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Hi Ufuk, > > >>>>>>>>>>>>> I'd like to get FLINK-6013 (Adding Datadog Http metrics > > >>>>>>> reporter) > > >>>>>>>>>>>>> into > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> release 1.3. It's in the final state of code review in > > >>>>>>>>>>>>> https://github.com/apache/flink/pull/3736 > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>> Bowen > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Thu, Apr 27, 2017 at 8:38 AM, > > >>>>>>>>>>>>> Zhijiang(wangzhijiang999) < wangzhijiang...@aliyun.com> > wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Hi Ufuk, > > >>>>>>>>>>>>> Thank you for launching this topic! > > >>>>>>>>>>>>> I wish my latest refinement of buffer provider ( > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> https://issues.apache.org/ > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> jira/browse/FLINK-6337) to be included in 1.3 and most > > >>>>>>>>>>>>> of the > > >>>>>>> jobs > > >>>>>>>>> can > > >>>>>>>>>>>>> get benefit from it. And I think it can be completed > > >>>>>>>>>>>>> with the > > >>>> help > > >>>>>>>> of > > >>>>>>>>>>>>> your > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> reviews this week. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Cheers,Zhijiang----------------------------------------- > > >>>>>>>>>>>>> ---- > > >>>>>>>>>>>>> ---------------------发件人：Ufuk > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Celebi <u...@apache.org>发送时间：2017年4月27日(星期四) 22:25收件人：dev > > >>>>>>>>>>>>> < > > >>>>>>>>>>>>> dev@flink.apache.org>抄 送：Robert Metzger > > >>>>>>>>>>>>> <rmetz...@apache.org > > >主 > > >>>>>>>>>>>>> 题：[DISCUSS] Feature Freeze Hey devs! :-) > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> We decided to follow a time-based release model with the > > >>>> upcoming > > >>>>>>>> 1.3 > > >>>>>>>>>>>>> release and the planned feature freeze is on Monday, May > 1st. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> I wanted to start a discussion to get a quick overview > > >>>>>>>>>>>>> of the > > >>>>>>>> current > > >>>>>>>>>>>>> state of things. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> - Is everyone on track and aware of the feature freeze? > > >>>>>>>>>>>>> ;) > > >>>>>>>>>>>>> - Are there any major features we want in 1.3 that have > > >>>>>>>>>>>>> not been merged yet? > > >>>>>>>>>>>>> - Do we need to extend the feature freeze, because of an > > >>>>>>>>>>>>> important feature? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Would be great to gather a list of features/PRs that we > > >>>>>>>>>>>>> want > > in > > >>>>>>> the > > >>>>>>>>>>>>> 1.3 release. This could be a good starting point for the > > >> release > > >>>>>>>>>>>>> manager (@Robert?). > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Best, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Ufuk > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>> > > >>>>> > > >>>> > > >> > > >> > > > > >
