Re: [DISCUSS] Release flink-shaded 10.0

2020-02-07 Thread Jeff Zhang
+1, Thanks for driving this Chesnay Yu Li 于2020年2月8日周六 上午11:53写道: > +1 for starting a new flink-shaded release. > > Thanks for bringing this up Chesnay. > > Best Regards, > Yu > > > On Thu, 6 Feb 2020 at 20:59, Hequn Cheng wrote: > > > +1. It sounds great to allow us to support zk 3.4 and

Re: [DISCUSS] Release flink-shaded 10.0

2020-02-07 Thread Yu Li
+1 for starting a new flink-shaded release. Thanks for bringing this up Chesnay. Best Regards, Yu On Thu, 6 Feb 2020 at 20:59, Hequn Cheng wrote: > +1. It sounds great to allow us to support zk 3.4 and 3.5. > Thanks for starting the discussion. > > Best, > Hequn > > On Thu, Feb 6, 2020 at

Re: [DISCUSS] Remove registration of TableSource/TableSink in Table Env and ConnectTableDescriptor

2020-02-07 Thread Kurt Young
Hi Timo, tableEnv.fromElements/values() sounds good, do we have a jira ticket to track the issue? Best, Kurt On Fri, Feb 7, 2020 at 10:56 PM Timo Walther wrote: > Hi Kurt, > > Dawid is currently working on making a tableEnv.fromElements/values() > kind of source possible in the future. We

[VOTE] Release 1.10.0, release candidate #3

2020-02-07 Thread Gary Yao
Hi everyone, Please review and vote on the release candidate #3 for the version 1.10.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Re: [VOTE] Release 1.10.0, release candidate #2

2020-02-07 Thread Gary Yao
Hi everyone, I am hereby canceling the vote due to: FLINK-15917 FLINK-15918 FLINK-15935 FLINK-15937 RC3 will be created later today. Best, Gary On Thu, Feb 6, 2020 at 4:39 PM Andrey Zagrebin wrote: > alright, thanks for confirming this Benchao! > > On Thu, Feb 6, 2020 at

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

2020-02-07 Thread Rong Rong
CC @Xu Yang Thanks for starting the discussion @Hequn Cheng and sorry for joining the discussion late. I've mainly helped merging the code in flink-ml-api and flink-ml-lib in the past several months. IMO the flink-ml-api are an extension on top of the table API and agree that it should be

Re: [DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system

2020-02-07 Thread Zhenghua Gao
Thanks Timo! Look forward your design! *Best Regards,* *Zhenghua Gao* On Fri, Feb 7, 2020 at 5:26 PM Timo Walther wrote: > Hi Zhenghua, > > Jark is right. The reason why we haven't updated those interfaces yet is > because we are actually would like to introduce new interfaces. We > should

[jira] [Created] (FLINK-15958) Fully support RAW types in the API

2020-02-07 Thread Timo Walther (Jira)
Timo Walther created FLINK-15958: Summary: Fully support RAW types in the API Key: FLINK-15958 URL: https://issues.apache.org/jira/browse/FLINK-15958 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Remove registration of TableSource/TableSink in Table Env and ConnectTableDescriptor

2020-02-07 Thread Timo Walther
Hi Kurt, Dawid is currently working on making a tableEnv.fromElements/values() kind of source possible in the future. We can use this to replace some of the tests. Otherwise I guess we should come up with a better test infrastructure to make defining source not necessary anymore. Regards,

[DISCUSS] FLINK-15831: Add Docker image publication to release documentation

2020-02-07 Thread Patrick Lucas
Hi all, For FLINK-15831[1], I think the way to start is for the flink-docker repo[2] itself to sufficiently document the workflow for publishing new Dockerfiles, and then update the Flink release guide in the wiki to refer to this documentation and to include this step in the "Finalize the

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

2020-02-07 Thread Hequn Cheng
Hi, @Till Rohrmann Thanks for the great inputs. I agree with you that we should have a long term plan for this. It definitely deserves another discussion. @Jeff Zhang Thanks for your reports and ideas. It's a good idea to improve the error messages. Do we have any JIRAs for it or maybe we can

Re: [DISCUSS] Does removing deprecated interfaces needs another FLIP

2020-02-07 Thread Stephan Ewen
I would also agree with the above. Changing a stable API and deprecating stable methods would need a FLIP in my opinion. But then executing the removal of previously deprecated methods would be fine in my understanding. On Fri, Feb 7, 2020 at 11:17 AM Kurt Young wrote: > Thanks for the

[jira] [Created] (FLINK-15957) Document the release process for Stateful Functions in the community wiki

2020-02-07 Thread Tzu-Li (Gordon) Tai (Jira)
Tzu-Li (Gordon) Tai created FLINK-15957: --- Summary: Document the release process for Stateful Functions in the community wiki Key: FLINK-15957 URL: https://issues.apache.org/jira/browse/FLINK-15957

Re: [VOTE] FLIP-27 - Refactor Source Interface

2020-02-07 Thread Stephan Ewen
+1 (binding) (belated) Quick addendum to clarify some questions from recent discussions in other threads: - The core interfaces (Source, SourceReader, Enumerator) and the core architecture (Enumerator as coordinators on the JobManager, SourceReaders in Tasks) seem to have no open questions

[jira] [Created] (FLINK-15956) Introduce an HttpRemoteFunction

2020-02-07 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-15956: Summary: Introduce an HttpRemoteFunction Key: FLINK-15956 URL: https://issues.apache.org/jira/browse/FLINK-15956 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-15955) Split RemoteFunction into GRPC Function/HttpFunction

2020-02-07 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-15955: Summary: Split RemoteFunction into GRPC Function/HttpFunction Key: FLINK-15955 URL: https://issues.apache.org/jira/browse/FLINK-15955 Project: Flink Issue

[jira] [Created] (FLINK-15954) Introduce a PersistedTable to the SDK

2020-02-07 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-15954: Summary: Introduce a PersistedTable to the SDK Key: FLINK-15954 URL: https://issues.apache.org/jira/browse/FLINK-15954 Project: Flink Issue Type: Task

Re: Question: Modifying Checkpoint Interval at runtime

2020-02-07 Thread Congxian Qiu
In current Flink, the checkpoint interval can not be modified after job started, If you want to change the code to implement this feature, maybe you can try to see the logic in CheckpointCoordinator, and the usage of function checkMinPauseBetweenCheckpoints[1] and scheduleTriggerWithDelay()[2]

[jira] [Created] (FLINK-15953) Job Status is hard to read for some Statuses

2020-02-07 Thread Gary Yao (Jira)
Gary Yao created FLINK-15953: Summary: Job Status is hard to read for some Statuses Key: FLINK-15953 URL: https://issues.apache.org/jira/browse/FLINK-15953 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-15952) Don't use raw byte[] in MultiplexedState

2020-02-07 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-15952: Summary: Don't use raw byte[] in MultiplexedState Key: FLINK-15952 URL: https://issues.apache.org/jira/browse/FLINK-15952 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-15951) Travis build fails on npm install

2020-02-07 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-15951: - Summary: Travis build fails on npm install Key: FLINK-15951 URL: https://issues.apache.org/jira/browse/FLINK-15951 Project: Flink Issue Type: Bug

[DISCUSS] Allow for getting late elements via side outputs from IntervalJoinOperator

2020-02-07 Thread Bakht Kahloon
I was hoping to make an enhancement to the interval join operator in regards to having access to late elements from either the left or right stream. I was thinking this could possibly be done using the side outputs feature.

Re: [DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system

2020-02-07 Thread Jark Wu
Cool! Looking forward to the design doc. Best, Jark On Fri, 7 Feb 2020 at 17:26, Timo Walther wrote: > Hi Zhenghua, > > Jark is right. The reason why we haven't updated those interfaces yet is > because we are actually would like to introduce new interfaces. We > should target new interfaces

Question: Modifying Checkpoint Interval at runtime

2020-02-07 Thread Morgan Geldenhuys
I am working with the Flink at the moment and am interested in modifying the codebase so that the checkpoint interval can be changed at runtime, (e.g. if it was set to kickoff the distributed snapshot process every 10 seconds, and then I wanted to change it to every 15 seconds instead without

Re: performances of S3 writing with many buckets in parallel

2020-02-07 Thread Kostas Kloudas
Hi Enrico, Nice to hear from you and thanks for checking it out! This can be helpful for people using the BucketingSink but I would recommend you to switch to the StreamingFileSink which is the "new version" of the BucketingSink. In fact the BucketingSink is going to be removed in one of the

Re: [DISCUSS] Remove registration of TableSource/TableSink in Table Env and ConnectTableDescriptor

2020-02-07 Thread Kurt Young
Thanks all for your feedback, since no objection has been raised, I've created https://issues.apache.org/jira/browse/FLINK-15950 to track this issue. Since this issue would require lots of tests adjustment before it really happen, it won't be done in a short time. Feel free to give feedback

[jira] [Created] (FLINK-15950) Remove registration of TableSource/TableSink in Table Env and ConnectTableDescriptor

2020-02-07 Thread Kurt Young (Jira)
Kurt Young created FLINK-15950: -- Summary: Remove registration of TableSource/TableSink in Table Env and ConnectTableDescriptor Key: FLINK-15950 URL: https://issues.apache.org/jira/browse/FLINK-15950

Re: performances of S3 writing with many buckets in parallel

2020-02-07 Thread Enrico Agnoli
I finally found the time to dig a little more on this and found the real problem. The culprit of the slow-down is this piece of code:

Re: [DISCUSS] Does removing deprecated interfaces needs another FLIP

2020-02-07 Thread Kurt Young
Thanks for the clarification, that make sense to me. Best, Kurt On Fri, Feb 7, 2020 at 4:56 PM Timo Walther wrote: > Hi Kurt, > > I agree with Aljoscha. We don't need to introduce a big process or do > voting but we should ensure that all stakeholders are notified and have > a chance to raise

[jira] [Created] (FLINK-15949) Harden jackson dependency constraints

2020-02-07 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-15949: Summary: Harden jackson dependency constraints Key: FLINK-15949 URL: https://issues.apache.org/jira/browse/FLINK-15949 Project: Flink Issue Type:

[jira] [Created] (FLINK-15948) Resource will be wasted when the task manager memory is not a multiple of Yarn minimum allocation

2020-02-07 Thread Yang Wang (Jira)
Yang Wang created FLINK-15948: - Summary: Resource will be wasted when the task manager memory is not a multiple of Yarn minimum allocation Key: FLINK-15948 URL: https://issues.apache.org/jira/browse/FLINK-15948

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-02-07 Thread Gyula Fóra
Maybe we could improve the Pipeline interface in the long run, but as a temporary solution the JobClient could expose a getPipeline() method. That way the implementation of the JobListener could check if its a StreamGraph or a Plan. How bad does that sound? Gyula On Fri, Feb 7, 2020 at 10:19

Re: [DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system

2020-02-07 Thread Timo Walther
Hi Zhenghua, Jark is right. The reason why we haven't updated those interfaces yet is because we are actually would like to introduce new interfaces. We should target new interfaces in this release. Even a short-term fix as you proposed with `getRecordDataType` does actually not help as

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-02-07 Thread Gyula Fóra
Hi Aljoscha! That's a valid concert but we should try to figure something out, many users need this before they can use Flink. I think the closest thing we have right now is the StreamGraph. In contrast with the JobGraph the StreamGraph is pretty nice from a metadata perspective :D The big

Re: [VOTE] Improve TableFactory to add Context

2020-02-07 Thread Leonard Xu
+1(non-binding), nice design! after read full discussion mail list. Best, Leonard Xu > 在 2020年2月6日,23:12,Timo Walther 写道: > > +1 > > On 06.02.20 05:54, Bowen Li wrote: >> +1, LGTM >> On Tue, Feb 4, 2020 at 11:28 PM Jark Wu wrote: >>> +1 form my side. >>> Thanks for driving this. >>> >>>

Re: [DISCUSS] FLIP-75: Flink Web UI Improvement Proposal

2020-02-07 Thread Yadong Xie
Hi Till FLIP-75 has been open since September, and the design doc has been iterated over 3 versions and more than 20 patches. I had a try, but it is hard to split the design docs into sub FLIP and keep all the discussion history at the same time. Maybe it is better to start another discussion to

Re: [DISCUSS] Does removing deprecated interfaces needs another FLIP

2020-02-07 Thread Timo Walther
Hi Kurt, I agree with Aljoscha. We don't need to introduce a big process or do voting but we should ensure that all stakeholders are notified and have a chance to raise doubts. Regards, Timo On 07.02.20 09:51, Aljoscha Krettek wrote: I would say a ML discussion or even a Jira issue is

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-02-07 Thread Aljoscha Krettek
If we need it, we can probably beef up the JobListener to allow accessing some information about the whole graph or sources and sinks. My only concern right now is that we don't have a stable interface for our job graphs/pipelines right now. Best, Aljoscha On 06.02.20 23:00, Gyula Fóra

Re: [DISCUSS] Does removing deprecated interfaces needs another FLIP

2020-02-07 Thread Aljoscha Krettek
I would say a ML discussion or even a Jira issue is enough because a) the methods are already deprecated b) the methods are @PublicEvolving, which I don't consider a super strong guarantee to users (we still shouldn't remove them lightly, but we can if we have to...) Best, Aljoscha On

Re: [VOTE] FLIP-27 - Refactor Source Interface

2020-02-07 Thread Becket Qin
Thanks everyone for voting. The voting result is following: +1 (Binding): 5 (Yu, Jark, Zhijiang, Piotr, Becket) +1 (Non-binding): 4 (Jingsong, Danny, Wei, Guowei) -1: 0 FLIP-27 has passed. Thanks, Jiangjie (Becket) Qin On Tue, Feb 4, 2020 at 3:42 PM Piotr Nowojski wrote: > +1 (binding) >

[jira] [Created] (FLINK-15947) Finish moving scala expression DSL to flink-table-api-scala

2020-02-07 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-15947: Summary: Finish moving scala expression DSL to flink-table-api-scala Key: FLINK-15947 URL: https://issues.apache.org/jira/browse/FLINK-15947 Project: Flink