Re: [DISCUSS] Ground Source and Sink Concepts in Flink SQL

[Timo Walther]

Thanks for working on this great design document Jark. I think having 
well-defined terminilogy and semantics around tables, changelogs, table 
sources/sinks, and DDL should have been done much earlier. I will take a 
closer look at the concepts and give feedback soon. I think having those 
concepts defined and implemented should be the goal for Flink 1.10. It 
also allows us to align it to the efforts of FLIP-27.


Introducing a DDL is a step that cannot be evolved easily as a DDL is 
basically just a string that is being parsed. We should aim to involve 
as many people as possible to have a future-proof design.


Thanks,
Timo

Am 27.05.19 um 10:40 schrieb Kurt Young:

Thanks Jark for bringing this topic. I think proper concepts is very
important for users who are using Table API & SQL. Especially for
them to have a clear understanding about the behavior of the SQL job. Also
this is essential for connector developers to have a better
understanding why we abstracted the interfaces in this way, and have a
smooth experience when developing connectors for Table & SQL.

Best,
Kurt


On Mon, May 27, 2019 at 3:35 PM Jark Wu  wrote:


Hi all,

We have prepared a design doc [1] about source and sink concepts in Flink
SQL. This is actually an extended discussion about SQL DDL [2].

In the design doc, we want to figure out some concept problems. For
examples:

1. How to define boundedness in DDL
2. How to define a changelog in DDL, what's the behavior of a changelog
source and changelog sink?
3. How to define primary key in DDL and what's the semantic when we have a
primary key on a table and stream?

They are mostly related to DDL because DDL is plain text and we need to
keep close to standard as much as possible.

This is an important step before we starting to refactor our
TableSource/TableSink/TableFactory interfaces. Because we need to know what
changes we need to introduce to support these concepts.

Please feel free to leave feedbacks in the thread or the design doc.

Regards,
Jark

[1].

https://docs.google.com/document/d/1yrKXEIRATfxHJJ0K3t6wUgXAtZq8D-XgvEnvl2uUcr0/edit#
[2].

http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-SQL-DDL-Design-tt25006.html

Rolling policy when using StreamingFileSink for bulk-encoded output

[Ying Xu]

Dear Flink community:

We have a use case where StreamingFileSink

is used for persisting bulk-encoded data to AWS s3. In our case, the data
sources consist of hybrid types of events, for which each type is uploaded
to an individual s3 prefix location. Because the event size is highly
skewed, the uploaded file size may differ dramatically.  In order to have a
better control over the uploaded file size, we would like to adopt a
rolling policy based on file sizes (e.g., roll the file every 100MB). Yet
it appears bulk-encoding StreamingFileSink only supports checkpoint-based
file rolling.

IMPORTANT: Bulk-encoding formats can only be combined with the
`OnCheckpointRollingPolicy`, which rolls the in-progress part file on every
checkpoint.

Checkpoint-based file rolling appears to have other side effects. For
instance, quite a lot of the heavy liftings (e.g file parts uploading) are
performed at the checkpointing time. As a result, checkpointing takes
longer duration when data volume is high.

Having a customized file rolling policy can be achieved by small
adjustments on the BulkFormatBuilder interface in StreamingFileSink. In the
case of using S3RecoverableWriter, file rolling triggers data uploading and
corresponding S3Committer is also constructed and stored. Hence on the
surface, adding a simple file-size based rolling policy would NOT
compromise the established exact-once guarantee.

Any advises on whether the above idea makes sense? Or perhaps there are
pitfalls that one might pay attention when introducing such rolling policy.
Thanks a lot!


-
Ying

Re: [DISCUSS] Start a user...@flink.apache.org mailing list for the Chinese-speaking community?

[Biao Liu]

Hi community,

I have tested two most widely used mailbox website mail.qq.com and
mail.163.com(belongs to Netease).

qq.mail works fine, good job @vino yang !
163.mail almost works fine. I have received all the response mails,
although the last one is marked as spam.

I received the new mail of the mailing list from both of them. But there is
a delay of 7 or 8 minutes of 163.com. I'm not sure it's always like this or
not. Will keep observing it.

Another thing, the survey of @Hequn Cheng  shows that
about 14 people said their questions are not answered in mailing list.
There always be some questions that are hard to answer in the user-zh
mailing list. Like "why my job is delayed" but without providing any
detail, or some questions that are more relevant with Java or operating
system but not Flink. Mail like these probably be ignored.

I have a proposal that we Chinese-speaker of community could do better if
we can give them a response even it's not an answer.
A positive feedback is helpful for the user without a good technical
background. I think the user-zh mailing list would be more active if each
question gets a feedback.


Hequn Cheng  于2019年6月24日周一 上午10:34写道：

> Hi,
>
> I'd like to share you the result of the survey. Thanks for the help
> from @Gordon, @Jark and the Flink China operation team when conducting the
> survey.
>
> A total of 81 people participated in the survey. 46 of them choose not to
> use the mailing list. Among these people, the reasons are(Note that a
> respondent may choose multiple reasons):
> 1. 22 people report that problems can be solved in the Dingtalk group and
> it's more convenient than the mailing list.
> 2. 22 people even don't know there is a Chinese user mailing list.
> 3. 20 people don't know how to use the mailing list even though they have
> ever heard of it.
> 4. 14 people said that problems can't be solved in the mailing list even
> they asked in it.
> 5. 16 people choose to use the English user mailing list.
>
> From the result, the biggest obstacle that stops more people involved in
> the mailing list is people don't know it or don't know how to use it. To
> solve the problem, we can do more publicity. I have also recorded a usage
> video about how to subscribe and use the mailing list. Hope it will help.
>
> However, doing more publicity is not enough as 14 people said problems
> can't be solved efficiently in the mailing list. More people should also be
> involved to answer the problems. I don't know whether is it possible for
> the Chinese Flink team on duty to answer the problems. I think that would
> help.
>
> Great to have other opinions.
>
> Best, Hequn
>
>
> On Fri, Jun 21, 2019 at 7:50 PM Hequn Cheng  wrote:
>
> > Hi vino,
> >
> > Thanks a lot for unblocking the email address. I have told the user about
> > this.
> > Hope things can get better.
> >
> > Best, Hequn
> >
> > On Fri, Jun 21, 2019 at 3:14 PM vino yang  wrote:
> >
> >> Hi Hequn,
> >>
> >> Thanks for reporting this case.
> >>
> >> The reason replied by QQ mail team is also caused by *bounce attack*.
> So
> >> this mail address has been intercepted and it's an IP level
> interception.
> >>
> >> Today, the QQ mail team has unblocked this email address. So it can
> >> receive
> >> the follow-up email from Apache mail server normally.
> >>
> >> If this email address still can not work normally in the future. Please
> >> report it here again.
> >>
> >> Best,
> >> Vino
> >>
> >>
> >> Hequn Cheng  于2019年6月21日周五 下午2:39写道：
> >>
> >> > Hi Vino,
> >> >
> >> > Great thanks for your help.
> >> >
> >> > > So if someone reports that they can't receive the email from Apache
> >> mail
> >> > server, they can provide more detailed information to the QQ mailbox
> to
> >> > facilitate the location problem.
> >> >
> >> > I just got one feedback.
> >> > A user(173855...@qq.com) report that he can't receive the emails from
> >> the
> >> > Chinese-speaking mailing list. He had subscripted successfully on
> >> > 2019-05-10. Everything goes well until 2019-05-10 and no more emails
> >> come
> >> > again from the mailing list.
> >> >
> >> > Best, Hequn
> >> >
> >> > On Fri, Jun 21, 2019 at 12:56 PM vino yang 
> >> wrote:
> >> >
> >> > > Hi Kurt,
> >> > >
> >> > > I have copied my reply to the Jira issue of INFRA[1].
> >> > >
> >> > > Within my ability, I am happy to coordinate and promote this
> problem.
> >> > >
> >> > > Best,
> >> > > Vino
> >> > >
> >> > > [1]: https://issues.apache.org/jira/browse/INFRA-18249
> >> > >
> >> > > Kurt Young  于2019年6月21日周五 下午12:11写道：
> >> > >
> >> > > > Hi vino,
> >> > > >
> >> > > > Thanks for your effort. Could you also share this information with
> >> > apache
> >> > > > INFRA? Maybe we can find a workable solution together.
> >> > > > You can try to leave comments in this jira:
> >> > > > https://issues.apache.org/jira/browse/INFRA-18249)
> >> > > >
> >> > > > Best,
> >> > > > Kurt
> >> > > >
> >> > > >
> >> > > > On Fri, Jun 21, 2019 at 11:45 AM vino yang  >
> >> > > wrote:
> >> > > 

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[vino yang]

Hi all,

I am happy we have a wonderful discussion and received many valuable
opinions in the last few days.

Now, let me try to summarize what we have reached consensus about the
changes in the design.

   - provide a unified abstraction to support two kinds of implementation;
   - reuse WindowOperator and try to enhance it so that we can make the
   intermediate result of the local aggregation can be buffered and flushed to
   support two kinds of implementation;
   - keep the API design of localKeyBy, but declare the disabled some APIs
   we cannot support currently, and provide a configurable API for users to
   choose how to handle intermediate result;

The above three points have been updated in the design doc. Any
questions, please let me know.

@Aljoscha Krettek  What do you think? Any further
comments?

Best,
Vino

vino yang  于2019年6月20日周四 下午2:02写道：

> Hi Kurt,
>
> Thanks for your comments.
>
> It seems we come to a consensus that we should alleviate the performance
> degraded by data skew with local aggregation. In this FLIP, our key
> solution is to introduce local keyed partition to achieve this goal.
>
> I also agree that we can benefit a lot from the usage of
> AggregateFunction. In combination with localKeyBy, We can easily use it to
> achieve local aggregation:
>
>- input.localKeyBy(0).aggregate()
>- input.localKeyBy(0).window().aggregate()
>
>
> I think the only problem here is the choices between
>
>- (1) Introducing a new primitive called localKeyBy and implement
>local aggregation with existing operators, or
>- (2) Introducing an operator called localAggregation which is
>composed of a key selector, a window-like operator, and an aggregate
>function.
>
>
> There may exist some optimization opportunities by providing a composited
> interface for local aggregation. But at the same time, in my opinion, we
> lose flexibility (Or we need certain efforts to achieve the same
> flexibility).
>
> As said in the previous mails, we have many use cases where the
> aggregation is very complicated and cannot be performed with
> AggregateFunction. For example, users may perform windowed aggregations
> according to time, data values, or even external storage. Typically, they
> now use KeyedProcessFunction or customized triggers to implement these
> aggregations. It's not easy to address data skew in such cases with a
> composited interface for local aggregation.
>
> Given that Data Stream API is exactly targeted at these cases where the
> application logic is very complicated and optimization does not matter, I
> think it's a better choice to provide a relatively low-level and canonical
> interface.
>
> The composited interface, on the other side, may be a good choice in
> declarative interfaces, including SQL and Table API, as it allows more
> optimization opportunities.
>
> Best,
> Vino
>
>
> Kurt Young  于2019年6月20日周四 上午10:15写道：
>
>> Hi all,
>>
>> As vino said in previous emails, I think we should first discuss and
>> decide
>> what kind of use cases this FLIP want to
>> resolve, and what the API should look like. From my side, I think this is
>> probably the root cause of current divergence.
>>
>> My understand is (from the FLIP title and motivation section of the
>> document), we want to have a proper support of
>> local aggregation, or pre aggregation. This is not a very new idea, most
>> SQL engine already did this improvement. And
>> the core concept about this is, there should be an AggregateFunction, no
>> matter it's a Flink runtime's AggregateFunction or
>> SQL's UserDefinedAggregateFunction. Both aggregation have concept of
>> intermediate data type, sometimes we call it ACC.
>> I quickly went through the POC piotr did before [1], it also directly uses
>> AggregateFunction.
>>
>> But the thing is, after reading the design of this FLIP, I can't help
>> myself feeling that this FLIP is not targeting to have a proper
>> local aggregation support. It actually want to introduce another concept:
>> LocalKeyBy, and how to split and merge local key groups,
>> and how to properly support state on local key. Local aggregation just
>> happened to be one possible use case of LocalKeyBy.
>> But it lacks supporting the essential concept of local aggregation, which
>> is intermediate data type. Without this, I really don't thing
>> it is a good fit of local aggregation.
>>
>> Here I want to make sure of the scope or the goal about this FLIP, do we
>> want to have a proper local aggregation engine, or we
>> just want to introduce a new concept called LocalKeyBy?
>>
>> [1]: https://github.com/apache/flink/pull/4626
>>
>> Best,
>> Kurt
>>
>>
>> On Wed, Jun 19, 2019 at 5:13 PM vino yang  wrote:
>>
>> > Hi Hequn,
>> >
>> > Thanks for your comments!
>> >
>> > I agree that allowing local aggregation reusing window API and refining
>> > window operator to make it match both requirements (come from our and
>> Kurt)
>> > is a good decision!
>> >
>> > Concerning your questions:
>> >
>> > 

Re: [DISCUSS] Ground Source and Sink Concepts in Flink SQL

[Jark Wu]

Thanks Timo,

I think it's fine to target it for Flink 1.10.  Looking forward for your
feedback.

On Mon, 24 Jun 2019 at 15:07, Timo Walther  wrote:

> Thanks for working on this great design document Jark. I think having
> well-defined terminilogy and semantics around tables, changelogs, table
> sources/sinks, and DDL should have been done much earlier. I will take a
> closer look at the concepts and give feedback soon. I think having those
> concepts defined and implemented should be the goal for Flink 1.10. It
> also allows us to align it to the efforts of FLIP-27.
>
> Introducing a DDL is a step that cannot be evolved easily as a DDL is
> basically just a string that is being parsed. We should aim to involve
> as many people as possible to have a future-proof design.
>
> Thanks,
> Timo
>
> Am 27.05.19 um 10:40 schrieb Kurt Young:
> > Thanks Jark for bringing this topic. I think proper concepts is very
> > important for users who are using Table API & SQL. Especially for
> > them to have a clear understanding about the behavior of the SQL job.
> Also
> > this is essential for connector developers to have a better
> > understanding why we abstracted the interfaces in this way, and have a
> > smooth experience when developing connectors for Table & SQL.
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, May 27, 2019 at 3:35 PM Jark Wu  wrote:
> >
> >> Hi all,
> >>
> >> We have prepared a design doc [1] about source and sink concepts in
> Flink
> >> SQL. This is actually an extended discussion about SQL DDL [2].
> >>
> >> In the design doc, we want to figure out some concept problems. For
> >> examples:
> >>
> >> 1. How to define boundedness in DDL
> >> 2. How to define a changelog in DDL, what's the behavior of a changelog
> >> source and changelog sink?
> >> 3. How to define primary key in DDL and what's the semantic when we
> have a
> >> primary key on a table and stream?
> >>
> >> They are mostly related to DDL because DDL is plain text and we need to
> >> keep close to standard as much as possible.
> >>
> >> This is an important step before we starting to refactor our
> >> TableSource/TableSink/TableFactory interfaces. Because we need to know
> what
> >> changes we need to introduce to support these concepts.
> >>
> >> Please feel free to leave feedbacks in the thread or the design doc.
> >>
> >> Regards,
> >> Jark
> >>
> >> [1].
> >>
> >>
> https://docs.google.com/document/d/1yrKXEIRATfxHJJ0K3t6wUgXAtZq8D-XgvEnvl2uUcr0/edit#
> >> [2].
> >>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-SQL-DDL-Design-tt25006.html
> >>
>
>

Re: [DISCUSS] Flink client api enhancement for downstream project

[Jeff Zhang]

Hi, Tison,

Thanks for your comments. Overall I agree with you that it is difficult for
down stream project to integrate with flink and we need to refactor the
current flink client api.
And I agree that CliFrontend should only parsing command line arguments and
then pass them to ExecutionEnvironment. It is ExecutionEnvironment's
responsibility to compile job, create cluster, and submit job. Besides
that, Currently flink has many ExecutionEnvironment implementations, and
flink will use the specific one based on the context. IMHO, it is not
necessary, ExecutionEnvironment should be able to do the right thing based
on the FlinkConf it is received. Too many ExecutionEnvironment
implementation is another burden for downstream project integration.

One thing I'd like to mention is flink's scala shell and sql client,
although they are sub-modules of flink, they could be treated as downstream
project which use flink's client api. Currently you will find it is not
easy for them to integrate with flink, they share many duplicated code.  It
is another sign that we should refactor flink client api.

I believe it is a large and hard change, and I am afraid we can not keep
compatibility since many of changes are user facing.



Zili Chen  于2019年6月24日周一 下午2:53写道：

> Hi all,
>
> After a closer look on our client apis, I can see there are two major
> issues to consistency and integration, namely different deployment of
> job cluster which couples job graph creation and cluster deployment,
> and submission via CliFrontend confusing control flow of job graph
> compilation and job submission. I'd like to follow the discuss above,
> mainly the process described by Jeff and Stephan, and share my
> ideas on these issues.
>
> 1) CliFrontend confuses the control flow of job compilation and submission.
> Following the process of job submission Stephan and Jeff described,
> execution environment knows all configs of the cluster and topos/settings
> of the job. Ideally, in the main method of user program, it calls #execute
> (or named #submit) and Flink deploys the cluster, compile the job graph
> and submit it to the cluster. However, current CliFrontend does all these
> things inside its #runProgram method, which introduces a lot of subclasses
> of (stream) execution environment.
>
> Actually, it sets up an exec env that hijacks the #execute/executePlan
> method, initializes the job graph and abort execution. And then
> control flow back to CliFrontend, it deploys the cluster(or retrieve
> the client) and submits the job graph. This is quite a specific internal
> process inside Flink and none of consistency to anything.
>
> 2) Deployment of job cluster couples job graph creation and cluster
> deployment. Abstractly, from user job to a concrete submission, it requires
>
>  create JobGraph --\
>
> create ClusterClient -->  submit JobGraph
>
> such a dependency. ClusterClient was created by deploying or retrieving.
> JobGraph submission requires a compiled JobGraph and valid ClusterClient,
> but the creation of ClusterClient is abstractly independent of that of
> JobGraph. However, in job cluster mode, we deploy job cluster with a job
> graph, which means we use another process:
>
> create JobGraph --> deploy cluster with the JobGraph
>
> Here is another inconsistency and downstream projects/client apis are
> forced to handle different cases with rare supports from Flink.
>
> Since we likely reached a consensus on
>
> 1. all configs gathered by Flink configuration and passed
> 2. execution environment knows all configs and handles execution(both
> deployment and submission)
>
> to the issues above I propose eliminating inconsistencies by following
> approach:
>
> 1) CliFrontend should exactly be a front end, at least for "run" command.
> That means it just gathered and passed all config from command line to
> the main method of user program. Execution environment knows all the info
> and with an addition to utils for ClusterClient, we gracefully get a
> ClusterClient by deploying or retrieving. In this way, we don't need to
> hijack #execute/executePlan methods and can remove various hacking
> subclasses of exec env, as well as #run methods in ClusterClient(for an
> interface-ized ClusterClient). Now the control flow flows from CliFrontend
> to the main method and never returns.
>
> 2) Job cluster means a cluster for the specific job. From another
> perspective, it is an ephemeral session. We may decouple the deployment
> with a compiled job graph, but start a session with idle timeout
> and submit the job following.
>
> These topics, before we go into more details on design or implementation,
> are better to be aware and discussed for a consensus.
>
> Best,
> tison.
>
>
> Zili Chen  于2019年6月20日周四 上午3:21写道：
>
>> Hi Jeff,
>>
>> Thanks for raising this thread and the design document!
>>
>> As @Thomas Weise mentioned above, extending config to flink
>> requires far more effort than it should be. Another example
>> is we achieve detac

Re: [DISCUSS] Connectors and NULL handling

[Becket Qin]

Hi Aljoscha,

Thanks for raising the issue. It seems there are two issues here: 1) The
null value handling, and 2) The error handling.

For null value handling, my understanding is the following:
  - Null values could have a realistic meaning in some systems. So Flink
needs to support them.
  - By design, in Flink, the records passed between Flink operators have
already supported null values. They are wrapped in StreamRecord.
  - Some user facing APIs, however, seem not fully support null values.
e.g. the Collector.
  - The connector code are sort of "user code" from Flink's perspective. So
each connector should decide how null value should be treated.
If we want to support null values in Flink everywhere, we may need to look
into those user facing APIs that do not take null values. Wrapping the user
returned value looks reasonable, ideally the wrapper class should also be
StreamRecord so it is consistent with what we have for those records passed
between operators.

WRT error handling, I agree with Xiaowei that the error handling mechanism
should be something generic to the entire project instead of just for
connectors. This reminds of another discussion thread which proposes to add
a pluggable to categorize and report exceptions causing job failure [1]. It
might worth thinking to see whether it makes sense to design the error
handling and reporting as a whole.

Thanks,

Jiangjie (Becket) Qin

[1]
https://docs.google.com/document/d/1oLF3w1wYyr8vqoFoQZhw1QxTofmAtlD8IF694oPLjNI/edit?usp=sharing




On Sun, Jun 23, 2019 at 9:40 PM Xiaowei Jiang  wrote:

> Error handling policy for streaming jobs goes beyond potential corrupted
> messages in the source. Users may have subtle bugs while processing some
> messages which may cause the streaming jobs to fail. Even though this can
> be considered as a bug in user's code, users may prefer skip such messages
> (or log them) and let the job continue in some cases. This may be an
> opportunity to take such cases into consideration as well.
>
> Xiaowei
>
> On Fri, Jun 21, 2019 at 11:43 PM Rong Rong  wrote:
>
>> Hi Aljoscha,
>>
>> Sorry for the late reply, I think the solution makes sense. Using the NULL
>> return value to mark a message is corrupted is not a valid way since NULL
>> value has semantic meaning in not just Kafka but also in a lot of other
>> contexts.
>>
>> I was wondering if we can have a more meaningful interface for dealing
>> with
>> corrupted messages. I am thinking of 2 options on top of my head:
>> 1. Create some special deserializer attribute (or a special record) to
>> indicate corrupted messages like you suggested; this way we can not only
>> encode the deserializing error but allow users to encode any corruption
>> information for downstream processing.
>> 2. Create a standard fetch error handling API on AbstractFetcher (for
>> Kafka) and DataFetcher (for Kinesis); this way we can also handle error's
>> other than deserializing problem, for example some even lower level
>> exceptions like CRC check failure.
>>
>> I think either way will work. Also, as long as there's a way for end users
>> to extend the error handling for message corruption, it will not
>> reintroduce the problems these 2 original JIRA was trying to address.
>>
>> --
>> Rong
>>
>> On Tue, Jun 18, 2019 at 2:06 AM Aljoscha Krettek 
>> wrote:
>>
>> > Hi All,
>> >
>> > Thanks to Gary, I recently came upon an interesting cluster of issues:
>> >  - https://issues.apache.org/jira/browse/FLINK-3679: Allow Kafka
>> consumer
>> > to skip corrupted messages
>> >  - https://issues.apache.org/jira/browse/FLINK-5583: Support flexible
>> > error handling in the Kafka consumer
>> >  - https://issues.apache.org/jira/browse/FLINK-11820:
>> SimpleStringSchema
>> > handle message record which value is null
>> >
>> > In light of the last one I’d like to look again at the first two. What
>> > they introduced is that when the deserialisation schema returns NULL,
>> the
>> > Kafka consumer (and maybe also the Kinesis consumer) silently drops the
>> > record. In Kafka NULL values have semantic meaning, i.e. they usually
>> > encode a DELETE for the key of the message. If SimpleStringSchema
>> returned
>> > that null, our consumer would silently drop it and we would lose that
>> > DELETE message. That doesn’t seem right.
>> >
>> > I think the right solution for null handling is to introduce a custom
>> > record type that encodes both Kafka NULL values and the possibility of a
>> > corrupt message that cannot be deserialised. Something like an Either
>> type.
>> > It’s then up to the application to handle those cases.
>> >
>> > Concretely, I want to discuss whether we should change our consumers to
>> > not silently drop null records, but instead see them as errors. For
>> > FLINK-11820, the solution is for users to write their own custom schema
>> > that handles null values and returns a user-defined types that signals
>> null
>> > values.
>> >
>> > What do you think?
>> >
>> > Aljoscha
>> >
>> >
>>
>

[jira] [Created] (FLINK-12957) Fix thrift and protobuf dependency examples in documentation

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-12957:
---

 Summary: Fix thrift and protobuf dependency examples in 
documentation
 Key: FLINK-12957
 URL: https://issues.apache.org/jira/browse/FLINK-12957
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.8.0, 1.7.2, 1.9.0
Reporter: Nico Kruber
Assignee: Nico Kruber


The examples in the docs are not up-to-date anymore and should be updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[Kurt Young]

Hi vino,

Sorry I don't see the consensus about reusing window operator and keep the
API design of localKeyBy. But I think we should definitely more thoughts
about this topic.

I also try to loop in Stephan for this discussion.

Best,
Kurt


On Mon, Jun 24, 2019 at 3:26 PM vino yang  wrote:

> Hi all,
>
> I am happy we have a wonderful discussion and received many valuable
> opinions in the last few days.
>
> Now, let me try to summarize what we have reached consensus about the
> changes in the design.
>
>- provide a unified abstraction to support two kinds of implementation;
>- reuse WindowOperator and try to enhance it so that we can make the
>intermediate result of the local aggregation can be buffered and
> flushed to
>support two kinds of implementation;
>- keep the API design of localKeyBy, but declare the disabled some APIs
>we cannot support currently, and provide a configurable API for users to
>choose how to handle intermediate result;
>
> The above three points have been updated in the design doc. Any
> questions, please let me know.
>
> @Aljoscha Krettek  What do you think? Any further
> comments?
>
> Best,
> Vino
>
> vino yang  于2019年6月20日周四 下午2:02写道：
>
> > Hi Kurt,
> >
> > Thanks for your comments.
> >
> > It seems we come to a consensus that we should alleviate the performance
> > degraded by data skew with local aggregation. In this FLIP, our key
> > solution is to introduce local keyed partition to achieve this goal.
> >
> > I also agree that we can benefit a lot from the usage of
> > AggregateFunction. In combination with localKeyBy, We can easily use it
> to
> > achieve local aggregation:
> >
> >- input.localKeyBy(0).aggregate()
> >- input.localKeyBy(0).window().aggregate()
> >
> >
> > I think the only problem here is the choices between
> >
> >- (1) Introducing a new primitive called localKeyBy and implement
> >local aggregation with existing operators, or
> >- (2) Introducing an operator called localAggregation which is
> >composed of a key selector, a window-like operator, and an aggregate
> >function.
> >
> >
> > There may exist some optimization opportunities by providing a composited
> > interface for local aggregation. But at the same time, in my opinion, we
> > lose flexibility (Or we need certain efforts to achieve the same
> > flexibility).
> >
> > As said in the previous mails, we have many use cases where the
> > aggregation is very complicated and cannot be performed with
> > AggregateFunction. For example, users may perform windowed aggregations
> > according to time, data values, or even external storage. Typically, they
> > now use KeyedProcessFunction or customized triggers to implement these
> > aggregations. It's not easy to address data skew in such cases with a
> > composited interface for local aggregation.
> >
> > Given that Data Stream API is exactly targeted at these cases where the
> > application logic is very complicated and optimization does not matter, I
> > think it's a better choice to provide a relatively low-level and
> canonical
> > interface.
> >
> > The composited interface, on the other side, may be a good choice in
> > declarative interfaces, including SQL and Table API, as it allows more
> > optimization opportunities.
> >
> > Best,
> > Vino
> >
> >
> > Kurt Young  于2019年6月20日周四 上午10:15写道：
> >
> >> Hi all,
> >>
> >> As vino said in previous emails, I think we should first discuss and
> >> decide
> >> what kind of use cases this FLIP want to
> >> resolve, and what the API should look like. From my side, I think this
> is
> >> probably the root cause of current divergence.
> >>
> >> My understand is (from the FLIP title and motivation section of the
> >> document), we want to have a proper support of
> >> local aggregation, or pre aggregation. This is not a very new idea, most
> >> SQL engine already did this improvement. And
> >> the core concept about this is, there should be an AggregateFunction, no
> >> matter it's a Flink runtime's AggregateFunction or
> >> SQL's UserDefinedAggregateFunction. Both aggregation have concept of
> >> intermediate data type, sometimes we call it ACC.
> >> I quickly went through the POC piotr did before [1], it also directly
> uses
> >> AggregateFunction.
> >>
> >> But the thing is, after reading the design of this FLIP, I can't help
> >> myself feeling that this FLIP is not targeting to have a proper
> >> local aggregation support. It actually want to introduce another
> concept:
> >> LocalKeyBy, and how to split and merge local key groups,
> >> and how to properly support state on local key. Local aggregation just
> >> happened to be one possible use case of LocalKeyBy.
> >> But it lacks supporting the essential concept of local aggregation,
> which
> >> is intermediate data type. Without this, I really don't thing
> >> it is a good fit of local aggregation.
> >>
> >> Here I want to make sure of the scope or the goal about this FLIP, do we
> >> want to have a p

[jira] [Created] (FLINK-12958) Integrate AsyncWaitOperator with mailbox

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-12958:
--

 Summary: Integrate AsyncWaitOperator with mailbox
 Key: FLINK-12958
 URL: https://issues.apache.org/jira/browse/FLINK-12958
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Stefan Richter
Assignee: Stefan Richter






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Flink client api enhancement for downstream project

[Flavio Pompermaier]

That's exactly what I suggested a long time ago: the Flink REST client
should not require any Flink dependency, only http library to call the REST
services to submit and monitor a job.
What I suggested also in [1] was to have a way to automatically suggest the
user (via a UI) the available main classes and their required parameters[2].
Another problem we have with Flink is that the Rest client and the CLI one
behaves differently and we use the CLI client (via ssh) because it allows
to call some other method after env.execute() [3] (we have to call another
REST service to signal the end of the job).
Int his regard, a dedicated interface, like the JobListener suggested in
the previous emails, would be very helpful (IMHO).

[1] https://issues.apache.org/jira/browse/FLINK-10864
[2] https://issues.apache.org/jira/browse/FLINK-10862
[3] https://issues.apache.org/jira/browse/FLINK-10879

Best,
Flavio

On Mon, Jun 24, 2019 at 9:54 AM Jeff Zhang  wrote:

> Hi, Tison,
>
> Thanks for your comments. Overall I agree with you that it is difficult for
> down stream project to integrate with flink and we need to refactor the
> current flink client api.
> And I agree that CliFrontend should only parsing command line arguments and
> then pass them to ExecutionEnvironment. It is ExecutionEnvironment's
> responsibility to compile job, create cluster, and submit job. Besides
> that, Currently flink has many ExecutionEnvironment implementations, and
> flink will use the specific one based on the context. IMHO, it is not
> necessary, ExecutionEnvironment should be able to do the right thing based
> on the FlinkConf it is received. Too many ExecutionEnvironment
> implementation is another burden for downstream project integration.
>
> One thing I'd like to mention is flink's scala shell and sql client,
> although they are sub-modules of flink, they could be treated as downstream
> project which use flink's client api. Currently you will find it is not
> easy for them to integrate with flink, they share many duplicated code.  It
> is another sign that we should refactor flink client api.
>
> I believe it is a large and hard change, and I am afraid we can not keep
> compatibility since many of changes are user facing.
>
>
>
> Zili Chen  于2019年6月24日周一 下午2:53写道：
>
> > Hi all,
> >
> > After a closer look on our client apis, I can see there are two major
> > issues to consistency and integration, namely different deployment of
> > job cluster which couples job graph creation and cluster deployment,
> > and submission via CliFrontend confusing control flow of job graph
> > compilation and job submission. I'd like to follow the discuss above,
> > mainly the process described by Jeff and Stephan, and share my
> > ideas on these issues.
> >
> > 1) CliFrontend confuses the control flow of job compilation and
> submission.
> > Following the process of job submission Stephan and Jeff described,
> > execution environment knows all configs of the cluster and topos/settings
> > of the job. Ideally, in the main method of user program, it calls
> #execute
> > (or named #submit) and Flink deploys the cluster, compile the job graph
> > and submit it to the cluster. However, current CliFrontend does all these
> > things inside its #runProgram method, which introduces a lot of
> subclasses
> > of (stream) execution environment.
> >
> > Actually, it sets up an exec env that hijacks the #execute/executePlan
> > method, initializes the job graph and abort execution. And then
> > control flow back to CliFrontend, it deploys the cluster(or retrieve
> > the client) and submits the job graph. This is quite a specific internal
> > process inside Flink and none of consistency to anything.
> >
> > 2) Deployment of job cluster couples job graph creation and cluster
> > deployment. Abstractly, from user job to a concrete submission, it
> requires
> >
> >  create JobGraph --\
> >
> > create ClusterClient -->  submit JobGraph
> >
> > such a dependency. ClusterClient was created by deploying or retrieving.
> > JobGraph submission requires a compiled JobGraph and valid ClusterClient,
> > but the creation of ClusterClient is abstractly independent of that of
> > JobGraph. However, in job cluster mode, we deploy job cluster with a job
> > graph, which means we use another process:
> >
> > create JobGraph --> deploy cluster with the JobGraph
> >
> > Here is another inconsistency and downstream projects/client apis are
> > forced to handle different cases with rare supports from Flink.
> >
> > Since we likely reached a consensus on
> >
> > 1. all configs gathered by Flink configuration and passed
> > 2. execution environment knows all configs and handles execution(both
> > deployment and submission)
> >
> > to the issues above I propose eliminating inconsistencies by following
> > approach:
> >
> > 1) CliFrontend should exactly be a front end, at least for "run" command.
> > That means it just gathered and passed all config from command line to
> > the main method of

[jira] [Created] (FLINK-12959) Use BoundedInput and InputSelectable in blink

[Jingsong Lee (JIRA)]

Jingsong Lee created FLINK-12959:


 Summary: Use BoundedInput and InputSelectable in blink
 Key: FLINK-12959
 URL: https://issues.apache.org/jira/browse/FLINK-12959
 Project: Flink
  Issue Type: New Feature
Reporter: Jingsong Lee
Assignee: Jingsong Lee


Now BoundedInput and InputSelectable are ready in runtime. Blink planner should 
use it instead of invoking endInput in close.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Rolling policy when using StreamingFileSink for bulk-encoded output

[Kostas Kloudas]

Hi Ying,

Thanks for using the StreamingFileSink.

The reason why the StreamingFileSink only supports
OnCheckpointRollingPolicy with bulk
formats has to do with the fact that currently Flink relies on the Hadoop
writer for Parquet.

Bulk formats keep important details about how they write the actual data
(such as compression
schemes, offsets, etc) in metadata and they write this metadata with the
file (e.g. parquet writes
them as a footer). The hadoop writer gives no access to these metadata.
Given this, there is
no way for flink to be able to checkpoint a part file securely without
closing it.

The solution would be to write our own writer and not go through the hadoop
one, but there
are no concrete plans for this, as far as I know.

I hope this explains a bit more why the StreamingFileSink has this
limitation.

Cheers,
Kostas


On Mon, Jun 24, 2019 at 9:19 AM Ying Xu  wrote:

> Dear Flink community:
>
> We have a use case where StreamingFileSink
> <
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html
> >
> is used for persisting bulk-encoded data to AWS s3. In our case, the data
> sources consist of hybrid types of events, for which each type is uploaded
> to an individual s3 prefix location. Because the event size is highly
> skewed, the uploaded file size may differ dramatically.  In order to have a
> better control over the uploaded file size, we would like to adopt a
> rolling policy based on file sizes (e.g., roll the file every 100MB). Yet
> it appears bulk-encoding StreamingFileSink only supports checkpoint-based
> file rolling.
>
> IMPORTANT: Bulk-encoding formats can only be combined with the
> `OnCheckpointRollingPolicy`, which rolls the in-progress part file on every
> checkpoint.
>
> Checkpoint-based file rolling appears to have other side effects. For
> instance, quite a lot of the heavy liftings (e.g file parts uploading) are
> performed at the checkpointing time. As a result, checkpointing takes
> longer duration when data volume is high.
>
> Having a customized file rolling policy can be achieved by small
> adjustments on the BulkFormatBuilder interface in StreamingFileSink. In the
> case of using S3RecoverableWriter, file rolling triggers data uploading and
> corresponding S3Committer is also constructed and stored. Hence on the
> surface, adding a simple file-size based rolling policy would NOT
> compromise the established exact-once guarantee.
>
> Any advises on whether the above idea makes sense? Or perhaps there are
> pitfalls that one might pay attention when introducing such rolling policy.
> Thanks a lot!
>
>
> -
> Ying
>

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[vino yang]

Hi Kurt,

You did not give more further different opinions, so I thought you have
agreed with the design after we promised to support two kinds of
implementation.

In API level, we have answered your question about pass an
AggregateFunction to do the aggregation. No matter introduce localKeyBy API
or not, we can support AggregateFunction.

So what's your different opinion now? Can you share it with us?

Best,
Vino

Kurt Young  于2019年6月24日周一 下午4:24写道：

> Hi vino,
>
> Sorry I don't see the consensus about reusing window operator and keep the
> API design of localKeyBy. But I think we should definitely more thoughts
> about this topic.
>
> I also try to loop in Stephan for this discussion.
>
> Best,
> Kurt
>
>
> On Mon, Jun 24, 2019 at 3:26 PM vino yang  wrote:
>
> > Hi all,
> >
> > I am happy we have a wonderful discussion and received many valuable
> > opinions in the last few days.
> >
> > Now, let me try to summarize what we have reached consensus about the
> > changes in the design.
> >
> >- provide a unified abstraction to support two kinds of
> implementation;
> >- reuse WindowOperator and try to enhance it so that we can make the
> >intermediate result of the local aggregation can be buffered and
> > flushed to
> >support two kinds of implementation;
> >- keep the API design of localKeyBy, but declare the disabled some
> APIs
> >we cannot support currently, and provide a configurable API for users
> to
> >choose how to handle intermediate result;
> >
> > The above three points have been updated in the design doc. Any
> > questions, please let me know.
> >
> > @Aljoscha Krettek  What do you think? Any further
> > comments?
> >
> > Best,
> > Vino
> >
> > vino yang  于2019年6月20日周四 下午2:02写道：
> >
> > > Hi Kurt,
> > >
> > > Thanks for your comments.
> > >
> > > It seems we come to a consensus that we should alleviate the
> performance
> > > degraded by data skew with local aggregation. In this FLIP, our key
> > > solution is to introduce local keyed partition to achieve this goal.
> > >
> > > I also agree that we can benefit a lot from the usage of
> > > AggregateFunction. In combination with localKeyBy, We can easily use it
> > to
> > > achieve local aggregation:
> > >
> > >- input.localKeyBy(0).aggregate()
> > >- input.localKeyBy(0).window().aggregate()
> > >
> > >
> > > I think the only problem here is the choices between
> > >
> > >- (1) Introducing a new primitive called localKeyBy and implement
> > >local aggregation with existing operators, or
> > >- (2) Introducing an operator called localAggregation which is
> > >composed of a key selector, a window-like operator, and an aggregate
> > >function.
> > >
> > >
> > > There may exist some optimization opportunities by providing a
> composited
> > > interface for local aggregation. But at the same time, in my opinion,
> we
> > > lose flexibility (Or we need certain efforts to achieve the same
> > > flexibility).
> > >
> > > As said in the previous mails, we have many use cases where the
> > > aggregation is very complicated and cannot be performed with
> > > AggregateFunction. For example, users may perform windowed aggregations
> > > according to time, data values, or even external storage. Typically,
> they
> > > now use KeyedProcessFunction or customized triggers to implement these
> > > aggregations. It's not easy to address data skew in such cases with a
> > > composited interface for local aggregation.
> > >
> > > Given that Data Stream API is exactly targeted at these cases where the
> > > application logic is very complicated and optimization does not
> matter, I
> > > think it's a better choice to provide a relatively low-level and
> > canonical
> > > interface.
> > >
> > > The composited interface, on the other side, may be a good choice in
> > > declarative interfaces, including SQL and Table API, as it allows more
> > > optimization opportunities.
> > >
> > > Best,
> > > Vino
> > >
> > >
> > > Kurt Young  于2019年6月20日周四 上午10:15写道：
> > >
> > >> Hi all,
> > >>
> > >> As vino said in previous emails, I think we should first discuss and
> > >> decide
> > >> what kind of use cases this FLIP want to
> > >> resolve, and what the API should look like. From my side, I think this
> > is
> > >> probably the root cause of current divergence.
> > >>
> > >> My understand is (from the FLIP title and motivation section of the
> > >> document), we want to have a proper support of
> > >> local aggregation, or pre aggregation. This is not a very new idea,
> most
> > >> SQL engine already did this improvement. And
> > >> the core concept about this is, there should be an AggregateFunction,
> no
> > >> matter it's a Flink runtime's AggregateFunction or
> > >> SQL's UserDefinedAggregateFunction. Both aggregation have concept of
> > >> intermediate data type, sometimes we call it ACC.
> > >> I quickly went through the POC piotr did before [1], it also directly
> > uses
> > >> AggregateFunction.
> > >>
> > >> But

[jira] [Created] (FLINK-12960) Move ResultPartitionDeploymentDescriptor#releasedOnConsumption to PartitionDescriptor#releasedOnConsumption

[Andrey Zagrebin (JIRA)]

Andrey Zagrebin created FLINK-12960:
---

 Summary: Move 
ResultPartitionDeploymentDescriptor#releasedOnConsumption to 
PartitionDescriptor#releasedOnConsumption
 Key: FLINK-12960
 URL: https://issues.apache.org/jira/browse/FLINK-12960
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.9.0


ResultPartitionDeploymentDescriptor#releasedOnConsumption shows the intention 
how the partition is going to be used by the shuffle user. If it is not 
supported by the shuffle service for a certain type of partition, 
ShuffleMaster#registerPartitionWithProducer and 
ShuffleEnvironment#createResultPartitionWriters should throw an exception. 
ShuffleMaster#registerPartitionWithProducer takes PartitionDescriptor. 
ResultPartitionDeploymentDescriptor#releasedOnConsumption should be part of 
PartitionDescriptor so that not only ShuffleEnvironment but also ShuffleMaster 
is already aware about releasedOnConsumption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[Kurt Young]

Hi vino,

I think there are several things still need discussion.

a) We all agree that we should first go with a unified abstraction, but the
abstraction is not reflected by the FLIP.
If your answer is "locakKeyBy" API, then I would ask how do we combine with
`AggregateFunction`, and how do
we do proper local aggregation for those have different intermediate result
type, like AVG. Could you add these
to the document?

b) From implementation side, reusing window operator is one of the possible
solutions, but not we base on window
operator to have two different implementations. What I understanding is,
one of the possible implementations should
not touch window operator.

c) 80% of your FLIP content is actually describing how do we support local
keyed state. I don't know if this is necessary
to introduce at the first step and we should also involve committers work
on state backend to share their thoughts.

Best,
Kurt


On Mon, Jun 24, 2019 at 5:17 PM vino yang  wrote:

> Hi Kurt,
>
> You did not give more further different opinions, so I thought you have
> agreed with the design after we promised to support two kinds of
> implementation.
>
> In API level, we have answered your question about pass an
> AggregateFunction to do the aggregation. No matter introduce localKeyBy API
> or not, we can support AggregateFunction.
>
> So what's your different opinion now? Can you share it with us?
>
> Best,
> Vino
>
> Kurt Young  于2019年6月24日周一 下午4:24写道：
>
> > Hi vino,
> >
> > Sorry I don't see the consensus about reusing window operator and keep
> the
> > API design of localKeyBy. But I think we should definitely more thoughts
> > about this topic.
> >
> > I also try to loop in Stephan for this discussion.
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, Jun 24, 2019 at 3:26 PM vino yang  wrote:
> >
> > > Hi all,
> > >
> > > I am happy we have a wonderful discussion and received many valuable
> > > opinions in the last few days.
> > >
> > > Now, let me try to summarize what we have reached consensus about the
> > > changes in the design.
> > >
> > >- provide a unified abstraction to support two kinds of
> > implementation;
> > >- reuse WindowOperator and try to enhance it so that we can make the
> > >intermediate result of the local aggregation can be buffered and
> > > flushed to
> > >support two kinds of implementation;
> > >- keep the API design of localKeyBy, but declare the disabled some
> > APIs
> > >we cannot support currently, and provide a configurable API for
> users
> > to
> > >choose how to handle intermediate result;
> > >
> > > The above three points have been updated in the design doc. Any
> > > questions, please let me know.
> > >
> > > @Aljoscha Krettek  What do you think? Any further
> > > comments?
> > >
> > > Best,
> > > Vino
> > >
> > > vino yang  于2019年6月20日周四 下午2:02写道：
> > >
> > > > Hi Kurt,
> > > >
> > > > Thanks for your comments.
> > > >
> > > > It seems we come to a consensus that we should alleviate the
> > performance
> > > > degraded by data skew with local aggregation. In this FLIP, our key
> > > > solution is to introduce local keyed partition to achieve this goal.
> > > >
> > > > I also agree that we can benefit a lot from the usage of
> > > > AggregateFunction. In combination with localKeyBy, We can easily use
> it
> > > to
> > > > achieve local aggregation:
> > > >
> > > >- input.localKeyBy(0).aggregate()
> > > >- input.localKeyBy(0).window().aggregate()
> > > >
> > > >
> > > > I think the only problem here is the choices between
> > > >
> > > >- (1) Introducing a new primitive called localKeyBy and implement
> > > >local aggregation with existing operators, or
> > > >- (2) Introducing an operator called localAggregation which is
> > > >composed of a key selector, a window-like operator, and an
> aggregate
> > > >function.
> > > >
> > > >
> > > > There may exist some optimization opportunities by providing a
> > composited
> > > > interface for local aggregation. But at the same time, in my opinion,
> > we
> > > > lose flexibility (Or we need certain efforts to achieve the same
> > > > flexibility).
> > > >
> > > > As said in the previous mails, we have many use cases where the
> > > > aggregation is very complicated and cannot be performed with
> > > > AggregateFunction. For example, users may perform windowed
> aggregations
> > > > according to time, data values, or even external storage. Typically,
> > they
> > > > now use KeyedProcessFunction or customized triggers to implement
> these
> > > > aggregations. It's not easy to address data skew in such cases with a
> > > > composited interface for local aggregation.
> > > >
> > > > Given that Data Stream API is exactly targeted at these cases where
> the
> > > > application logic is very complicated and optimization does not
> > matter, I
> > > > think it's a better choice to provide a relatively low-level and
> > > canonical
> > > > interface.
> > > >
> > > > The composited interface

Re: [DISCUSS] Ground Source and Sink Concepts in Flink SQL

[Hequn Cheng]

Hi Jark,

Impressive document!
I have gone over the document quickly and left some comments. I will have a
detailed look later. Below are two main thoughts from my side:

1. In the TableSource interface, can we move the getBoundedness() method
into the underneath Source?
This brings some benefits like we don't have to add `boundedSource()` to
the env in FLIP-27 and it can also be used in the Table API level. We may
also need to target FLIP-27 for the Flink 1.10 and coordinate these two big
design.

2. How are we going to address the compatible problem?
Are we going to add a totally new TableSource class or made some compatible
design? Maybe a new TableSource class is better? as we change the interface
somehow big.

What do you think?

Best, Hequn


On Mon, Jun 24, 2019 at 3:29 PM Jark Wu  wrote:

> Thanks Timo,
>
> I think it's fine to target it for Flink 1.10.  Looking forward for your
> feedback.
>
> On Mon, 24 Jun 2019 at 15:07, Timo Walther  wrote:
>
> > Thanks for working on this great design document Jark. I think having
> > well-defined terminilogy and semantics around tables, changelogs, table
> > sources/sinks, and DDL should have been done much earlier. I will take a
> > closer look at the concepts and give feedback soon. I think having those
> > concepts defined and implemented should be the goal for Flink 1.10. It
> > also allows us to align it to the efforts of FLIP-27.
> >
> > Introducing a DDL is a step that cannot be evolved easily as a DDL is
> > basically just a string that is being parsed. We should aim to involve
> > as many people as possible to have a future-proof design.
> >
> > Thanks,
> > Timo
> >
> > Am 27.05.19 um 10:40 schrieb Kurt Young:
> > > Thanks Jark for bringing this topic. I think proper concepts is very
> > > important for users who are using Table API & SQL. Especially for
> > > them to have a clear understanding about the behavior of the SQL job.
> > Also
> > > this is essential for connector developers to have a better
> > > understanding why we abstracted the interfaces in this way, and have a
> > > smooth experience when developing connectors for Table & SQL.
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Mon, May 27, 2019 at 3:35 PM Jark Wu  wrote:
> > >
> > >> Hi all,
> > >>
> > >> We have prepared a design doc [1] about source and sink concepts in
> > Flink
> > >> SQL. This is actually an extended discussion about SQL DDL [2].
> > >>
> > >> In the design doc, we want to figure out some concept problems. For
> > >> examples:
> > >>
> > >> 1. How to define boundedness in DDL
> > >> 2. How to define a changelog in DDL, what's the behavior of a
> changelog
> > >> source and changelog sink?
> > >> 3. How to define primary key in DDL and what's the semantic when we
> > have a
> > >> primary key on a table and stream?
> > >>
> > >> They are mostly related to DDL because DDL is plain text and we need
> to
> > >> keep close to standard as much as possible.
> > >>
> > >> This is an important step before we starting to refactor our
> > >> TableSource/TableSink/TableFactory interfaces. Because we need to know
> > what
> > >> changes we need to introduce to support these concepts.
> > >>
> > >> Please feel free to leave feedbacks in the thread or the design doc.
> > >>
> > >> Regards,
> > >> Jark
> > >>
> > >> [1].
> > >>
> > >>
> >
> https://docs.google.com/document/d/1yrKXEIRATfxHJJ0K3t6wUgXAtZq8D-XgvEnvl2uUcr0/edit#
> > >> [2].
> > >>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-SQL-DDL-Design-tt25006.html
> > >>
> >
> >
>

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[Kurt Young]

Hi vino,

One thing to add,  for a), I think use one or two examples like how to do
local aggregation on a sliding window,
and how do we do local aggregation on an unbounded aggregate, will do a lot
help.

Best,
Kurt


On Mon, Jun 24, 2019 at 6:06 PM Kurt Young  wrote:

> Hi vino,
>
> I think there are several things still need discussion.
>
> a) We all agree that we should first go with a unified abstraction, but
> the abstraction is not reflected by the FLIP.
> If your answer is "locakKeyBy" API, then I would ask how do we combine
> with `AggregateFunction`, and how do
> we do proper local aggregation for those have different intermediate
> result type, like AVG. Could you add these
> to the document?
>
> b) From implementation side, reusing window operator is one of the
> possible solutions, but not we base on window
> operator to have two different implementations. What I understanding is,
> one of the possible implementations should
> not touch window operator.
>
> c) 80% of your FLIP content is actually describing how do we support local
> keyed state. I don't know if this is necessary
> to introduce at the first step and we should also involve committers work
> on state backend to share their thoughts.
>
> Best,
> Kurt
>
>
> On Mon, Jun 24, 2019 at 5:17 PM vino yang  wrote:
>
>> Hi Kurt,
>>
>> You did not give more further different opinions, so I thought you have
>> agreed with the design after we promised to support two kinds of
>> implementation.
>>
>> In API level, we have answered your question about pass an
>> AggregateFunction to do the aggregation. No matter introduce localKeyBy
>> API
>> or not, we can support AggregateFunction.
>>
>> So what's your different opinion now? Can you share it with us?
>>
>> Best,
>> Vino
>>
>> Kurt Young  于2019年6月24日周一 下午4:24写道：
>>
>> > Hi vino,
>> >
>> > Sorry I don't see the consensus about reusing window operator and keep
>> the
>> > API design of localKeyBy. But I think we should definitely more thoughts
>> > about this topic.
>> >
>> > I also try to loop in Stephan for this discussion.
>> >
>> > Best,
>> > Kurt
>> >
>> >
>> > On Mon, Jun 24, 2019 at 3:26 PM vino yang 
>> wrote:
>> >
>> > > Hi all,
>> > >
>> > > I am happy we have a wonderful discussion and received many valuable
>> > > opinions in the last few days.
>> > >
>> > > Now, let me try to summarize what we have reached consensus about the
>> > > changes in the design.
>> > >
>> > >- provide a unified abstraction to support two kinds of
>> > implementation;
>> > >- reuse WindowOperator and try to enhance it so that we can make
>> the
>> > >intermediate result of the local aggregation can be buffered and
>> > > flushed to
>> > >support two kinds of implementation;
>> > >- keep the API design of localKeyBy, but declare the disabled some
>> > APIs
>> > >we cannot support currently, and provide a configurable API for
>> users
>> > to
>> > >choose how to handle intermediate result;
>> > >
>> > > The above three points have been updated in the design doc. Any
>> > > questions, please let me know.
>> > >
>> > > @Aljoscha Krettek  What do you think? Any
>> further
>> > > comments?
>> > >
>> > > Best,
>> > > Vino
>> > >
>> > > vino yang  于2019年6月20日周四 下午2:02写道：
>> > >
>> > > > Hi Kurt,
>> > > >
>> > > > Thanks for your comments.
>> > > >
>> > > > It seems we come to a consensus that we should alleviate the
>> > performance
>> > > > degraded by data skew with local aggregation. In this FLIP, our key
>> > > > solution is to introduce local keyed partition to achieve this goal.
>> > > >
>> > > > I also agree that we can benefit a lot from the usage of
>> > > > AggregateFunction. In combination with localKeyBy, We can easily
>> use it
>> > > to
>> > > > achieve local aggregation:
>> > > >
>> > > >- input.localKeyBy(0).aggregate()
>> > > >- input.localKeyBy(0).window().aggregate()
>> > > >
>> > > >
>> > > > I think the only problem here is the choices between
>> > > >
>> > > >- (1) Introducing a new primitive called localKeyBy and implement
>> > > >local aggregation with existing operators, or
>> > > >- (2) Introducing an operator called localAggregation which is
>> > > >composed of a key selector, a window-like operator, and an
>> aggregate
>> > > >function.
>> > > >
>> > > >
>> > > > There may exist some optimization opportunities by providing a
>> > composited
>> > > > interface for local aggregation. But at the same time, in my
>> opinion,
>> > we
>> > > > lose flexibility (Or we need certain efforts to achieve the same
>> > > > flexibility).
>> > > >
>> > > > As said in the previous mails, we have many use cases where the
>> > > > aggregation is very complicated and cannot be performed with
>> > > > AggregateFunction. For example, users may perform windowed
>> aggregations
>> > > > according to time, data values, or even external storage. Typically,
>> > they
>> > > > now use KeyedProcessFunction or customized triggers to implement
>> these
>> > > > a

[jira] [Created] (FLINK-12961) StreamExecutionEnvironment supports executing job with StreamGraph

[Biao Liu (JIRA)]

Biao Liu created FLINK-12961:


 Summary: StreamExecutionEnvironment supports executing job with 
StreamGraph
 Key: FLINK-12961
 URL: https://issues.apache.org/jira/browse/FLINK-12961
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Biao Liu
Assignee: Biao Liu
 Fix For: 1.9.0


Expose an internal method {{execute(StreamGraph)}} of 
{{StreamExecutionEnvironment}}. So the pluggable runner could get a chance to 
set properties of {{StreamGraph}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12962) Allows pyflink to be pip installed

[Dian Fu (JIRA)]

Dian Fu created FLINK-12962:
---

 Summary: Allows pyflink to be pip installed
 Key: FLINK-12962
 URL: https://issues.apache.org/jira/browse/FLINK-12962
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Dian Fu


The aim of this JIRA is to support to build a pip installable package.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[Shu Su]

Hi vino


Thanks for proposal.
For Local Aggregation I have a question about doing this in window aggregation. 
As we know , window aggregation like sliding window should based on 
Time trigger, and there may exists a problem in event time if we do local 
aggregation. For example if I want to do a 5s sliding window with count agg:


1. I have input with 4 parallelism and data are firstly randomly pass in 4 
partitions.
2. We do LocalAggregation in each of them and we get a partial count result.
3. Forward partial result to a node with same key then do the final aggregation.


It seems no problem but what will happen if data skew in event time ? If we 
have a continuous time sequence in 3 of 4 input partitions, for example , we 
have a continuous time sequence in partition 1, 2, 3 but data to partition 4 
was delay for some reason, and we just get 3 partial result for the moment, 
does final aggregation need to wait for the 4th partial result because of data 
delay ? If so , how long we need to wait for ? If not, does it mean that 
The final aggregation will wait forever ? 


Thanks,
Simon


On 06/18/2019 10:06，vino yang wrote：
Hi Jark,

We have done a comparative test. The effect is obvious.

From our observation, the optimized effect mainly depends on two factors:


- the degree of the skew: this factor depends on users business ;
- the size of the window: localKeyBy support all the type of window
which provided by Flink. Obviously, the larger the size of the window, the
more obvious the effect.

In production, we can not decide the first factor. About the second factor,
it's the result of a trade-off. The size of the window affects the latency
of the pre-aggregation. That's to say:


- the larger the size of the window, the more obvious the effect;
- the larger the size of the window, the larger latency of the result

Best,
Vino

Jark Wu  于2019年6月17日周一 下午7:32写道：

Hi Vino,

Thanks for the proposal.

Regarding to the "input.keyBy(0).sum(1)" vs
"input.localKeyBy(0).countWindow(5).sum(1).keyBy(0).sum(1)", have you done
some benchmark?
Because I'm curious about how much performance improvement can we get by
using count window as the local operator.

Best,
Jark



On Mon, 17 Jun 2019 at 17:48, vino yang  wrote:

Hi Hequn,

Thanks for your reply.

The purpose of localKeyBy API is to provide a tool which can let users do
pre-aggregation in the local. The behavior of the pre-aggregation is
similar to keyBy API.

So the three cases are different, I will describe them one by one:

1. input.keyBy(0).sum(1)

*In this case, the result is event-driven, each event can produce one sum
aggregation result and it is the latest one from the source start.*

2. input.localKeyBy(0).sum(1).keyBy(0).sum(1)

*In this case, the semantic may have a problem, it would do the local sum
aggregation and will produce the latest partial result from the source
start for every event. *
*These latest partial results from the same key are hashed to one node to
do the global sum aggregation.*
*In the global aggregation, when it received multiple partial results
(they
are all calculated from the source start) and sum them will get the wrong
result.*

3. input.localKeyBy(0).countWindow(5).sum(1).keyBy(0).sum(1)

*In this case, it would just get a partial aggregation result for the 5
records in the count window. The partial aggregation results from the
same
key will be aggregated globally.*

So the first case and the third case can get the *same* result, the
difference is the output-style and the latency.

Generally speaking, the local key API is just an optimization API. We do
not limit the user's usage, but the user has to understand its semantics
and use it correctly.

Best,
Vino

Hequn Cheng  于2019年6月17日周一 下午4:18写道：

Hi Vino,

Thanks for the proposal, I think it is a very good feature!

One thing I want to make sure is the semantics for the `localKeyBy`.
From
the document, the `localKeyBy` API returns an instance of `KeyedStream`
which can also perform sum(), so in this case, what's the semantics for
`localKeyBy()`. For example, will the following code share the same
result?
and what're the differences between them?

1. input.keyBy(0).sum(1)
2. input.localKeyBy(0).sum(1).keyBy(0).sum(1)
3. input.localKeyBy(0).countWindow(5).sum(1).keyBy(0).sum(1)

Would also be great if we can add this into the document. Thank you
very
much.

Best, Hequn


On Fri, Jun 14, 2019 at 11:34 AM vino yang 
wrote:

Hi Aljoscha,

I have looked at the "*Process*" section of FLIP wiki page.[1] This
mail
thread indicates that it has proceeded to the third step.

When I looked at the fourth step(vote step), I didn't find the
prerequisites for starting the voting process.

Considering that the discussion of this feature has been done in the
old
thread. [2] So can you tell me when should I start voting? Can I
start
now?

Best,
Vino

[1]:




https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-FLIPround-up
[2]:




http://ap

Re: [DISCUSS] A more restrictive JIRA workflow

[Robert Metzger]

Hey all,

I would like to drive this discussion to an end soon.
I've just merged the updated contribution guide to the Flink website:
https://flink.apache.org/contributing/contribute-code.html

I will now ask Apache IINFRA to change the permissions in our Jira.

Here's the updated TODO list:

1. I update the contribution guide DONE
2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
unassigned JIRAs IN PROGRESS
3. We ask Infra to change the permissions of our JIRA so that: IN PROGRESS
  a) only committers can assign users to tickets
  b) contributors can't assign users to tickets
  c) Every registered JIRA user is an assignable user in FLINK





On Fri, May 24, 2019 at 9:18 AM Robert Metzger  wrote:

> Hey,
>
> I started working on step 1 and proposed some changes to the Flink
> website: https://github.com/apache/flink-web/pull/217
>
>
>
> On Tue, Apr 30, 2019 at 4:08 PM Robert Metzger 
> wrote:
>
>> Hi Fabian,
>> You are right, I made a mistake. I don't think it makes sense to
>> introduce a new permission class. This will make the life of JIRA admins
>> unnecessarily complicated.
>> I updated the task list:
>>
>> 1. I update the contribution guide
>> 2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
>> unassigned JIRAs
>> 3. We ask Infra to change the permissions of our JIRA so that:
>>   a) only committers can assign users to tickets
>>   b) contributors can't assign users to tickets
>>   c) Every registered JIRA user is an assignable user in FLINK
>> 4. We remove all existing contributors
>>
>>
>> On Tue, Apr 30, 2019 at 12:00 PM Fabian Hueske  wrote:
>>
>>> Hi Robert,
>>>
>>> If I understood the decision correctly, we also need to ask Infra to make
>>> everybody an assignable user, right?
>>> Or do we want to add a new permission class "Assignable User" such that
>>> everyone still needs to ask for the right Jira permissions?
>>>
>>> Fabian
>>>
>>>
>>> Am Di., 30. Apr. 2019 um 10:46 Uhr schrieb Timo Walther <
>>> twal...@apache.org
>>> >:
>>>
>>> > Hi Robert,
>>> >
>>> > thanks for taking care of this. +1 to your suggested steps.
>>> >
>>> > Regards,
>>> > Timo
>>> >
>>> >
>>> > Am 30.04.19 um 10:42 schrieb Robert Metzger:
>>> > > @Stephan: I agree. Auto-closing PRs is quite aggressive.
>>> > > I will only do that for PRs without JIRA ID or "[hotfix]" in the
>>> title.
>>> > > We can always revisit this at a later stage.
>>> > >
>>> > >
>>> > > I'm proposing the following steps:
>>> > >
>>> > > 1. I update the contribution guide
>>> > > 2. Update Flinkbot to close invalid PRs, and show warnings on PRs
>>> with
>>> > > unassigned JIRAs
>>> > > 3. We ask Infra to change the permissions of our JIRA so that:
>>> > >a) only committers can assign users to tickets
>>> > >b) contributors can't assign users to tickets
>>> > > 4. We remove all existing contributors
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Wed, Apr 24, 2019 at 11:17 AM vino yang 
>>> > wrote:
>>> > >
>>> > >> also +1 for option 2.
>>> > >>
>>> > >> I think auto-close a PR sometimes a bit impertinency.
>>> > >> The reasons just like Stephan mentioned.
>>> > >>
>>> > >> Stephan Ewen  于2019年4月24日周三 下午4:08写道：
>>> > >>
>>> > >>> About auto-closing PRs from unassigned issues, consider the
>>> following
>>> > >> case
>>> > >>> that has happened quite a bit.
>>> > >>>
>>> > >>>- a user reports a small bug and immediately wants to provide a
>>> fix
>>> > for
>>> > >>> it
>>> > >>>- it makes sense to not stall the user for a few days until a
>>> > committer
>>> > >>> assigned the issue
>>> > >>>- not auto-closing the PR would at least allow the user to
>>> provide
>>> > >> their
>>> > >>> patch.
>>> > >>>
>>> > >>> On Wed, Apr 24, 2019 at 10:00 AM Stephan Ewen 
>>> > wrote:
>>> > >>>
>>> >  +1 for option #2
>>> > 
>>> >  Seems to me that this does not contradict option #1, it only
>>> extends
>>> > >> this
>>> >  a bit. I think there is a good case for that, to help frequent
>>> > >>> contributors
>>> >  on a way to committership.
>>> > 
>>> >  @Konstantin: Trivial fixes (typos, docs, javadocs, ...) should
>>> still
>>> > be
>>> >  possible as "hotfixes".
>>> > 
>>> >  On Mon, Apr 15, 2019 at 3:14 PM Timo Walther 
>>> > >> wrote:
>>> > > I think this really depends on the contribution.
>>> > >
>>> > > Sometimes "triviality" means that people just want to fix a typo
>>> in
>>> > >> some
>>> > > docs. For this, a hotfix PR is sufficient and does not need a
>>> JIRA
>>> > >>> issue.
>>> > > However, sometimes "triviality" is only trivial at first glance
>>> but
>>> > > introduces side effects. In any case, any contribution needs to
>>> be
>>> > > reviewed and merged by a committer so follow-up responses and
>>> > >> follow-up
>>> > > work might always be required. But you are right, committers
>>> need to
>>> > > respond quicker in any case.
>>> > >
>>> > > Timo
>>> > >
>>> > >
>>> > > Am 15.04.19 um 1

[ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Robert Metzger]

Hi all,

On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
part of the Apache Flink Project Management Committee (PMC).

Jincheng has been a committer since July 2017. He has been very active on
Flink's Table API / SQL component, as well as helping with releases.

Congratulations & Welcome Jincheng!

Best,
Robert

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Dian Fu]

Congratulations Jincheng!

On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger  wrote:

> Hi all,
>
> On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
> part of the Apache Flink Project Management Committee (PMC).
>
> Jincheng has been a committer since July 2017. He has been very active on
> Flink's Table API / SQL component, as well as helping with releases.
>
> Congratulations & Welcome Jincheng!
>
> Best,
> Robert
>

[jira] [Created] (FLINK-12963) Add savepoint writer for bootstrapping new savepoints

[Seth Wiesman (JIRA)]

Seth Wiesman created FLINK-12963:


 Summary: Add savepoint writer for bootstrapping new savepoints
 Key: FLINK-12963
 URL: https://issues.apache.org/jira/browse/FLINK-12963
 Project: Flink
  Issue Type: Sub-task
  Components: API / DataStream
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Implement a savepoint writer for bootstrapping new savepoints.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Hequn Cheng]

Congratulations Jincheng!

Best, Hequn

On Mon, Jun 24, 2019 at 11:43 PM Dian Fu  wrote:

> Congratulations Jincheng!
>
> On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger 
> wrote:
>
> > Hi all,
> >
> > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> now
> > part of the Apache Flink Project Management Committee (PMC).
> >
> > Jincheng has been a committer since July 2017. He has been very active on
> > Flink's Table API / SQL component, as well as helping with releases.
> >
> > Congratulations & Welcome Jincheng!
> >
> > Best,
> > Robert
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Zili Chen]

Congratulations Jincheng!

Best,
tison.


Hequn Cheng  于2019年6月24日周一 下午11:48写道：

> Congratulations Jincheng!
>
> Best, Hequn
>
> On Mon, Jun 24, 2019 at 11:43 PM Dian Fu  wrote:
>
> > Congratulations Jincheng!
> >
> > On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger 
> > wrote:
> >
> > > Hi all,
> > >
> > > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> > now
> > > part of the Apache Flink Project Management Committee (PMC).
> > >
> > > Jincheng has been a committer since July 2017. He has been very active
> on
> > > Flink's Table API / SQL component, as well as helping with releases.
> > >
> > > Congratulations & Welcome Jincheng!
> > >
> > > Best,
> > > Robert
> > >
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Yu Li]

Congratulations Jincheng! Well deserved!

Best Regards,
Yu


On Tue, 25 Jun 2019 at 00:01, Zili Chen  wrote:

> Congratulations Jincheng!
>
> Best,
> tison.
>
>
> Hequn Cheng  于2019年6月24日周一 下午11:48写道：
>
> > Congratulations Jincheng!
> >
> > Best, Hequn
> >
> > On Mon, Jun 24, 2019 at 11:43 PM Dian Fu  wrote:
> >
> > > Congratulations Jincheng!
> > >
> > > On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun
> is
> > > now
> > > > part of the Apache Flink Project Management Committee (PMC).
> > > >
> > > > Jincheng has been a committer since July 2017. He has been very
> active
> > on
> > > > Flink's Table API / SQL component, as well as helping with releases.
> > > >
> > > > Congratulations & Welcome Jincheng!
> > > >
> > > > Best,
> > > > Robert
> > > >
> > >
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Yun Tang]

Congratulations Jincheng!

Best
Yun Tang

From: Yu Li 
Sent: Tuesday, June 25, 2019 0:07
To: dev
Subject: Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

Congratulations Jincheng! Well deserved!

Best Regards,
Yu


On Tue, 25 Jun 2019 at 00:01, Zili Chen  wrote:

> Congratulations Jincheng!
>
> Best,
> tison.
>
>
> Hequn Cheng  于2019年6月24日周一 下午11:48写道：
>
> > Congratulations Jincheng!
> >
> > Best, Hequn
> >
> > On Mon, Jun 24, 2019 at 11:43 PM Dian Fu  wrote:
> >
> > > Congratulations Jincheng!
> > >
> > > On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun
> is
> > > now
> > > > part of the Apache Flink Project Management Committee (PMC).
> > > >
> > > > Jincheng has been a committer since July 2017. He has been very
> active
> > on
> > > > Flink's Table API / SQL component, as well as helping with releases.
> > > >
> > > > Congratulations & Welcome Jincheng!
> > > >
> > > > Best,
> > > > Robert
> > > >
> > >
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Bowen Li]

Congratulations!

On Mon, Jun 24, 2019 at 9:53 AM Yun Tang  wrote:

> Congratulations Jincheng!
>
> Best
> Yun Tang
> 
> From: Yu Li 
> Sent: Tuesday, June 25, 2019 0:07
> To: dev
> Subject: Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC
>
> Congratulations Jincheng! Well deserved!
>
> Best Regards,
> Yu
>
>
> On Tue, 25 Jun 2019 at 00:01, Zili Chen  wrote:
>
> > Congratulations Jincheng!
> >
> > Best,
> > tison.
> >
> >
> > Hequn Cheng  于2019年6月24日周一 下午11:48写道：
> >
> > > Congratulations Jincheng!
> > >
> > > Best, Hequn
> > >
> > > On Mon, Jun 24, 2019 at 11:43 PM Dian Fu 
> wrote:
> > >
> > > > Congratulations Jincheng!
> > > >
> > > > On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger  >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun
> > is
> > > > now
> > > > > part of the Apache Flink Project Management Committee (PMC).
> > > > >
> > > > > Jincheng has been a committer since July 2017. He has been very
> > active
> > > on
> > > > > Flink's Table API / SQL component, as well as helping with
> releases.
> > > > >
> > > > > Congratulations & Welcome Jincheng!
> > > > >
> > > > > Best,
> > > > > Robert
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Feature freeze for Apache Flink 1.9.0 release

[Bowen Li]

Hi Gordon,

Thanks for driving this effort.

Xuefu responded to the discussion thread [1] and I want to bring that to
our attention here:

Hive integration depends on a few features that are actively developed. If
the completion of those features don't leave enough time for us to
integrate, then our work can potentially go beyond the proposed date.

Just wanted to point out such a dependency adds uncertainty.

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html

On Thu, Jun 20, 2019 at 1:05 AM Tzu-Li (Gordon) Tai 
wrote:

> Hi devs,
>
> Per the feature discussions for 1.9.0 [1], I hereby announce the official
> feature freeze for Flink 1.9.0 to be on June 28. A release feature branch
> for 1.9 will be cut following that date.
>
> We’re roughly one week away from this date, but please keep in mind that we
> still shouldn’t rush things. If you feel that there may be problems with
> this schedule for the things you are working on, please let us know here.
>
> Cheers,
> Gordon
>
> [1]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html
>

Re: [DISCUSS] Adopting a Code Style and Quality Guide

[Stephan Ewen]

Thanks for the pointer David.

I was not aware of this tool and I have no experience with its practical
impact. For example I would be curious what the effect of this is for
existing PRs, merge conflicts, etc.

If you want, feel free to start another discuss thread on the introduction
of such a tool.

On Sun, Jun 23, 2019 at 6:32 PM David Morávek  wrote:

> I love this kind of unification being pushed forward, the benefit it has on
> code reviews is enormous. Just my two cents, did you guys think about
> adopting an automatic code formatter for java, the same way as Apache Beam
> did?
>
> It is super easy to use for contributors as they don't need to keep any
> particular coding style in mind and they can only focus on functionality
> they want to fix, and it's also great for reviewers, because they only see
> the important changes. This also eliminates need for any special editor /
> checkstyle configs as the code formatting is part of the build itself.
>
> The one Beam uses is https://github.com/diffplug/spotless with
> GoogleJavaFormat, it may be worth to look into.
>
> Best,
> David
>
> On Fri, Jun 21, 2019 at 4:40 PM Stephan Ewen  wrote:
>
> > Thanks, everyone, for the positive feedback :-)
> >
> > @Robert - It probably makes sense to break this down into various pages,
> > like PR, general code style guide, Java, component specific guides,
> > formats, etc.
> >
> > Best,
> > Stephan
> >
> >
> > On Fri, Jun 21, 2019 at 4:29 PM Robert Metzger 
> > wrote:
> >
> > > It seems that the discussion around this topic has settled.
> > >
> > > I'm going to turn the Google Doc into a markdown file (maybe also
> > multiple,
> > > I'll try out different things) and then open a pull request for the
> Flink
> > > website.
> > > I'll post a link to the PR here once I'm done.
> > >
> > > On Fri, Jun 14, 2019 at 9:36 AM zhijiang  > > .invalid>
> > > wrote:
> > >
> > > > Thanks for providing this useful guide for benefiting both
> contributors
> > > > and committers in consistency.
> > > >
> > > > I just reviewed and learned the whole doc which covers a lot of
> > > > information. Wish it further categoried and put onto somewhere for
> > easily
> > > > traced future.
> > > >
> > > > Best,
> > > > Zhijiang
> > > > --
> > > > From:Robert Metzger 
> > > > Send Time:2019年6月14日(星期五) 14:24
> > > > To:dev 
> > > > Subject:Re: [DISCUSS] Adopting a Code Style and Quality Guide
> > > >
> > > > Thanks a lot for putting this together!
> > > >
> > > > I'm in the process of reworking the "How to contribute" pages [1] and
> > I'm
> > > > happy to add the guide to the Flink website, once the discussion here
> > is
> > > > over.
> > > >
> > > > [1] https://github.com/apache/flink-web/pull/217
> > > >
> > > > On Fri, Jun 14, 2019 at 3:21 AM Kurt Young  wrote:
> > > >
> > > > > Big +1 and thanks for preparing this.
> > > > >
> > > > > I think wha't more important is making sure most all the
> contributors
> > > can
> > > > > follow
> > > > > the same guide, a clear document is definitely a great start.
> > > Committers
> > > > > can
> > > > > first try to follow the guide by self, and spread the standard
> during
> > > > code
> > > > > reviewing.
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Thu, Jun 13, 2019 at 8:28 PM Congxian Qiu <
> qcx978132...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > +1 for this, I think all contributors can benefit from this.
> > > > > >
> > > > > > Best,
> > > > > > Congxian
> > > > > >
> > > > > >
> > > > > > Aljoscha Krettek  于2019年6月13日周四 下午8:14写道：
> > > > > >
> > > > > > > +1 I think this is a very good effort and should put to rest
> some
> > > > > > > back-and-forth discussions on PRs and some differences in
> “style”
> > > > > between
> > > > > > > committers. ;-)
> > > > > > >
> > > > > > > > On 13. Jun 2019, at 10:21, JingsongLee <
> > lzljs3620...@aliyun.com
> > > > > > .INVALID>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > big +1, the content is very useful and enlightening.
> > > > > > > > But it's really too long to look at.
> > > > > > > > +1 for splitting it and expose it to contributors.
> > > > > > > >
> > > > > > > > Even I think it's possible to put its link on the default
> > > > description
> > > > > > of
> > > > > > > > pull request, so that the user has to see it when submits the
> > > code.
> > > > > > > >
> > > > > > > > Best, JingsongLee
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > --
> > > > > > > > From:Piotr Nowojski 
> > > > > > > > Send Time:2019年6月13日(星期四) 16:03
> > > > > > > > To:dev 
> > > > > > > > Subject:Re: [DISCUSS] Adopting a Code Style and Quality Guide
> > > > > > > >
> > > > > > > > +1 for it and general content and thank everybody that was
> > > involved
> > > > > in
> > > > > > > creating & writing this down.
> > > > > > > >
> > > > > > > > +1 for splitting it up into some easily naviga

Re: [DISCUSS] Adopting a Code Style and Quality Guide

[Stephan Ewen]

I think it makes sense to also start individual [DISCUSS] threads about
extensions and follow-ups.
Various suggestions came up, partly as comments in the doc, some as
questions in other threads.

Examples:
  - use of null in return types versus Optional versus @Nullable/@Nonnull
  - initialization of collection sizes
  - logging

I think these would be best discussed in separate threads each.
So, for contributors to whom these issues are dear, feel free to spawn
these additional threads.
(Bear in mind it is close to 1.9 feature freeze time, so please leave this
discussions a bit of time so that all community members have a chance to
participate)



On Mon, Jun 24, 2019 at 7:51 PM Stephan Ewen  wrote:

> Thanks for the pointer David.
>
> I was not aware of this tool and I have no experience with its practical
> impact. For example I would be curious what the effect of this is for
> existing PRs, merge conflicts, etc.
>
> If you want, feel free to start another discuss thread on the introduction
> of such a tool.
>
> On Sun, Jun 23, 2019 at 6:32 PM David Morávek  wrote:
>
>> I love this kind of unification being pushed forward, the benefit it has
>> on
>> code reviews is enormous. Just my two cents, did you guys think about
>> adopting an automatic code formatter for java, the same way as Apache Beam
>> did?
>>
>> It is super easy to use for contributors as they don't need to keep any
>> particular coding style in mind and they can only focus on functionality
>> they want to fix, and it's also great for reviewers, because they only see
>> the important changes. This also eliminates need for any special editor /
>> checkstyle configs as the code formatting is part of the build itself.
>>
>> The one Beam uses is https://github.com/diffplug/spotless with
>> GoogleJavaFormat, it may be worth to look into.
>>
>> Best,
>> David
>>
>> On Fri, Jun 21, 2019 at 4:40 PM Stephan Ewen  wrote:
>>
>> > Thanks, everyone, for the positive feedback :-)
>> >
>> > @Robert - It probably makes sense to break this down into various pages,
>> > like PR, general code style guide, Java, component specific guides,
>> > formats, etc.
>> >
>> > Best,
>> > Stephan
>> >
>> >
>> > On Fri, Jun 21, 2019 at 4:29 PM Robert Metzger 
>> > wrote:
>> >
>> > > It seems that the discussion around this topic has settled.
>> > >
>> > > I'm going to turn the Google Doc into a markdown file (maybe also
>> > multiple,
>> > > I'll try out different things) and then open a pull request for the
>> Flink
>> > > website.
>> > > I'll post a link to the PR here once I'm done.
>> > >
>> > > On Fri, Jun 14, 2019 at 9:36 AM zhijiang > > > .invalid>
>> > > wrote:
>> > >
>> > > > Thanks for providing this useful guide for benefiting both
>> contributors
>> > > > and committers in consistency.
>> > > >
>> > > > I just reviewed and learned the whole doc which covers a lot of
>> > > > information. Wish it further categoried and put onto somewhere for
>> > easily
>> > > > traced future.
>> > > >
>> > > > Best,
>> > > > Zhijiang
>> > > > --
>> > > > From:Robert Metzger 
>> > > > Send Time:2019年6月14日(星期五) 14:24
>> > > > To:dev 
>> > > > Subject:Re: [DISCUSS] Adopting a Code Style and Quality Guide
>> > > >
>> > > > Thanks a lot for putting this together!
>> > > >
>> > > > I'm in the process of reworking the "How to contribute" pages [1]
>> and
>> > I'm
>> > > > happy to add the guide to the Flink website, once the discussion
>> here
>> > is
>> > > > over.
>> > > >
>> > > > [1] https://github.com/apache/flink-web/pull/217
>> > > >
>> > > > On Fri, Jun 14, 2019 at 3:21 AM Kurt Young 
>> wrote:
>> > > >
>> > > > > Big +1 and thanks for preparing this.
>> > > > >
>> > > > > I think wha't more important is making sure most all the
>> contributors
>> > > can
>> > > > > follow
>> > > > > the same guide, a clear document is definitely a great start.
>> > > Committers
>> > > > > can
>> > > > > first try to follow the guide by self, and spread the standard
>> during
>> > > > code
>> > > > > reviewing.
>> > > > >
>> > > > > Best,
>> > > > > Kurt
>> > > > >
>> > > > >
>> > > > > On Thu, Jun 13, 2019 at 8:28 PM Congxian Qiu <
>> qcx978132...@gmail.com
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > +1 for this, I think all contributors can benefit from this.
>> > > > > >
>> > > > > > Best,
>> > > > > > Congxian
>> > > > > >
>> > > > > >
>> > > > > > Aljoscha Krettek  于2019年6月13日周四 下午8:14写道：
>> > > > > >
>> > > > > > > +1 I think this is a very good effort and should put to rest
>> some
>> > > > > > > back-and-forth discussions on PRs and some differences in
>> “style”
>> > > > > between
>> > > > > > > committers. ;-)
>> > > > > > >
>> > > > > > > > On 13. Jun 2019, at 10:21, JingsongLee <
>> > lzljs3620...@aliyun.com
>> > > > > > .INVALID>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > big +1, the content is very useful and enlightening.
>> > > > > > > > But it's really too long to look at.
>> > > > > > > > +1 f

[jira] [Created] (FLINK-12964) add commented-out defaults to sql client yaml file to make it easier for users to adopt

[Bowen Li (JIRA)]

Bowen Li created FLINK-12964:


 Summary: add commented-out defaults to sql client yaml file to 
make it easier for users to adopt
 Key: FLINK-12964
 URL: https://issues.apache.org/jira/browse/FLINK-12964
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Client
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.9.0


Add defaults for catalogs, similar to existing defaults of tables and 
functions, e.g.


{code:java}

tables: [] # empty list
# A typical table source definition looks like:
# - name: ...
#   type: source-table
#   connector: ...
#   format: ...
#   schema: ...

# A typical view definition looks like:
# - name: ...
#   type: view
#   query: "SELECT ..."

# A typical temporal table definition looks like:
# - name: ...
#   type: temporal-table
#   history-table: ...
#   time-attribute: ...
#   primary-key: ...

#==
# User-defined functions
#==

# Define scalar, aggregate, or table functions here.

functions: [] # empty list
# A typical function definition looks like:
# - name: ...
#   from: class
#   class: ...
#   constructor: ...
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12965) unify catalog view implementations

[Bowen Li (JIRA)]

Bowen Li created FLINK-12965:


 Summary: unify catalog view implementations
 Key: FLINK-12965
 URL: https://issues.apache.org/jira/browse/FLINK-12965
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive, Table SQL / API
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] solve unstable build capacity problem on TravisCI

[Bowen Li]

Hi devs,

I've been experiencing the pain resulting from lack of stable build
capacity on Travis for Flink PRs [1]. Specifically, I noticed often that no
build in the queue is making any progress for hours, and suddenly 5 or 6
builds kick off all together after the long pause. I'm at PST (UTC-08) time
zone, and I've seen pause can be as long as 6 hours from PST 9am to 3pm
(let alone the time needed to drain the queue afterwards).

I think this has greatly impacted our productivity. I've experienced that
PRs submitted in the early morning of PST time zone won't finish their
build until late night of the same day.

So my questions are:

- Has anyone else experienced the same problem or have similar observation
on TravisCI? (I suspect it has things to do with time zone)

- What pricing plan of TravisCI is Flink currently using? Is it the free
plan for open source projects? What are the guaranteed build capacity of
the current plan?

- If the current pricing plan (either free or paid) can't provide stable
build capacity, can we upgrade to a higher priced plan with larger and more
stable build capacity?

BTW, another factor that contribute to the productivity problem is that our
build is slow - we run full build for every PR and a successful full build
takes ~5h. We definitely have more options to solve it, for instance,
modularize the build graphs and reuse artifacts from the previous build.
But I think that can be a big effort which is much harder to accomplish in
a short period of time and may deserve its own separate discussion.

[1] https://travis-ci.org/apache/flink/pull_requests

[jira] [Created] (FLINK-12966) finalize package name of Hive table source/sink

[Bowen Li (JIRA)]

Bowen Li created FLINK-12966:


 Summary: finalize package name of Hive table source/sink
 Key: FLINK-12966
 URL: https://issues.apache.org/jira/browse/FLINK-12966
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Affects Versions: 1.9.0
Reporter: Bowen Li
Assignee: zjuwangg
 Fix For: 1.9.0


It's been brought by [~ykt836] and Jeff Zhang that the package name of 
`org.apache.flink.batch.connector` may not be proper.

I think @zjuwangg named it this way because most connector packages are named 
as org.apache.flink.streaming.connectors.xxx and he is just following the 
convention. However, as we are forwarding to streaming-batch unification, we 
probably don't need "streaming/batch" in the package names any more, coz, like 
file source/sink, hive source/sink can (doesn't mean we necessarily will) be 
made as streaming in the future. I'm thinking of just 
org.apache.flink.connectors.hive. What do you think?

cc [~ykt836] [~xuefuz] [~lirui]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Bowen Li]

https://travis-ci.org/apache/flink/builds/549681530  This build request has
been sitting at **HEAD of the queue** since I first saw it at PST 10:30am
(not sure how long it's been there before 10:30am). It's PST 4:12pm now and
it hasn't started yet.

On Mon, Jun 24, 2019 at 2:48 PM Bowen Li  wrote:

> Hi devs,
>
> I've been experiencing the pain resulting from lack of stable build
> capacity on Travis for Flink PRs [1]. Specifically, I noticed often that no
> build in the queue is making any progress for hours, and suddenly 5 or 6
> builds kick off all together after the long pause. I'm at PST (UTC-08) time
> zone, and I've seen pause can be as long as 6 hours from PST 9am to 3pm
> (let alone the time needed to drain the queue afterwards).
>
> I think this has greatly impacted our productivity. I've experienced that
> PRs submitted in the early morning of PST time zone won't finish their
> build until late night of the same day.
>
> So my questions are:
>
> - Has anyone else experienced the same problem or have similar observation
> on TravisCI? (I suspect it has things to do with time zone)
>
> - What pricing plan of TravisCI is Flink currently using? Is it the free
> plan for open source projects? What are the guaranteed build capacity of
> the current plan?
>
> - If the current pricing plan (either free or paid) can't provide stable
> build capacity, can we upgrade to a higher priced plan with larger and more
> stable build capacity?
>
> BTW, another factor that contribute to the productivity problem is that
> our build is slow - we run full build for every PR and a successful full
> build takes ~5h. We definitely have more options to solve it, for instance,
> modularize the build graphs and reuse artifacts from the previous build.
> But I think that can be a big effort which is much harder to accomplish in
> a short period of time and may deserve its own separate discussion.
>
> [1] https://travis-ci.org/apache/flink/pull_requests
>
>

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Steven Wu]

long and sometimes unstable build is definitely a pain point.

I suspect the build failure here in flink-connector-kafka is not related to
my change. but there is no easy re-run the build on travis UI. Google
search showed a trick of close-and-open the PR will trigger rebuild. but
that could add noises to the PR activities.
https://travis-ci.org/apache/flink/jobs/54519

travis-ci for my personal repo often failed with exceeding time limit after
4+ hours.
The job exceeded the maximum time limit for jobs, and has been terminated.

On Mon, Jun 24, 2019 at 4:15 PM Bowen Li  wrote:

> https://travis-ci.org/apache/flink/builds/549681530  This build request
> has
> been sitting at **HEAD of the queue** since I first saw it at PST 10:30am
> (not sure how long it's been there before 10:30am). It's PST 4:12pm now and
> it hasn't started yet.
>
> On Mon, Jun 24, 2019 at 2:48 PM Bowen Li  wrote:
>
> > Hi devs,
> >
> > I've been experiencing the pain resulting from lack of stable build
> > capacity on Travis for Flink PRs [1]. Specifically, I noticed often that
> no
> > build in the queue is making any progress for hours, and suddenly 5 or 6
> > builds kick off all together after the long pause. I'm at PST (UTC-08)
> time
> > zone, and I've seen pause can be as long as 6 hours from PST 9am to 3pm
> > (let alone the time needed to drain the queue afterwards).
> >
> > I think this has greatly impacted our productivity. I've experienced that
> > PRs submitted in the early morning of PST time zone won't finish their
> > build until late night of the same day.
> >
> > So my questions are:
> >
> > - Has anyone else experienced the same problem or have similar
> observation
> > on TravisCI? (I suspect it has things to do with time zone)
> >
> > - What pricing plan of TravisCI is Flink currently using? Is it the free
> > plan for open source projects? What are the guaranteed build capacity of
> > the current plan?
> >
> > - If the current pricing plan (either free or paid) can't provide stable
> > build capacity, can we upgrade to a higher priced plan with larger and
> more
> > stable build capacity?
> >
> > BTW, another factor that contribute to the productivity problem is that
> > our build is slow - we run full build for every PR and a successful full
> > build takes ~5h. We definitely have more options to solve it, for
> instance,
> > modularize the build graphs and reuse artifacts from the previous build.
> > But I think that can be a big effort which is much harder to accomplish
> in
> > a short period of time and may deserve its own separate discussion.
> >
> > [1] https://travis-ci.org/apache/flink/pull_requests
> >
> >
>

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Bowen Li]

Hi Steven,

I think you may not read what I wrote. The discussion is about "unstable
build **capacity**", in another word "unstable / lack of build resources",
not "unstable build".

On Mon, Jun 24, 2019 at 4:40 PM Steven Wu  wrote:

> long and sometimes unstable build is definitely a pain point.
>
> I suspect the build failure here in flink-connector-kafka is not related to
> my change. but there is no easy re-run the build on travis UI. Google
> search showed a trick of close-and-open the PR will trigger rebuild. but
> that could add noises to the PR activities.
> https://travis-ci.org/apache/flink/jobs/54519
>
> travis-ci for my personal repo often failed with exceeding time limit after
> 4+ hours.
> The job exceeded the maximum time limit for jobs, and has been terminated.
>
> On Mon, Jun 24, 2019 at 4:15 PM Bowen Li  wrote:
>
> > https://travis-ci.org/apache/flink/builds/549681530  This build request
> > has
> > been sitting at **HEAD of the queue** since I first saw it at PST 10:30am
> > (not sure how long it's been there before 10:30am). It's PST 4:12pm now
> and
> > it hasn't started yet.
> >
> > On Mon, Jun 24, 2019 at 2:48 PM Bowen Li  wrote:
> >
> > > Hi devs,
> > >
> > > I've been experiencing the pain resulting from lack of stable build
> > > capacity on Travis for Flink PRs [1]. Specifically, I noticed often
> that
> > no
> > > build in the queue is making any progress for hours, and suddenly 5 or
> 6
> > > builds kick off all together after the long pause. I'm at PST (UTC-08)
> > time
> > > zone, and I've seen pause can be as long as 6 hours from PST 9am to 3pm
> > > (let alone the time needed to drain the queue afterwards).
> > >
> > > I think this has greatly impacted our productivity. I've experienced
> that
> > > PRs submitted in the early morning of PST time zone won't finish their
> > > build until late night of the same day.
> > >
> > > So my questions are:
> > >
> > > - Has anyone else experienced the same problem or have similar
> > observation
> > > on TravisCI? (I suspect it has things to do with time zone)
> > >
> > > - What pricing plan of TravisCI is Flink currently using? Is it the
> free
> > > plan for open source projects? What are the guaranteed build capacity
> of
> > > the current plan?
> > >
> > > - If the current pricing plan (either free or paid) can't provide
> stable
> > > build capacity, can we upgrade to a higher priced plan with larger and
> > more
> > > stable build capacity?
> > >
> > > BTW, another factor that contribute to the productivity problem is that
> > > our build is slow - we run full build for every PR and a successful
> full
> > > build takes ~5h. We definitely have more options to solve it, for
> > instance,
> > > modularize the build graphs and reuse artifacts from the previous
> build.
> > > But I think that can be a big effort which is much harder to accomplish
> > in
> > > a short period of time and may deserve its own separate discussion.
> > >
> > > [1] https://travis-ci.org/apache/flink/pull_requests
> > >
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.8.1

[jincheng sun]

Due to no new blocker jumps up, I am preparing the RC1 now ...

jincheng sun  于2019年6月21日周五 下午4:55写道：

> Hi All,
>
> The last blocker(FLINK-12863) of 1.8.1 release have been fixed!
> But also welcome report any issues you think is a blocker!
> I will also do the final check, if no new problems are found, I will
> prepare RC1 as soon as possible! :)
>
> Cheers,
> Jincheng
>
> jincheng sun  于2019年6月17日周一 上午9:24写道：
>
>> Hi Till,
>>
>> Thank you for this timely discovery! @Till Rohrmann
>> 
>> We would start the release process after FLINK-12863
>>  resolved!
>>
>> Best,
>> Jincheng
>>
>> Till Rohrmann  于2019年6月16日周日 下午10:03写道：
>>
>>> Hi Jincheng,
>>>
>>> I just realized that we introduced a race condition with FLINK-11059. We
>>> first need to fix this problem [1] before we can start the release
>>> process.
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-12863
>>>
>>> Cheers,
>>> Till
>>>
>>> On Sat, Jun 15, 2019 at 3:48 PM jincheng sun 
>>> wrote:
>>>
>>> > I am very happy to say that all the blockers and critical issues for
>>> > released on 1.8.1 have been solved!!
>>> >
>>> > Great thanks to Aitozi, Yu Li, Congxian Qiu, Yun Tang, shuai.xu,
>>> JiangJie
>>> > Qin and zhijiang for the quick fix!
>>> > Great thanks to tzulitai, Aljoscha, Till  and pnowojski  for the high
>>> > priority review!
>>> >
>>> > Great thanks to all of you for the help(fix or review) in promoting the
>>> > 1.8.1 release. Thank you!!!
>>> >
>>> > I will prepare the first RC of release 1.8.1 as soon as possible :)
>>> >
>>> > Cheers,
>>> > Jincheng
>>> >
>>> > jincheng sun  于2019年6月14日周五 上午9:23写道：
>>> >
>>> > > Hi all,
>>> > >
>>> > > After the recent efforts of all of you, all the Issues of Blocker and
>>> > > Critical will be completed soon! :)
>>> > > There are only 2 issues left and they are also almost done:
>>> > >
>>> > > [Blocker]
>>> > > - FLINK-12297  Work by  @Aitozi  Being
>>> > > reviewed by @Aljoscha Krettek ! [almost done].
>>> > > [Critical]
>>> > > - FLINK-11059 Work by @shuai-xu  Being
>>> > > reviewed by @Till Rohrmann ! [almost done]
>>> > >
>>> > > The detail can be found here:
>>> > >
>>> > >
>>> >
>>> https://docs.google.com/document/d/1858C7HdyDPIxxm2Rvnu4bYahq0Tr9NaY1mj7BzQi_0w/edit?usp=sharing
>>> > >
>>> > > Great thanks to all of you for the help(fix or review) in promoting
>>> the
>>> > > 1.8.1 release. Thank you!!!
>>> > >
>>> > > I think today would finish all the Blocker and Critical issues. And
>>> I'll
>>> > > do the last check before preparing the RC1 of release 1.8.1.  then
>>> the
>>> > > first   RC of 1.8.1 will be coming :)
>>> > >
>>> > > Best,
>>> > > Jincheng
>>> > >
>>> > > jincheng sun  于2019年6月10日周一 上午5:24写道：
>>> > >
>>> > >> Hi all,
>>> > >> I am here to quickly update the progress of the issue that needs to
>>> be
>>> > >> tracked(going well):
>>> > >>
>>> > >> [Blocker]
>>> > >> - FLINK-12297 
>>> Work
>>> > >> by @Aitozi  Being reviewed by @Aljoscha
>>> > >> Krettek !
>>> > >> - FLINK-11107 
>>> Work
>>> > >> by @Myasuka  Being reviewed by @Tzu-Li
>>> > >> (Gordon) Tai 
>>> > >>
>>> > >> [Critical]
>>> > >> - FLINK-11059 
>>> Work
>>> > >> by @shuai-xu  Being reviewed by @Till
>>> > >> Rohrmann !
>>> > >>
>>> > >> [Nice to have]
>>> > >> - FLINK-10455 
>>> Work
>>> > >> by @becketqin Need someone to volunteer review the PR
>>> > >>
>>> > >> The detail can be found here:
>>> > >>
>>> > >>
>>> >
>>> https://docs.google.com/document/d/1858C7HdyDPIxxm2Rvnu4bYahq0Tr9NaY1mj7BzQi_0w/edit?usp=sharing
>>> > >>
>>> > >> Great thanks to all of you for the help(fix or review) in promoting
>>> the
>>> > >> 1.8.1 release. Thank you!!!
>>> > >>
>>> > >> I hope to prepare the first RC of release 1.8.1 on Thursday, and
>>> > >> FLINK-12297 ,
>>> > >> FLINK-11107 ,
>>> > >> FLINK-11059 
>>> should
>>> > >> be merged before the RC1.
>>> > >> If the relevant PR can't be Merged, please let me know, and we will
>>> put
>>> > >> more energy into solving! :)
>>> > >>
>>> > >> Best,
>>> > >> Jincheng
>>> > >>
>>> > >> jincheng sun  于2019年6月5日周三 下午5:33写道：
>>> > >>
>>> > >>> I am here to quickly update the progress of the issue that needs
>>> to be
>>> > >>> tracked(going well):
>>> > >>>
>>> > >>> [Blocker]
>>> > >>> - FLINK-12296 
>>> > [done]
>>> > >>> - FLINK-11987 
>>> > [done]
>>> > >>> - FLINK-12297 
>>> > >>>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Jark Wu]

Congratulations Jincheng! Well deserved!

On Tue, 25 Jun 2019 at 01:38, Bowen Li  wrote:

> Congratulations!
>
> On Mon, Jun 24, 2019 at 9:53 AM Yun Tang  wrote:
>
> > Congratulations Jincheng!
> >
> > Best
> > Yun Tang
> > 
> > From: Yu Li 
> > Sent: Tuesday, June 25, 2019 0:07
> > To: dev
> > Subject: Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC
> >
> > Congratulations Jincheng! Well deserved!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 25 Jun 2019 at 00:01, Zili Chen  wrote:
> >
> > > Congratulations Jincheng!
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Hequn Cheng  于2019年6月24日周一 下午11:48写道：
> > >
> > > > Congratulations Jincheng!
> > > >
> > > > Best, Hequn
> > > >
> > > > On Mon, Jun 24, 2019 at 11:43 PM Dian Fu 
> > wrote:
> > > >
> > > > > Congratulations Jincheng!
> > > > >
> > > > > On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger <
> rmetz...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > On behalf of the Flink PMC, I'm happy to announce that Jincheng
> Sun
> > > is
> > > > > now
> > > > > > part of the Apache Flink Project Management Committee (PMC).
> > > > > >
> > > > > > Jincheng has been a committer since July 2017. He has been very
> > > active
> > > > on
> > > > > > Flink's Table API / SQL component, as well as helping with
> > releases.
> > > > > >
> > > > > > Congratulations & Welcome Jincheng!
> > > > > >
> > > > > > Best,
> > > > > > Robert
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Congxian Qiu]

Congratulations Jincheng! Well deserved!
Best,
Congxian


Jark Wu  于2019年6月25日周二 上午9:28写道：

> Congratulations Jincheng! Well deserved!
>
> On Tue, 25 Jun 2019 at 01:38, Bowen Li  wrote:
>
> > Congratulations!
> >
> > On Mon, Jun 24, 2019 at 9:53 AM Yun Tang  wrote:
> >
> > > Congratulations Jincheng!
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Yu Li 
> > > Sent: Tuesday, June 25, 2019 0:07
> > > To: dev
> > > Subject: Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC
> > >
> > > Congratulations Jincheng! Well deserved!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 25 Jun 2019 at 00:01, Zili Chen  wrote:
> > >
> > > > Congratulations Jincheng!
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > Hequn Cheng  于2019年6月24日周一 下午11:48写道：
> > > >
> > > > > Congratulations Jincheng!
> > > > >
> > > > > Best, Hequn
> > > > >
> > > > > On Mon, Jun 24, 2019 at 11:43 PM Dian Fu 
> > > wrote:
> > > > >
> > > > > > Congratulations Jincheng!
> > > > > >
> > > > > > On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger <
> > rmetz...@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > On behalf of the Flink PMC, I'm happy to announce that Jincheng
> > Sun
> > > > is
> > > > > > now
> > > > > > > part of the Apache Flink Project Management Committee (PMC).
> > > > > > >
> > > > > > > Jincheng has been a committer since July 2017. He has been very
> > > > active
> > > > > on
> > > > > > > Flink's Table API / SQL component, as well as helping with
> > > releases.
> > > > > > >
> > > > > > > Congratulations & Welcome Jincheng!
> > > > > > >
> > > > > > > Best,
> > > > > > > Robert
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[vino yang]

Congratulations Jincheng!

Jark Wu  于2019年6月25日周二 上午9:28写道：

> Congratulations Jincheng! Well deserved!
>
> On Tue, 25 Jun 2019 at 01:38, Bowen Li  wrote:
>
> > Congratulations!
> >
> > On Mon, Jun 24, 2019 at 9:53 AM Yun Tang  wrote:
> >
> > > Congratulations Jincheng!
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Yu Li 
> > > Sent: Tuesday, June 25, 2019 0:07
> > > To: dev
> > > Subject: Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC
> > >
> > > Congratulations Jincheng! Well deserved!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 25 Jun 2019 at 00:01, Zili Chen  wrote:
> > >
> > > > Congratulations Jincheng!
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > Hequn Cheng  于2019年6月24日周一 下午11:48写道：
> > > >
> > > > > Congratulations Jincheng!
> > > > >
> > > > > Best, Hequn
> > > > >
> > > > > On Mon, Jun 24, 2019 at 11:43 PM Dian Fu 
> > > wrote:
> > > > >
> > > > > > Congratulations Jincheng!
> > > > > >
> > > > > > On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger <
> > rmetz...@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > On behalf of the Flink PMC, I'm happy to announce that Jincheng
> > Sun
> > > > is
> > > > > > now
> > > > > > > part of the Apache Flink Project Management Committee (PMC).
> > > > > > >
> > > > > > > Jincheng has been a committer since July 2017. He has been very
> > > > active
> > > > > on
> > > > > > > Flink's Table API / SQL component, as well as helping with
> > > releases.
> > > > > > >
> > > > > > > Congratulations & Welcome Jincheng!
> > > > > > >
> > > > > > > Best,
> > > > > > > Robert
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re:[ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Haibo Sun]

Congratulations!


Best,
Haibo

At 2019-06-24 23:08:54, "Robert Metzger"  wrote:
>Hi all,
>
>On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
>part of the Apache Flink Project Management Committee (PMC).
>
>Jincheng has been a committer since July 2017. He has been very active on
>Flink's Table API / SQL component, as well as helping with releases.
>
>Congratulations & Welcome Jincheng!
>
>Best,
>Robert

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[LakeShen]

Congratulations! Jincheng Sun

Best,
LakeShen

Robert Metzger  于2019年6月24日周一 下午11:09写道：

> Hi all,
>
> On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
> part of the Apache Flink Project Management Committee (PMC).
>
> Jincheng has been a committer since July 2017. He has been very active on
> Flink's Table API / SQL component, as well as helping with releases.
>
> Congratulations & Welcome Jincheng!
>
> Best,
> Robert
>

[jira] [Created] (FLINK-12967) Change the input selection switching in StreamTwoInputSelectableProcessor#checkFinished to invoking the nextSelection method of the stream operator

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-12967:
-

 Summary: Change the input selection switching in 
StreamTwoInputSelectableProcessor#checkFinished to invoking the nextSelection 
method of the stream operator
 Key: FLINK-12967
 URL: https://issues.apache.org/jira/browse/FLINK-12967
 Project: Flink
  Issue Type: Sub-task
Reporter: Haibo Sun
Assignee: Haibo Sun


The runtime (`StreamTwoInputSelectableProcessor#checkFinished()`) switches the 
input selection when one input is finished, because `BoundedxInput.endInput()` 
was not supported before the PR#8731 
(https://github.com/apache/flink/pull/8731) is merged. Now we should change the 
logic of `StreamTwoInputSelectableProcessor#checkFinished()` to invoke 
`InputSelectable#nextSelection()`, and the input selection should been switched 
in `endInput()` by the operator.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Jark Wu]

Hi Bowen,

Thanks for bringing this. We also suffered from the long build time.
I agree that we should focus on solving build capacity problem in the
thread.

My observation is there is only one build is running, all the others (other
PRs, master) are pending.
The pricing plan[1] of travis shows it can support concurrent build jobs.
But I don't know which plan we are using, might be the free plan for open
source.

I cc-ed Chesnay who may have some experience on Travis.

Regards,
Jark

[1]: https://travis-ci.com/plans

On Tue, 25 Jun 2019 at 08:11, Bowen Li  wrote:

> Hi Steven,
>
> I think you may not read what I wrote. The discussion is about "unstable
> build **capacity**", in another word "unstable / lack of build resources",
> not "unstable build".
>
> On Mon, Jun 24, 2019 at 4:40 PM Steven Wu  wrote:
>
> > long and sometimes unstable build is definitely a pain point.
> >
> > I suspect the build failure here in flink-connector-kafka is not related
> to
> > my change. but there is no easy re-run the build on travis UI. Google
> > search showed a trick of close-and-open the PR will trigger rebuild. but
> > that could add noises to the PR activities.
> > https://travis-ci.org/apache/flink/jobs/54519
> >
> > travis-ci for my personal repo often failed with exceeding time limit
> after
> > 4+ hours.
> > The job exceeded the maximum time limit for jobs, and has been
> terminated.
> >
> > On Mon, Jun 24, 2019 at 4:15 PM Bowen Li  wrote:
> >
> > > https://travis-ci.org/apache/flink/builds/549681530  This build
> request
> > > has
> > > been sitting at **HEAD of the queue** since I first saw it at PST
> 10:30am
> > > (not sure how long it's been there before 10:30am). It's PST 4:12pm now
> > and
> > > it hasn't started yet.
> > >
> > > On Mon, Jun 24, 2019 at 2:48 PM Bowen Li  wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I've been experiencing the pain resulting from lack of stable build
> > > > capacity on Travis for Flink PRs [1]. Specifically, I noticed often
> > that
> > > no
> > > > build in the queue is making any progress for hours, and suddenly 5
> or
> > 6
> > > > builds kick off all together after the long pause. I'm at PST
> (UTC-08)
> > > time
> > > > zone, and I've seen pause can be as long as 6 hours from PST 9am to
> 3pm
> > > > (let alone the time needed to drain the queue afterwards).
> > > >
> > > > I think this has greatly impacted our productivity. I've experienced
> > that
> > > > PRs submitted in the early morning of PST time zone won't finish
> their
> > > > build until late night of the same day.
> > > >
> > > > So my questions are:
> > > >
> > > > - Has anyone else experienced the same problem or have similar
> > > observation
> > > > on TravisCI? (I suspect it has things to do with time zone)
> > > >
> > > > - What pricing plan of TravisCI is Flink currently using? Is it the
> > free
> > > > plan for open source projects? What are the guaranteed build capacity
> > of
> > > > the current plan?
> > > >
> > > > - If the current pricing plan (either free or paid) can't provide
> > stable
> > > > build capacity, can we upgrade to a higher priced plan with larger
> and
> > > more
> > > > stable build capacity?
> > > >
> > > > BTW, another factor that contribute to the productivity problem is
> that
> > > > our build is slow - we run full build for every PR and a successful
> > full
> > > > build takes ~5h. We definitely have more options to solve it, for
> > > instance,
> > > > modularize the build graphs and reuse artifacts from the previous
> > build.
> > > > But I think that can be a big effort which is much harder to
> accomplish
> > > in
> > > > a short period of time and may deserve its own separate discussion.
> > > >
> > > > [1] https://travis-ci.org/apache/flink/pull_requests
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Becket Qin]

Congrats Jincheng! Well deserved!

On Tue, Jun 25, 2019 at 9:56 AM LakeShen  wrote:

> Congratulations! Jincheng Sun
>
> Best,
> LakeShen
>
> Robert Metzger  于2019年6月24日周一 下午11:09写道：
>
> > Hi all,
> >
> > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> now
> > part of the Apache Flink Project Management Committee (PMC).
> >
> > Jincheng has been a committer since July 2017. He has been very active on
> > Flink's Table API / SQL component, as well as helping with releases.
> >
> > Congratulations & Welcome Jincheng!
> >
> > Best,
> > Robert
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Kurt Young]

Congratulations Jincheng!

Best,
Kurt


On Tue, Jun 25, 2019 at 9:56 AM LakeShen  wrote:

> Congratulations! Jincheng Sun
>
> Best,
> LakeShen
>
> Robert Metzger  于2019年6月24日周一 下午11:09写道：
>
> > Hi all,
> >
> > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> now
> > part of the Apache Flink Project Management Committee (PMC).
> >
> > Jincheng has been a committer since July 2017. He has been very active on
> > Flink's Table API / SQL component, as well as helping with releases.
> >
> > Congratulations & Welcome Jincheng!
> >
> > Best,
> > Robert
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Tao Yangyu]

Congratulations! Jincheng well deserved!

BR,
Ryan

Kurt Young  于2019年6月25日周二 上午10:10写道：

> Congratulations Jincheng!
>
> Best,
> Kurt
>
>
> On Tue, Jun 25, 2019 at 9:56 AM LakeShen 
> wrote:
>
> > Congratulations! Jincheng Sun
> >
> > Best,
> > LakeShen
> >
> > Robert Metzger  于2019年6月24日周一 下午11:09写道：
> >
> > > Hi all,
> > >
> > > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> > now
> > > part of the Apache Flink Project Management Committee (PMC).
> > >
> > > Jincheng has been a committer since July 2017. He has been very active
> on
> > > Flink's Table API / SQL component, as well as helping with releases.
> > >
> > > Congratulations & Welcome Jincheng!
> > >
> > > Best,
> > > Robert
> > >
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Bo WANG]

Congratulations Jincheng! Well deserved!

On Tue, Jun 25, 2019 at 10:10 AM Kurt Young  wrote:

> Congratulations Jincheng!
>
> Best,
> Kurt
>
>
> On Tue, Jun 25, 2019 at 9:56 AM LakeShen 
> wrote:
>
> > Congratulations! Jincheng Sun
> >
> > Best,
> > LakeShen
> >
> > Robert Metzger  于2019年6月24日周一 下午11:09写道：
> >
> > > Hi all,
> > >
> > > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> > now
> > > part of the Apache Flink Project Management Committee (PMC).
> > >
> > > Jincheng has been a committer since July 2017. He has been very active
> on
> > > Flink's Table API / SQL component, as well as helping with releases.
> > >
> > > Congratulations & Welcome Jincheng!
> > >
> > > Best,
> > > Robert
> > >
> >
>

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Kurt Young]

Hi Bowen,

Thanks for bringing this up. We actually have discussed about this, and I
think Till and George have
already spend sometime investigating it. I have cced both of them, and
maybe they can share
their findings.

Best,
Kurt


On Tue, Jun 25, 2019 at 10:08 AM Jark Wu  wrote:

> Hi Bowen,
>
> Thanks for bringing this. We also suffered from the long build time.
> I agree that we should focus on solving build capacity problem in the
> thread.
>
> My observation is there is only one build is running, all the others (other
> PRs, master) are pending.
> The pricing plan[1] of travis shows it can support concurrent build jobs.
> But I don't know which plan we are using, might be the free plan for open
> source.
>
> I cc-ed Chesnay who may have some experience on Travis.
>
> Regards,
> Jark
>
> [1]: https://travis-ci.com/plans
>
> On Tue, 25 Jun 2019 at 08:11, Bowen Li  wrote:
>
> > Hi Steven,
> >
> > I think you may not read what I wrote. The discussion is about "unstable
> > build **capacity**", in another word "unstable / lack of build
> resources",
> > not "unstable build".
> >
> > On Mon, Jun 24, 2019 at 4:40 PM Steven Wu  wrote:
> >
> > > long and sometimes unstable build is definitely a pain point.
> > >
> > > I suspect the build failure here in flink-connector-kafka is not
> related
> > to
> > > my change. but there is no easy re-run the build on travis UI. Google
> > > search showed a trick of close-and-open the PR will trigger rebuild.
> but
> > > that could add noises to the PR activities.
> > > https://travis-ci.org/apache/flink/jobs/54519
> > >
> > > travis-ci for my personal repo often failed with exceeding time limit
> > after
> > > 4+ hours.
> > > The job exceeded the maximum time limit for jobs, and has been
> > terminated.
> > >
> > > On Mon, Jun 24, 2019 at 4:15 PM Bowen Li  wrote:
> > >
> > > > https://travis-ci.org/apache/flink/builds/549681530  This build
> > request
> > > > has
> > > > been sitting at **HEAD of the queue** since I first saw it at PST
> > 10:30am
> > > > (not sure how long it's been there before 10:30am). It's PST 4:12pm
> now
> > > and
> > > > it hasn't started yet.
> > > >
> > > > On Mon, Jun 24, 2019 at 2:48 PM Bowen Li 
> wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I've been experiencing the pain resulting from lack of stable build
> > > > > capacity on Travis for Flink PRs [1]. Specifically, I noticed often
> > > that
> > > > no
> > > > > build in the queue is making any progress for hours, and suddenly 5
> > or
> > > 6
> > > > > builds kick off all together after the long pause. I'm at PST
> > (UTC-08)
> > > > time
> > > > > zone, and I've seen pause can be as long as 6 hours from PST 9am to
> > 3pm
> > > > > (let alone the time needed to drain the queue afterwards).
> > > > >
> > > > > I think this has greatly impacted our productivity. I've
> experienced
> > > that
> > > > > PRs submitted in the early morning of PST time zone won't finish
> > their
> > > > > build until late night of the same day.
> > > > >
> > > > > So my questions are:
> > > > >
> > > > > - Has anyone else experienced the same problem or have similar
> > > > observation
> > > > > on TravisCI? (I suspect it has things to do with time zone)
> > > > >
> > > > > - What pricing plan of TravisCI is Flink currently using? Is it the
> > > free
> > > > > plan for open source projects? What are the guaranteed build
> capacity
> > > of
> > > > > the current plan?
> > > > >
> > > > > - If the current pricing plan (either free or paid) can't provide
> > > stable
> > > > > build capacity, can we upgrade to a higher priced plan with larger
> > and
> > > > more
> > > > > stable build capacity?
> > > > >
> > > > > BTW, another factor that contribute to the productivity problem is
> > that
> > > > > our build is slow - we run full build for every PR and a successful
> > > full
> > > > > build takes ~5h. We definitely have more options to solve it, for
> > > > instance,
> > > > > modularize the build graphs and reuse artifacts from the previous
> > > build.
> > > > > But I think that can be a big effort which is much harder to
> > accomplish
> > > > in
> > > > > a short period of time and may deserve its own separate discussion.
> > > > >
> > > > > [1] https://travis-ci.org/apache/flink/pull_requests
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Kurt Young]

(Forgot to cc George)

Best,
Kurt


On Tue, Jun 25, 2019 at 10:16 AM Kurt Young  wrote:

> Hi Bowen,
>
> Thanks for bringing this up. We actually have discussed about this, and I
> think Till and George have
> already spend sometime investigating it. I have cced both of them, and
> maybe they can share
> their findings.
>
> Best,
> Kurt
>
>
> On Tue, Jun 25, 2019 at 10:08 AM Jark Wu  wrote:
>
>> Hi Bowen,
>>
>> Thanks for bringing this. We also suffered from the long build time.
>> I agree that we should focus on solving build capacity problem in the
>> thread.
>>
>> My observation is there is only one build is running, all the others
>> (other
>> PRs, master) are pending.
>> The pricing plan[1] of travis shows it can support concurrent build jobs.
>> But I don't know which plan we are using, might be the free plan for open
>> source.
>>
>> I cc-ed Chesnay who may have some experience on Travis.
>>
>> Regards,
>> Jark
>>
>> [1]: https://travis-ci.com/plans
>>
>> On Tue, 25 Jun 2019 at 08:11, Bowen Li  wrote:
>>
>> > Hi Steven,
>> >
>> > I think you may not read what I wrote. The discussion is about "unstable
>> > build **capacity**", in another word "unstable / lack of build
>> resources",
>> > not "unstable build".
>> >
>> > On Mon, Jun 24, 2019 at 4:40 PM Steven Wu  wrote:
>> >
>> > > long and sometimes unstable build is definitely a pain point.
>> > >
>> > > I suspect the build failure here in flink-connector-kafka is not
>> related
>> > to
>> > > my change. but there is no easy re-run the build on travis UI. Google
>> > > search showed a trick of close-and-open the PR will trigger rebuild.
>> but
>> > > that could add noises to the PR activities.
>> > > https://travis-ci.org/apache/flink/jobs/54519
>> > >
>> > > travis-ci for my personal repo often failed with exceeding time limit
>> > after
>> > > 4+ hours.
>> > > The job exceeded the maximum time limit for jobs, and has been
>> > terminated.
>> > >
>> > > On Mon, Jun 24, 2019 at 4:15 PM Bowen Li  wrote:
>> > >
>> > > > https://travis-ci.org/apache/flink/builds/549681530  This build
>> > request
>> > > > has
>> > > > been sitting at **HEAD of the queue** since I first saw it at PST
>> > 10:30am
>> > > > (not sure how long it's been there before 10:30am). It's PST 4:12pm
>> now
>> > > and
>> > > > it hasn't started yet.
>> > > >
>> > > > On Mon, Jun 24, 2019 at 2:48 PM Bowen Li 
>> wrote:
>> > > >
>> > > > > Hi devs,
>> > > > >
>> > > > > I've been experiencing the pain resulting from lack of stable
>> build
>> > > > > capacity on Travis for Flink PRs [1]. Specifically, I noticed
>> often
>> > > that
>> > > > no
>> > > > > build in the queue is making any progress for hours, and suddenly
>> 5
>> > or
>> > > 6
>> > > > > builds kick off all together after the long pause. I'm at PST
>> > (UTC-08)
>> > > > time
>> > > > > zone, and I've seen pause can be as long as 6 hours from PST 9am
>> to
>> > 3pm
>> > > > > (let alone the time needed to drain the queue afterwards).
>> > > > >
>> > > > > I think this has greatly impacted our productivity. I've
>> experienced
>> > > that
>> > > > > PRs submitted in the early morning of PST time zone won't finish
>> > their
>> > > > > build until late night of the same day.
>> > > > >
>> > > > > So my questions are:
>> > > > >
>> > > > > - Has anyone else experienced the same problem or have similar
>> > > > observation
>> > > > > on TravisCI? (I suspect it has things to do with time zone)
>> > > > >
>> > > > > - What pricing plan of TravisCI is Flink currently using? Is it
>> the
>> > > free
>> > > > > plan for open source projects? What are the guaranteed build
>> capacity
>> > > of
>> > > > > the current plan?
>> > > > >
>> > > > > - If the current pricing plan (either free or paid) can't provide
>> > > stable
>> > > > > build capacity, can we upgrade to a higher priced plan with larger
>> > and
>> > > > more
>> > > > > stable build capacity?
>> > > > >
>> > > > > BTW, another factor that contribute to the productivity problem is
>> > that
>> > > > > our build is slow - we run full build for every PR and a
>> successful
>> > > full
>> > > > > build takes ~5h. We definitely have more options to solve it, for
>> > > > instance,
>> > > > > modularize the build graphs and reuse artifacts from the previous
>> > > build.
>> > > > > But I think that can be a big effort which is much harder to
>> > accomplish
>> > > > in
>> > > > > a short period of time and may deserve its own separate
>> discussion.
>> > > > >
>> > > > > [1] https://travis-ci.org/apache/flink/pull_requests
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Yun Gao]

Congratulations! 

Best,
Yun


--
From:Robert Metzger 
Send Time:2019 Jun. 24 (Mon.) 23:16
To:dev 
Subject:[ANNOUNCE] Jincheng Sun is now part of the Flink PMC

Hi all,

On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
part of the Apache Flink Project Management Committee (PMC).

Jincheng has been a committer since July 2017. He has been very active on
Flink's Table API / SQL component, as well as helping with releases.

Congratulations & Welcome Jincheng!

Best,
Robert

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[zhijiang]

Congratulations Jincheng!

Best,
Zhijiang
--
From:Kurt Young 
Send Time:2019年6月25日(星期二) 10:10
To:dev 
Subject:Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

Congratulations Jincheng!

Best,
Kurt


On Tue, Jun 25, 2019 at 9:56 AM LakeShen  wrote:

> Congratulations! Jincheng Sun
>
> Best,
> LakeShen
>
> Robert Metzger  于2019年6月24日周一 下午11:09写道：
>
> > Hi all,
> >
> > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> now
> > part of the Apache Flink Project Management Committee (PMC).
> >
> > Jincheng has been a committer since July 2017. He has been very active on
> > Flink's Table API / SQL component, as well as helping with releases.
> >
> > Congratulations & Welcome Jincheng!
> >
> > Best,
> > Robert
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[qianjin Xu]

Congratulations Jincheng!


Best

Forward




Robert Metzger  于2019年6月24日周一 下午11:09写道：

> Hi all,
>
> On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
> part of the Apache Flink Project Management Committee (PMC).
>
> Jincheng has been a committer since July 2017. He has been very active on
> Flink's Table API / SQL component, as well as helping with releases.
>
> Congratulations & Welcome Jincheng!
>
> Best,
> Robert
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Biao Liu]

Congratulations Jincheng!


qianjin Xu  于2019年6月25日周二 上午10:25写道：

> Congratulations Jincheng!
>
>
> Best
>
> Forward
>
>
>
>
> Robert Metzger  于2019年6月24日周一 下午11:09写道：
>
> > Hi all,
> >
> > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> now
> > part of the Apache Flink Project Management Committee (PMC).
> >
> > Jincheng has been a committer since July 2017. He has been very active on
> > Flink's Table API / SQL component, as well as helping with releases.
> >
> > Congratulations & Welcome Jincheng!
> >
> > Best,
> > Robert
> >
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Jeff Zhang]

Congratulations Jincheng!

Biao Liu  于2019年6月25日周二 上午10:29写道：

> Congratulations Jincheng!
>
>
> qianjin Xu  于2019年6月25日周二 上午10:25写道：
>
> > Congratulations Jincheng!
> >
> >
> > Best
> >
> > Forward
> >
> >
> >
> >
> > Robert Metzger  于2019年6月24日周一 下午11:09写道：
> >
> > > Hi all,
> > >
> > > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> > now
> > > part of the Apache Flink Project Management Committee (PMC).
> > >
> > > Jincheng has been a committer since July 2017. He has been very active
> on
> > > Flink's Table API / SQL component, as well as helping with releases.
> > >
> > > Congratulations & Welcome Jincheng!
> > >
> > > Best,
> > > Robert
> > >
> >
>


-- 
Best Regards

Jeff Zhang

the contributor permission

[gygz...@gmail.com]

hi,
   I've been using Flink for more than a year.
   I want to contribute to Apache Flink.
   Would you please give me the contributor permission?
   My JIRA ID is sadfddd


gygz...@gmail.com

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[vino yang]

Hi Simon,

Good question!

For event time semantics, we reuse the window operator can keep the correct
behavior which is the same as the current window operator. The window
operator will trigger based on the watermark.

About your example, the window of three partitions will trigger normally.
For the delayed partition, it should not trigger if there is no correct
watermark. The behavior is the same as input.keyBy(0).window in event time
semantics.

For processing idle partition scenarios, currently, Flink allows
calling markAsTemporarilyIdle to send StreamStatus.IDLE to the downstream.

Best,
Vino

Shu Su  于2019年6月24日周一 下午9:13写道：

> Hi vino
>
>
> Thanks for proposal.
> For Local Aggregation I have a question about doing this in window
> aggregation. As we know , window aggregation like sliding window should
> based on
> Time trigger, and there may exists a problem in event time if we do local
> aggregation. For example if I want to do a 5s sliding window with count agg:
>
>
> 1. I have input with 4 parallelism and data are firstly randomly pass in 4
> partitions.
> 2. We do LocalAggregation in each of them and we get a partial count
> result.
> 3. Forward partial result to a node with same key then do the final
> aggregation.
>
>
> It seems no problem but what will happen if data skew in event time ? If
> we have a continuous time sequence in 3 of 4 input partitions, for example
> , we have a continuous time sequence in partition 1, 2, 3 but data to
> partition 4 was delay for some reason, and we just get 3 partial result for
> the moment, does final aggregation need to wait for the 4th partial result
> because of data delay ? If so , how long we need to wait for ? If not, does
> it mean that
> The final aggregation will wait forever ?
>
>
> Thanks,
> Simon
>
>
> On 06/18/2019 10:06，vino yang wrote：
> Hi Jark,
>
> We have done a comparative test. The effect is obvious.
>
> From our observation, the optimized effect mainly depends on two factors:
>
>
> - the degree of the skew: this factor depends on users business ;
> - the size of the window: localKeyBy support all the type of window
> which provided by Flink. Obviously, the larger the size of the window, the
> more obvious the effect.
>
> In production, we can not decide the first factor. About the second factor,
> it's the result of a trade-off. The size of the window affects the latency
> of the pre-aggregation. That's to say:
>
>
> - the larger the size of the window, the more obvious the effect;
> - the larger the size of the window, the larger latency of the result
>
> Best,
> Vino
>
> Jark Wu  于2019年6月17日周一 下午7:32写道：
>
> Hi Vino,
>
> Thanks for the proposal.
>
> Regarding to the "input.keyBy(0).sum(1)" vs
> "input.localKeyBy(0).countWindow(5).sum(1).keyBy(0).sum(1)", have you done
> some benchmark?
> Because I'm curious about how much performance improvement can we get by
> using count window as the local operator.
>
> Best,
> Jark
>
>
>
> On Mon, 17 Jun 2019 at 17:48, vino yang  wrote:
>
> Hi Hequn,
>
> Thanks for your reply.
>
> The purpose of localKeyBy API is to provide a tool which can let users do
> pre-aggregation in the local. The behavior of the pre-aggregation is
> similar to keyBy API.
>
> So the three cases are different, I will describe them one by one:
>
> 1. input.keyBy(0).sum(1)
>
> *In this case, the result is event-driven, each event can produce one sum
> aggregation result and it is the latest one from the source start.*
>
> 2. input.localKeyBy(0).sum(1).keyBy(0).sum(1)
>
> *In this case, the semantic may have a problem, it would do the local sum
> aggregation and will produce the latest partial result from the source
> start for every event. *
> *These latest partial results from the same key are hashed to one node to
> do the global sum aggregation.*
> *In the global aggregation, when it received multiple partial results
> (they
> are all calculated from the source start) and sum them will get the wrong
> result.*
>
> 3. input.localKeyBy(0).countWindow(5).sum(1).keyBy(0).sum(1)
>
> *In this case, it would just get a partial aggregation result for the 5
> records in the count window. The partial aggregation results from the
> same
> key will be aggregated globally.*
>
> So the first case and the third case can get the *same* result, the
> difference is the output-style and the latency.
>
> Generally speaking, the local key API is just an optimization API. We do
> not limit the user's usage, but the user has to understand its semantics
> and use it correctly.
>
> Best,
> Vino
>
> Hequn Cheng  于2019年6月17日周一 下午4:18写道：
>
> Hi Vino,
>
> Thanks for the proposal, I think it is a very good feature!
>
> One thing I want to make sure is the semantics for the `localKeyBy`.
> From
> the document, the `localKeyBy` API returns an instance of `KeyedStream`
> which can also perform sum(), so in this case, what's the semantics for
> `localKeyBy()`. For example, will the following code share the same
> result?
> and what're the differences betwee

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Jeff Zhang]

Hi Folks,

Zeppelin meet this kind of issue before, we solve it by delegating each
one's PR build to his travis account (Everyone can have 5 free slot for
travis build).
Apache account travis build is only triggered when PR is merged.



Kurt Young  于2019年6月25日周二 上午10:16写道：

> (Forgot to cc George)
>
> Best,
> Kurt
>
>
> On Tue, Jun 25, 2019 at 10:16 AM Kurt Young  wrote:
>
> > Hi Bowen,
> >
> > Thanks for bringing this up. We actually have discussed about this, and I
> > think Till and George have
> > already spend sometime investigating it. I have cced both of them, and
> > maybe they can share
> > their findings.
> >
> > Best,
> > Kurt
> >
> >
> > On Tue, Jun 25, 2019 at 10:08 AM Jark Wu  wrote:
> >
> >> Hi Bowen,
> >>
> >> Thanks for bringing this. We also suffered from the long build time.
> >> I agree that we should focus on solving build capacity problem in the
> >> thread.
> >>
> >> My observation is there is only one build is running, all the others
> >> (other
> >> PRs, master) are pending.
> >> The pricing plan[1] of travis shows it can support concurrent build
> jobs.
> >> But I don't know which plan we are using, might be the free plan for
> open
> >> source.
> >>
> >> I cc-ed Chesnay who may have some experience on Travis.
> >>
> >> Regards,
> >> Jark
> >>
> >> [1]: https://travis-ci.com/plans
> >>
> >> On Tue, 25 Jun 2019 at 08:11, Bowen Li  wrote:
> >>
> >> > Hi Steven,
> >> >
> >> > I think you may not read what I wrote. The discussion is about
> "unstable
> >> > build **capacity**", in another word "unstable / lack of build
> >> resources",
> >> > not "unstable build".
> >> >
> >> > On Mon, Jun 24, 2019 at 4:40 PM Steven Wu 
> wrote:
> >> >
> >> > > long and sometimes unstable build is definitely a pain point.
> >> > >
> >> > > I suspect the build failure here in flink-connector-kafka is not
> >> related
> >> > to
> >> > > my change. but there is no easy re-run the build on travis UI.
> Google
> >> > > search showed a trick of close-and-open the PR will trigger rebuild.
> >> but
> >> > > that could add noises to the PR activities.
> >> > > https://travis-ci.org/apache/flink/jobs/54519
> >> > >
> >> > > travis-ci for my personal repo often failed with exceeding time
> limit
> >> > after
> >> > > 4+ hours.
> >> > > The job exceeded the maximum time limit for jobs, and has been
> >> > terminated.
> >> > >
> >> > > On Mon, Jun 24, 2019 at 4:15 PM Bowen Li 
> wrote:
> >> > >
> >> > > > https://travis-ci.org/apache/flink/builds/549681530  This build
> >> > request
> >> > > > has
> >> > > > been sitting at **HEAD of the queue** since I first saw it at PST
> >> > 10:30am
> >> > > > (not sure how long it's been there before 10:30am). It's PST
> 4:12pm
> >> now
> >> > > and
> >> > > > it hasn't started yet.
> >> > > >
> >> > > > On Mon, Jun 24, 2019 at 2:48 PM Bowen Li 
> >> wrote:
> >> > > >
> >> > > > > Hi devs,
> >> > > > >
> >> > > > > I've been experiencing the pain resulting from lack of stable
> >> build
> >> > > > > capacity on Travis for Flink PRs [1]. Specifically, I noticed
> >> often
> >> > > that
> >> > > > no
> >> > > > > build in the queue is making any progress for hours, and
> suddenly
> >> 5
> >> > or
> >> > > 6
> >> > > > > builds kick off all together after the long pause. I'm at PST
> >> > (UTC-08)
> >> > > > time
> >> > > > > zone, and I've seen pause can be as long as 6 hours from PST 9am
> >> to
> >> > 3pm
> >> > > > > (let alone the time needed to drain the queue afterwards).
> >> > > > >
> >> > > > > I think this has greatly impacted our productivity. I've
> >> experienced
> >> > > that
> >> > > > > PRs submitted in the early morning of PST time zone won't finish
> >> > their
> >> > > > > build until late night of the same day.
> >> > > > >
> >> > > > > So my questions are:
> >> > > > >
> >> > > > > - Has anyone else experienced the same problem or have similar
> >> > > > observation
> >> > > > > on TravisCI? (I suspect it has things to do with time zone)
> >> > > > >
> >> > > > > - What pricing plan of TravisCI is Flink currently using? Is it
> >> the
> >> > > free
> >> > > > > plan for open source projects? What are the guaranteed build
> >> capacity
> >> > > of
> >> > > > > the current plan?
> >> > > > >
> >> > > > > - If the current pricing plan (either free or paid) can't
> provide
> >> > > stable
> >> > > > > build capacity, can we upgrade to a higher priced plan with
> larger
> >> > and
> >> > > > more
> >> > > > > stable build capacity?
> >> > > > >
> >> > > > > BTW, another factor that contribute to the productivity problem
> is
> >> > that
> >> > > > > our build is slow - we run full build for every PR and a
> >> successful
> >> > > full
> >> > > > > build takes ~5h. We definitely have more options to solve it,
> for
> >> > > > instance,
> >> > > > > modularize the build graphs and reuse artifacts from the
> previous
> >> > > build.
> >> > > > > But I think that can be a big effort which is much harder to
> >> > accomplish
> >> > > > in
> >> > > > > a short p

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Terry Wang]

Congratulations Jincheng!

> 在 2019年6月24日，下午11:08，Robert Metzger  写道：
> 
> Hi all,
> 
> On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
> part of the Apache Flink Project Management Committee (PMC).
> 
> Jincheng has been a committer since July 2017. He has been very active on
> Flink's Table API / SQL component, as well as helping with releases.
> 
> Congratulations & Welcome Jincheng!
> 
> Best,
> Robert

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[Shu Su]

Hi Vino


Thanks for your reply.


It seems feasible if a StreamStatus.IDLE was send to downstream, Still two 
questions.
 1. Do we need to add a method to allow users control when to send 
StreamStatus.IDLE to downsteram in this case?
 2. If a partial data comes after your IDLE status to downstream, does this 
means final side will get rid of the partial data ? Or we have a mechanism to 
handle this ? 


Thanks,
Simon
On 06/25/2019 10:56，vino yang wrote：
Hi Simon,

Good question!

For event time semantics, we reuse the window operator can keep the correct
behavior which is the same as the current window operator. The window
operator will trigger based on the watermark.

About your example, the window of three partitions will trigger normally.
For the delayed partition, it should not trigger if there is no correct
watermark. The behavior is the same as input.keyBy(0).window in event time
semantics.

For processing idle partition scenarios, currently, Flink allows
calling markAsTemporarilyIdle to send StreamStatus.IDLE to the downstream.

Best,
Vino

Shu Su  于2019年6月24日周一 下午9:13写道：

Hi vino


Thanks for proposal.
For Local Aggregation I have a question about doing this in window
aggregation. As we know , window aggregation like sliding window should
based on
Time trigger, and there may exists a problem in event time if we do local
aggregation. For example if I want to do a 5s sliding window with count agg:


1. I have input with 4 parallelism and data are firstly randomly pass in 4
partitions.
2. We do LocalAggregation in each of them and we get a partial count
result.
3. Forward partial result to a node with same key then do the final
aggregation.


It seems no problem but what will happen if data skew in event time ? If
we have a continuous time sequence in 3 of 4 input partitions, for example
, we have a continuous time sequence in partition 1, 2, 3 but data to
partition 4 was delay for some reason, and we just get 3 partial result for
the moment, does final aggregation need to wait for the 4th partial result
because of data delay ? If so , how long we need to wait for ? If not, does
it mean that
The final aggregation will wait forever ?


Thanks,
Simon


On 06/18/2019 10:06，vino yang wrote：
Hi Jark,

We have done a comparative test. The effect is obvious.

From our observation, the optimized effect mainly depends on two factors:


- the degree of the skew: this factor depends on users business ;
- the size of the window: localKeyBy support all the type of window
which provided by Flink. Obviously, the larger the size of the window, the
more obvious the effect.

In production, we can not decide the first factor. About the second factor,
it's the result of a trade-off. The size of the window affects the latency
of the pre-aggregation. That's to say:


- the larger the size of the window, the more obvious the effect;
- the larger the size of the window, the larger latency of the result

Best,
Vino

Jark Wu  于2019年6月17日周一 下午7:32写道：

Hi Vino,

Thanks for the proposal.

Regarding to the "input.keyBy(0).sum(1)" vs
"input.localKeyBy(0).countWindow(5).sum(1).keyBy(0).sum(1)", have you done
some benchmark?
Because I'm curious about how much performance improvement can we get by
using count window as the local operator.

Best,
Jark



On Mon, 17 Jun 2019 at 17:48, vino yang  wrote:

Hi Hequn,

Thanks for your reply.

The purpose of localKeyBy API is to provide a tool which can let users do
pre-aggregation in the local. The behavior of the pre-aggregation is
similar to keyBy API.

So the three cases are different, I will describe them one by one:

1. input.keyBy(0).sum(1)

*In this case, the result is event-driven, each event can produce one sum
aggregation result and it is the latest one from the source start.*

2. input.localKeyBy(0).sum(1).keyBy(0).sum(1)

*In this case, the semantic may have a problem, it would do the local sum
aggregation and will produce the latest partial result from the source
start for every event. *
*These latest partial results from the same key are hashed to one node to
do the global sum aggregation.*
*In the global aggregation, when it received multiple partial results
(they
are all calculated from the source start) and sum them will get the wrong
result.*

3. input.localKeyBy(0).countWindow(5).sum(1).keyBy(0).sum(1)

*In this case, it would just get a partial aggregation result for the 5
records in the count window. The partial aggregation results from the
same
key will be aggregated globally.*

So the first case and the third case can get the *same* result, the
difference is the output-style and the latency.

Generally speaking, the local key API is just an optimization API. We do
not limit the user's usage, but the user has to understand its semantics
and use it correctly.

Best,
Vino

Hequn Cheng  于2019年6月17日周一 下午4:18写道：

Hi Vino,

Thanks for the proposal, I think it is a very good feature!

One thing I want to make sure is the semantics for the `localKeyBy`.
From
the document, t

[jira] [Created] (FLINK-12968) Add a casting utility

[Timo Walther (JIRA)]

Timo Walther created FLINK-12968:


 Summary: Add a casting utility
 Key: FLINK-12968
 URL: https://issues.apache.org/jira/browse/FLINK-12968
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Timo Walther
Assignee: Timo Walther


In order to make the new type system fully usable, we need to define logical 
casting rules that include both unsafe and safe casting, type widening etc. 
similar to the existing class: {{org.apache.flink.table.typeutils.TypeCoercion}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[vino yang]

Hi Simon,

IMO, we do not need special processing for your example scenarios, Flink
suggests users extracting watermarks in source function.

Generally, the IDLE is temporary status, when the data coming, it will send
ACTIVE status to the downstream and the processing will continue.

Keep in mind, all the cases are the same as window operator, there is no
difference. So, I think we do not need to worry about these things.

Best,
Vino

Shu Su  于2019年6月25日周二 上午11:36写道：

>
>
> Hi Vino
>
>
> Thanks for your reply.
>
>
> It seems feasible if a StreamStatus.IDLE was send to downstream, Still two
> questions.
>  1. Do we need to add a method to allow users control when to send
> StreamStatus.IDLE to downsteram in this case?
>  2. If a partial data comes after your IDLE status to downstream, does
> this means final side will get rid of the partial data ? Or we have a
> mechanism to handle this ?
>
>
> Thanks,
> Simon
> On 06/25/2019 10:56，vino yang wrote：
> Hi Simon,
>
> Good question!
>
> For event time semantics, we reuse the window operator can keep the correct
> behavior which is the same as the current window operator. The window
> operator will trigger based on the watermark.
>
> About your example, the window of three partitions will trigger normally.
> For the delayed partition, it should not trigger if there is no correct
> watermark. The behavior is the same as input.keyBy(0).window in event time
> semantics.
>
> For processing idle partition scenarios, currently, Flink allows
> calling markAsTemporarilyIdle to send StreamStatus.IDLE to the downstream.
>
> Best,
> Vino
>
> Shu Su  于2019年6月24日周一 下午9:13写道：
>
> Hi vino
>
>
> Thanks for proposal.
> For Local Aggregation I have a question about doing this in window
> aggregation. As we know , window aggregation like sliding window should
> based on
> Time trigger, and there may exists a problem in event time if we do local
> aggregation. For example if I want to do a 5s sliding window with count
> agg:
>
>
> 1. I have input with 4 parallelism and data are firstly randomly pass in 4
> partitions.
> 2. We do LocalAggregation in each of them and we get a partial count
> result.
> 3. Forward partial result to a node with same key then do the final
> aggregation.
>
>
> It seems no problem but what will happen if data skew in event time ? If
> we have a continuous time sequence in 3 of 4 input partitions, for example
> , we have a continuous time sequence in partition 1, 2, 3 but data to
> partition 4 was delay for some reason, and we just get 3 partial result for
> the moment, does final aggregation need to wait for the 4th partial result
> because of data delay ? If so , how long we need to wait for ? If not, does
> it mean that
> The final aggregation will wait forever ?
>
>
> Thanks,
> Simon
>
>
> On 06/18/2019 10:06，vino yang wrote：
> Hi Jark,
>
> We have done a comparative test. The effect is obvious.
>
> From our observation, the optimized effect mainly depends on two factors:
>
>
> - the degree of the skew: this factor depends on users business ;
> - the size of the window: localKeyBy support all the type of window
> which provided by Flink. Obviously, the larger the size of the window, the
> more obvious the effect.
>
> In production, we can not decide the first factor. About the second factor,
> it's the result of a trade-off. The size of the window affects the latency
> of the pre-aggregation. That's to say:
>
>
> - the larger the size of the window, the more obvious the effect;
> - the larger the size of the window, the larger latency of the result
>
> Best,
> Vino
>
> Jark Wu  于2019年6月17日周一 下午7:32写道：
>
> Hi Vino,
>
> Thanks for the proposal.
>
> Regarding to the "input.keyBy(0).sum(1)" vs
> "input.localKeyBy(0).countWindow(5).sum(1).keyBy(0).sum(1)", have you done
> some benchmark?
> Because I'm curious about how much performance improvement can we get by
> using count window as the local operator.
>
> Best,
> Jark
>
>
>
> On Mon, 17 Jun 2019 at 17:48, vino yang  wrote:
>
> Hi Hequn,
>
> Thanks for your reply.
>
> The purpose of localKeyBy API is to provide a tool which can let users do
> pre-aggregation in the local. The behavior of the pre-aggregation is
> similar to keyBy API.
>
> So the three cases are different, I will describe them one by one:
>
> 1. input.keyBy(0).sum(1)
>
> *In this case, the result is event-driven, each event can produce one sum
> aggregation result and it is the latest one from the source start.*
>
> 2. input.localKeyBy(0).sum(1).keyBy(0).sum(1)
>
> *In this case, the semantic may have a problem, it would do the local sum
> aggregation and will produce the latest partial result from the source
> start for every event. *
> *These latest partial results from the same key are hashed to one node to
> do the global sum aggregation.*
> *In the global aggregation, when it received multiple partial results
> (they
> are all calculated from the source start) and sum them will get the wrong
> result.*
>
> 3. input.localKeyBy(0).countWi

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Tzu-Li (Gordon) Tai]

Congratulations Jincheng, great to have you on board :)

Cheers,
Gordon

On Tue, Jun 25, 2019, 11:31 AM Terry Wang  wrote:

> Congratulations Jincheng!
>
> > 在 2019年6月24日，下午11:08，Robert Metzger  写道：
> >
> > Hi all,
> >
> > On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
> now
> > part of the Apache Flink Project Management Committee (PMC).
> >
> > Jincheng has been a committer since July 2017. He has been very active on
> > Flink's Table API / SQL component, as well as helping with releases.
> >
> > Congratulations & Welcome Jincheng!
> >
> > Best,
> > Robert
>
>

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Jark Wu]

Hi Jeff,

Thanks for sharing the Zeppelin approach. I think it's a good idea to
leverage user's travis account.
In this way, we can have almost unlimited concurrent build jobs and
developers can restart build by themselves (currently only committers can
restart PR's build).

But I'm still not very clear how to integrate user's travis build into the
Flink pull request's build automatically. Can you explain more in detail?

Another question: does travis only build branches for user account?
My concern is that builds for PRs will rebase user's commits against
current master branch.
This will help us to find problems before merge.  Builds for branches will
lose the impact of new commits in master.
How does Zeppelin solve this problem?

Thanks again for sharing the idea.

Regards,
Jark

On Tue, 25 Jun 2019 at 11:01, Jeff Zhang  wrote:

> Hi Folks,
>
> Zeppelin meet this kind of issue before, we solve it by delegating each
> one's PR build to his travis account (Everyone can have 5 free slot for
> travis build).
> Apache account travis build is only triggered when PR is merged.
>
>
>
> Kurt Young  于2019年6月25日周二 上午10:16写道：
>
> > (Forgot to cc George)
> >
> > Best,
> > Kurt
> >
> >
> > On Tue, Jun 25, 2019 at 10:16 AM Kurt Young  wrote:
> >
> > > Hi Bowen,
> > >
> > > Thanks for bringing this up. We actually have discussed about this,
> and I
> > > think Till and George have
> > > already spend sometime investigating it. I have cced both of them, and
> > > maybe they can share
> > > their findings.
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Tue, Jun 25, 2019 at 10:08 AM Jark Wu  wrote:
> > >
> > >> Hi Bowen,
> > >>
> > >> Thanks for bringing this. We also suffered from the long build time.
> > >> I agree that we should focus on solving build capacity problem in the
> > >> thread.
> > >>
> > >> My observation is there is only one build is running, all the others
> > >> (other
> > >> PRs, master) are pending.
> > >> The pricing plan[1] of travis shows it can support concurrent build
> > jobs.
> > >> But I don't know which plan we are using, might be the free plan for
> > open
> > >> source.
> > >>
> > >> I cc-ed Chesnay who may have some experience on Travis.
> > >>
> > >> Regards,
> > >> Jark
> > >>
> > >> [1]: https://travis-ci.com/plans
> > >>
> > >> On Tue, 25 Jun 2019 at 08:11, Bowen Li  wrote:
> > >>
> > >> > Hi Steven,
> > >> >
> > >> > I think you may not read what I wrote. The discussion is about
> > "unstable
> > >> > build **capacity**", in another word "unstable / lack of build
> > >> resources",
> > >> > not "unstable build".
> > >> >
> > >> > On Mon, Jun 24, 2019 at 4:40 PM Steven Wu 
> > wrote:
> > >> >
> > >> > > long and sometimes unstable build is definitely a pain point.
> > >> > >
> > >> > > I suspect the build failure here in flink-connector-kafka is not
> > >> related
> > >> > to
> > >> > > my change. but there is no easy re-run the build on travis UI.
> > Google
> > >> > > search showed a trick of close-and-open the PR will trigger
> rebuild.
> > >> but
> > >> > > that could add noises to the PR activities.
> > >> > > https://travis-ci.org/apache/flink/jobs/54519
> > >> > >
> > >> > > travis-ci for my personal repo often failed with exceeding time
> > limit
> > >> > after
> > >> > > 4+ hours.
> > >> > > The job exceeded the maximum time limit for jobs, and has been
> > >> > terminated.
> > >> > >
> > >> > > On Mon, Jun 24, 2019 at 4:15 PM Bowen Li 
> > wrote:
> > >> > >
> > >> > > > https://travis-ci.org/apache/flink/builds/549681530  This build
> > >> > request
> > >> > > > has
> > >> > > > been sitting at **HEAD of the queue** since I first saw it at
> PST
> > >> > 10:30am
> > >> > > > (not sure how long it's been there before 10:30am). It's PST
> > 4:12pm
> > >> now
> > >> > > and
> > >> > > > it hasn't started yet.
> > >> > > >
> > >> > > > On Mon, Jun 24, 2019 at 2:48 PM Bowen Li 
> > >> wrote:
> > >> > > >
> > >> > > > > Hi devs,
> > >> > > > >
> > >> > > > > I've been experiencing the pain resulting from lack of stable
> > >> build
> > >> > > > > capacity on Travis for Flink PRs [1]. Specifically, I noticed
> > >> often
> > >> > > that
> > >> > > > no
> > >> > > > > build in the queue is making any progress for hours, and
> > suddenly
> > >> 5
> > >> > or
> > >> > > 6
> > >> > > > > builds kick off all together after the long pause. I'm at PST
> > >> > (UTC-08)
> > >> > > > time
> > >> > > > > zone, and I've seen pause can be as long as 6 hours from PST
> 9am
> > >> to
> > >> > 3pm
> > >> > > > > (let alone the time needed to drain the queue afterwards).
> > >> > > > >
> > >> > > > > I think this has greatly impacted our productivity. I've
> > >> experienced
> > >> > > that
> > >> > > > > PRs submitted in the early morning of PST time zone won't
> finish
> > >> > their
> > >> > > > > build until late night of the same day.
> > >> > > > >
> > >> > > > > So my questions are:
> > >> > > > >
> > >> > > > > - Has anyone else experienced the same problem or have similar
> > >> > > 

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Xingcan Cui]

Congratulations Jincheng and thanks for all you’ve done!

Cheers,
Xingcan

> On Jun 25, 2019, at 1:59 AM, Tzu-Li (Gordon) Tai  wrote:
> 
> Congratulations Jincheng, great to have you on board :)
> 
> Cheers,
> Gordon
> 
> On Tue, Jun 25, 2019, 11:31 AM Terry Wang  wrote:
> 
>> Congratulations Jincheng!
>> 
>>> 在 2019年6月24日，下午11:08，Robert Metzger  写道：
>>> 
>>> Hi all,
>>> 
>>> On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
>> now
>>> part of the Apache Flink Project Management Committee (PMC).
>>> 
>>> Jincheng has been a committer since July 2017. He has been very active on
>>> Flink's Table API / SQL component, as well as helping with releases.
>>> 
>>> Congratulations & Welcome Jincheng!
>>> 
>>> Best,
>>> Robert
>> 
>> 

答复: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Guo Shirong]

Congratulations Jincheng!

Good job!



From My Windows 10 Email App




发件人: Dian Fu 
发送时间: Monday, June 24, 2019 11:43:20 PM
收件人: dev
主题: Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

Congratulations Jincheng!

On Mon, Jun 24, 2019 at 11:09 PM Robert Metzger  wrote:

> Hi all,
>
> On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
> part of the Apache Flink Project Management Committee (PMC).
>
> Jincheng has been a committer since July 2017. He has been very active on
> Flink's Table API / SQL component, as well as helping with releases.
>
> Congratulations & Welcome Jincheng!
>
> Best,
> Robert
>

Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Dawid Wysakowicz]

Congratulations!

On 25/06/2019 08:06, Xingcan Cui wrote:
> Congratulations Jincheng and thanks for all you’ve done!
>
> Cheers,
> Xingcan
>
>> On Jun 25, 2019, at 1:59 AM, Tzu-Li (Gordon) Tai  wrote:
>>
>> Congratulations Jincheng, great to have you on board :)
>>
>> Cheers,
>> Gordon
>>
>> On Tue, Jun 25, 2019, 11:31 AM Terry Wang  wrote:
>>
>>> Congratulations Jincheng!
>>>
 在 2019年6月24日，下午11:08，Robert Metzger  写道：

 Hi all,

 On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is
>>> now
 part of the Apache Flink Project Management Committee (PMC).

 Jincheng has been a committer since July 2017. He has been very active on
 Flink's Table API / SQL component, as well as helping with releases.

 Congratulations & Welcome Jincheng!

 Best,
 Robert
>>>



signature.asc
Description: OpenPGP digital signature

Re: [Discuss] FLIP-43: Savepoint Connector

[Jark Wu]

Thanks for the awesome FLIP.

I think it will be very useful in state migration scenario. We are also
looking for a state reuse solution for SQL jobs. And I think this feature
will help a lot.
Looking forward to have it in the near future.

Regarding to the naming, I'm +1 to "State Processing API". Should we also
update the FLIP name in confluence?

Btw, what do you think to add a shortcut API for changing max parallelism
for savepoints? This is a very common scenario.
But from my understanding, it needs to do a lot of trivial thing to achieve
it under current API.

Best,
Jark


On Wed, 5 Jun 2019 at 10:52, Tzu-Li (Gordon) Tai 
wrote:

> On Wed, Jun 5, 2019 at 6:39 AM Xiaowei Jiang  wrote:
>
> >  Hi Gordon & Seth, this looks like a very useful feature for analyze and
> > manage states.
> > I agree that using DataSet is probably the most practical choice right
> > now. But in the longer adding the TableAPI support for this will be nice.
> >
>
> Agreed. Migrating this API in the future to the TableAPI is definitely
> something we have considered.
>
>
> > When analyzing the savepoint, I assume that the state backend restores
> the
> > state first? This approach is generic and works for any state backend.
>
>
> Yes, that is correct. The process of reading state in snapshots is
> currently:
> 1) the metadata file is read when creating the input splits for the
> InputFormat. Each input split is assigned operator state and key-group
> state handles.
> 2) For each input split, a state backend is launched and is restored with
> all state of the assigned state handles. Only partially some state is
> transformed into DataSets (using the savepoint.read*State(...) methods).
>
>
> > However, sometimes it may be more efficient to directly analyzing the
> > files on DFS without copying. We can probably add interface to allow
> state
> > backend optimize such behavior in the future.
>
>
> That sounds like an interesting direction, though at the moment it may only
> make sense for full savepoints / checkpoints.
> One blocker for enabling this, is having the type information of state
> available in the snapshot metadata file so that schema / type of state is
> known before actually reading state.
> Making state schema / type information available in the metadata file is
> already a recurring discussion in this thread that would be useful for not
> only this feature you mentioned, but also for features like SQL integration
> in the future.
> Therefore, this seems to be a reasonable next step when extending on top of
> the initial scope of the API proposed in the FLIP.
>
>
> > Also a quick question on the example in wiki: DataSet keyedState
> =
> > operator.readKeyedState("uid", new ReaderFunction());Should
> > operator.readKeyedState  be replaced with savepoint.readKeyedState here?
> >
>
> Correct, this is indeed a typo. I've corrected this in the FLIP.
>
> Cheers,
> Gordon
>
>
> >
> > Regards,Xiaowei
> >
> > On Tuesday, June 4, 2019, 6:56:00 AM PDT, Aljoscha Krettek <
> > aljos...@apache.org> wrote:
> >
> >  +1 I think is is a very valuable new additional and we should try and
> not
> > get stuck on trying to design the perfect solution for everything
> >
> > > On 4. Jun 2019, at 13:24, Tzu-Li (Gordon) Tai 
> > wrote:
> > >
> > > +1 to renaming it as State Processing API and adding it under the
> > > flink-libraries module.
> > >
> > > I also think we can start with the development of the feature. From the
> > > feedback so far, it seems like we're in a good spot to add in at least
> > the
> > > initial version of this API, hopefully making it ready for 1.9.0.
> > >
> > > Cheers,
> > > Gordon
> > >
> > > On Tue, Jun 4, 2019 at 7:14 PM Seth Wiesman 
> wrote:
> > >
> > >> It seems like a recurring piece of feedback was a different name. I’d
> > like
> > >> to propose moving the functionality to the libraries module and naming
> > this
> > >> the State Processing API.
> > >>
> > >> Seth
> > >>
> > >>> On May 31, 2019, at 3:47 PM, Seth Wiesman 
> wrote:
> > >>>
> > >>> The SavepointOutputFormat only writes out the savepoint metadata file
> > >> and should be mostly ignored.
> > >>>
> > >>> The actual state is written out by stream operators and tied into the
> > >> flink runtime[1, 2, 3].
> > >>>
> > >>> This is the most important part and the piece that I don’t think can
> be
> > >> reasonably extracted.
> > >>>
> > >>> Seth
> > >>>
> > >>> [1]
> > >>
> >
> https://github.com/sjwiesman/flink/blob/savepoint-connector/flink-connectors/flink-connector-savepoint/src/main/java/org/apache/flink/connectors/savepoint/operators/KeyedStateBootstrapOperator.java#L84
> > >>>
> > >>> [2]
> > >>
> >
> https://github.com/sjwiesman/flink/blob/savepoint-connector/flink-connectors/flink-connector-savepoint/src/main/java/org/apache/flink/connectors/savepoint/output/SnapshotUtils.java
> > >>>
> > >>> [3]
> > >>
> >
> https://github.com/sjwiesman/flink/blob/savepoint-connector/flink-connectors/flink-connector-savepoint/src/main/java/org/apache/

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

[vino yang]

Hi Kurt,

Answer your questions:

a) Sorry, I just updated the Google doc, still have no time update the
FLIP, will update FLIP as soon as possible.
About your description at this point, I have a question, what does it mean:
how do we combine with
`AggregateFunction`?

I have shown you the examples which Flink has supported:

   - input.localKeyBy(0).aggregate()
   - input.localKeyBy(0).window().aggregate()

You can show me a example about how do we combine with `AggregateFuncion`
through your localAggregate API.

About the example, how to do the local aggregation for AVG, consider this
code:









*DataStream> source = null; source .localKeyBy(0)
.timeWindow(Time.seconds(60)) .aggregate(agg1, new
WindowFunction, Tuple3, String,
TimeWindow>() {}) .keyBy(0) .timeWindow(Time.seconds(60)) .aggregate(agg2,
new WindowFunction, Tuple2, String,
TimeWindow>());*

*agg1:*
*signature : new AggregateFunction, Tuple2, Tuple2>() {}*
*input param type: Tuple2 f0: key, f1: value*
*intermediate result type: Tuple2, f0: local aggregated sum;
f1: local aggregated count*
*output param type:  Tuple2, f0: local aggregated sum; f1:
local aggregated count*

*agg2:*
*signature: new AggregateFunction, Long,
Tuple2>() {},*
*input param type: Tuple3, f0: key, f1:  local
aggregated sum; f2: local aggregated count*

*intermediate result type: Long  avg result*
*output param type:  Tuple2 f0: key, f1 avg result*

For sliding window, we just need to change the window type if users want to
do.
Again, we try to give the design and implementation in the DataStream
level. So I believe we can match all the requirements(It's just that the
implementation may be different) comes from the SQL level.

b) Yes, Theoretically, your thought is right. But in reality, it cannot
bring many benefits.
If we want to get the benefits from the window API, while we do not reuse
the window operator? And just copy some many duplicated code to another
operator?

c) OK, I agree to let the state backend committers join this discussion.

Best,
Vino


Kurt Young  于2019年6月24日周一 下午6:53写道：

> Hi vino,
>
> One thing to add,  for a), I think use one or two examples like how to do
> local aggregation on a sliding window,
> and how do we do local aggregation on an unbounded aggregate, will do a lot
> help.
>
> Best,
> Kurt
>
>
> On Mon, Jun 24, 2019 at 6:06 PM Kurt Young  wrote:
>
> > Hi vino,
> >
> > I think there are several things still need discussion.
> >
> > a) We all agree that we should first go with a unified abstraction, but
> > the abstraction is not reflected by the FLIP.
> > If your answer is "locakKeyBy" API, then I would ask how do we combine
> > with `AggregateFunction`, and how do
> > we do proper local aggregation for those have different intermediate
> > result type, like AVG. Could you add these
> > to the document?
> >
> > b) From implementation side, reusing window operator is one of the
> > possible solutions, but not we base on window
> > operator to have two different implementations. What I understanding is,
> > one of the possible implementations should
> > not touch window operator.
> >
> > c) 80% of your FLIP content is actually describing how do we support
> local
> > keyed state. I don't know if this is necessary
> > to introduce at the first step and we should also involve committers work
> > on state backend to share their thoughts.
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, Jun 24, 2019 at 5:17 PM vino yang  wrote:
> >
> >> Hi Kurt,
> >>
> >> You did not give more further different opinions, so I thought you have
> >> agreed with the design after we promised to support two kinds of
> >> implementation.
> >>
> >> In API level, we have answered your question about pass an
> >> AggregateFunction to do the aggregation. No matter introduce localKeyBy
> >> API
> >> or not, we can support AggregateFunction.
> >>
> >> So what's your different opinion now? Can you share it with us?
> >>
> >> Best,
> >> Vino
> >>
> >> Kurt Young  于2019年6月24日周一 下午4:24写道：
> >>
> >> > Hi vino,
> >> >
> >> > Sorry I don't see the consensus about reusing window operator and keep
> >> the
> >> > API design of localKeyBy. But I think we should definitely more
> thoughts
> >> > about this topic.
> >> >
> >> > I also try to loop in Stephan for this discussion.
> >> >
> >> > Best,
> >> > Kurt
> >> >
> >> >
> >> > On Mon, Jun 24, 2019 at 3:26 PM vino yang 
> >> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I am happy we have a wonderful discussion and received many valuable
> >> > > opinions in the last few days.
> >> > >
> >> > > Now, let me try to summarize what we have reached consensus about
> the
> >> > > changes in the design.
> >> > >
> >> > >- provide a unified abstraction to support two kinds of
> >> > implementation;
> >> > >- reuse WindowOperator and try to enhance it so that we can make
> >> the
> >> > >intermediate result of the local aggregation can be buffered and
> >> > > flushed to
> >> > >support two kinds of implementation;
> >> > >

[jira] [Created] (FLINK-12969) Remove dependencies on RelNode from TableImpl in blink planner

[godfrey he (JIRA)]

godfrey he created FLINK-12969:
--

 Summary: Remove dependencies on RelNode from TableImpl in blink 
planner
 Key: FLINK-12969
 URL: https://issues.apache.org/jira/browse/FLINK-12969
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: godfrey he
Assignee: godfrey he


This issue aims to remove dependencies on RelNode from TableImpl in blink 
planner, just as 
[FLINK-12737|https://issues.apache.org/jira/browse/FLINK-12737] does.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12970) Support writing Hive complex types

[Rui Li (JIRA)]

Rui Li created FLINK-12970:
--

 Summary: Support writing Hive complex types
 Key: FLINK-12970
 URL: https://issues.apache.org/jira/browse/FLINK-12970
 Project: Flink
  Issue Type: Sub-task
Reporter: Rui Li
Assignee: Rui Li


HiveTableOutputFormat currently only supports wring primitive types. Need to 
support complex types like array and map.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)