[jira] [Created] (FLINK-13092) Translate "kubernetes" page into Chinese

[yelun (JIRA)]

yelun created FLINK-13092:
-

 Summary: Translate "kubernetes" page into Chinese
 Key: FLINK-13092
 URL: https://issues.apache.org/jira/browse/FLINK-13092
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation, Documentation
Affects Versions: 1.9.0
Reporter: yelun
 Fix For: 1.9.0


The page url is 
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html
The markdown file is located in flink/docs/ops/deployment/kubernetes.zh.md



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Application for contributor permission

[jincheng sun]

Hi wudi,

Thanks for your confirmation, usually, the contributor permissions are
authorized as needed. I have not found that you have created Flink-related
JIRA, nor have found any relevant discussions about your participation(by
your id). So which JIRAs do you want to take now,  I appreciate it if you
can list them?  and then I can give you the contributor permission. :)

BTW, here the wiki talk about how to contribute Apache Flink, more detail
can be found here:
https://flink.apache.org/contributing/how-to-contribute.html

Best,
Jincheng

[1]

吴迪  于2019年7月4日周四 上午10:38写道：

> Yes,it's mine.
>
>
> -邮件原件-
> 发件人: jincheng sun [mailto:sunjincheng...@gmail.com]
> 发送时间: 2019年7月4日 10:26
> 收件人: dev
> 主题: Re: Application for contributor permission
>
> Hi wudi，
>
> Thank you for your willingness to contribute to the Flink community，just
> confirm, does the follows page is your profile?
>
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=wuyanzu
>
> Best, Jincheng
>
> 吴迪  于2019年7月4日周四 上午9:52写道：
>
> > Hi ,
> >
> > I want to contribute to Apache Flink, and my JIRA ID is wuyanzu.
> >
> > Would you please give me the contributor permission?
> >
> > Thx.
> >
> >
> >
>

Re: RE: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component

[vino yang]

Hi Jiayi,

Thanks for your comments.

It's valuable. I have accepted it and refined my design document.

You can have another review when you have time.

Best,
Vino

bupt_ljy  于2019年7月3日周三 下午2:48写道：

> Hi vino,
> Big +1 for this.
>
> Glad to see new progress on this topic! I’ve left some comments on it.
>
>
> Best Regards,
>
> Jiayi Liao
>
>  Original Message
> *Sender:* vino yang
> *Recipient:* Georgi Stoyanov
> *Cc:* dev; user; Stefan
> Richter; Aljoscha Krettek;
> kklou...@gmail.com; Stephan Ewen;
> l...@apache.org; Tzu-Li (Gordon) Tai
> *Date:* Tuesday, Jul 2, 2019 16:45
> *Subject:* Re: RE: [DISCUSS] Improve Queryable State and introduce
> aQueryServerProxy component
>
> Hi all,
>
> In the past, I have tried to further refine the design of this topic
> thread and wrote a design document to give more detailed design images and
> text description, so that it is more conducive to discussion.[1]
>
> Note: The document is not yet completed, for example, the "Implementation"
> section is missing. Therefore, it is still in an open discussion state. I
> will improve the rest while listening to the opinions of the community.
>
> Welcome and appreciate more discussions and feedback.
>
> Best,
> Vino
>
> [1]:
> https://docs.google.com/document/d/181qYVIiHQGrc3hCj3QBn1iEHF4bUztdw4XO8VSaf_uI/edit?usp=sharing
>
>
> yanghua1127  于2019年6月7日周五 下午11:32写道：
>
>> Hi Georgi,
>>
>> Thanks for your feedback. And glad to hear you are using queryable state.
>>
>> I agree that implementation of option 1 is easier than others. However,
>> when we design the new architecture we need to consider more aspects .e.g.
>> scalability. So it seems option 3 is more suitable. Actually, some
>> committers such as Stefan, Gordon and Aljoscha have given me feedback and
>> direction.
>>
>> Currently, I am writing the design document. If it is ready to be
>> presented. I will copy to this thread and we can discuss further details.
>>
>> 
>> Best,
>> Vino
>>
>>
>> On 2019-06-07 19:03 , Georgi Stoyanov  Wrote:
>>
>> Hi Vino,
>>
>>
>>
>> I was investigating the current architecture and AFAIK the first proposal
>> will be a lot easier to implement, cause currently JM has the information
>> about the states (where, which etc thanks to KvStateLocationRegistry.
>> Correct me if I’m wrong)
>>
>> We are using the feature and it’s indeed not very cool to iterate trough
>> ports, check which TM is the responsible one etc etc.
>>
>>
>>
>> It will be very useful if someone from the committers joins the topic and
>> give us some insights what’s going to happen with that feature.
>>
>>
>>
>>
>>
>> Kind Regards,
>>
>> Georgi
>>
>>
>>
>>
>>
>>
>>
>> *From:* vino yang 
>> *Sent:* Thursday, April 25, 2019 5:18 PM
>> *To:* dev ; user 
>> *Cc:* Stefan Richter ; Aljoscha Krettek <
>> aljos...@apache.org>; kklou...@gmail.com
>> *Subject:* [DISCUSS] Improve Queryable State and introduce a
>> QueryServerProxy component
>>
>>
>>
>> Hi all,
>>
>>
>>
>> I want to share my thought with you about improving the queryable state
>> and introducing a QueryServerProxy component.
>>
>>
>>
>> I think the current queryable state's client is hard to use. Because it
>> needs users to know the TaskManager's address and proxy's port. Actually,
>> some business users who do not have good knowledge about the Flink's inner
>> or runtime in production. However, sometimes they need to query the values
>> of states.
>>
>>
>>
>> IMO, the reason caused this problem is because of the queryable state's
>> architecture. Currently, the queryable state clients interact with
>> query state client proxy components which host on each TaskManager. This
>> design is difficult to encapsulate the point of change and exposes too much
>> detail to the user.
>>
>>
>>
>> My personal idea is that we could introduce a really queryable state
>> server, named e.g. *QueryStateProxyServer* which would delegate all the
>> query state request and query the local registry then redirect the request
>> to the specific *QueryStateClientProxy*(runs on each TaskManager). The
>> server is the users really want to care about. And it would make the users
>> ignorant to the TaskManagers' address and proxies' port. The current
>> *QueryStateClientProxy* would become *QueryStateProxyClient*.
>>
>>
>>
>> Generally speaking, the roles of the QueryStateProxyServer list below:
>>
>>
>>
>>- works as all the query client's proxy to receive all the request
>>and send response;
>>- a router to redirect the real query requests to the specific proxy
>>client;
>>- maintain route table registry (state <-> TaskManager,
>>TaskManager<->proxy client address)
>>- more fine-granted control, such as cache result, ACL, TTL, SLA(rate
>>limit) and so on
>>
>> About the implementation, there are three opts:
>>
>>
>>
>> opt 1:
>>
>>
>>
>> Let the JobManager acts as the query proxy server.
>>
>> ·  pros: reuse the exists JM, do not need to introduce a new process can
>> reduce the complexity;
>>
>> ·  cons: would ma

[jira] [Created] (FLINK-13093) Provide an easy way to modify max parallelism using the State Processor API

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-13093:
---

 Summary: Provide an easy way to modify max parallelism using the 
State Processor API
 Key: FLINK-13093
 URL: https://issues.apache.org/jira/browse/FLINK-13093
 Project: Flink
  Issue Type: Sub-task
Reporter: Tzu-Li (Gordon) Tai


Currently, the State Processor API does not easily allow one to modify the max 
parallelism of a job. To do that with the current state of the API, one would 
have to read all state that exists in a loaded {{ExistingSavepoint}}, extract 
them as {{DataSet}}s, and then create a {{NewSavepoint}} that has the new max 
parallelism with all the extracted data sets bootstrapped as new state.

It would be nice if the user could simply do something like the following (API 
is TBD):
{code}
ExistingSavepoint savepoint = Savepoint.load("path", env, backend);
savepoint.modifyMaxParallelism("newPath", newParallelism);
{code}

Under the hood, a batch job is launched that repartitions all existing operator 
state using the new max parallelism, and writes to the re-partitioned state 
data into the new savepoint path.

The API can be designed such that the user cannot modify the max parallelism 
and add / new operators at the same time, to not over complicate the batch job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13094) Provide an easy way to read / bootstrap timers using the State Processor API

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-13094:
---

 Summary: Provide an easy way to read / bootstrap timers using the 
State Processor API
 Key: FLINK-13094
 URL: https://issues.apache.org/jira/browse/FLINK-13094
 Project: Flink
  Issue Type: Sub-task
  Components: API / DataStream, Runtime / State Backends
Reporter: Tzu-Li (Gordon) Tai
Assignee: Seth Wiesman






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13095) Provide an easy way to read / bootstrap window state using the State Processor API

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-13095:
---

 Summary: Provide an easy way to read / bootstrap window state 
using the State Processor API
 Key: FLINK-13095
 URL: https://issues.apache.org/jira/browse/FLINK-13095
 Project: Flink
  Issue Type: Sub-task
  Components: API / DataStream, Runtime / State Backends
Reporter: Tzu-Li (Gordon) Tai
Assignee: Seth Wiesman






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Publish the PyFlink into PyPI

[jincheng sun]

Hi All,

Thanks for the feedback @Chesnay Schepler  @Dian!

I think using `apache-flink` for the project name also makes sense to me.
due to we should always keep in mind that Flink is owned by Apache. (And
beam also using this pattern `apache-beam` for Python API)

Regarding the Python API release with the JAVA JARs, I think the principle
of consideration is the convenience of the user. So, Thanks for the
explanation @Dian!

And your right @Chesnay Schepler   we can't make a
hasty decision and we need more people's opinions！

So, I appreciate it if anyone can give us feedback and suggestions!

Best,
Jincheng




Chesnay Schepler  于2019年7月3日周三 下午8:46写道：

> So this would not be a source release then, but a full-blown binary
> release.
>
> Maybe it is just me, but I find it a bit suspect to ship an entire java
> application via PyPI, just because there's a Python API for it.
>
> We definitely need input from more people here.
>
> On 03/07/2019 14:09, Dian Fu wrote:
> > Hi Chesnay,
> >
> > Thanks a lot for the suggestions.
> >
> > Regarding “distributing java/scala code to PyPI”:
> > The Python Table API is just a wrapper of the Java Table API and without
> the java/scala code, two steps will be needed to set up an environment to
> execute a Python Table API program:
> > 1) Install pyflink using "pip install apache-flink"
> > 2) Download the flink distribution and set the FLINK_HOME to it.
> > Besides, users have to make sure that the manually installed Flink is
> compatible with the pip installed pyflink.
> >
> > Bundle the java/scala code inside the Python package will eliminate step
> 2) and makes it more simple for users to install pyflink. There was a short
> discussion  on this in
> Spark community and they finally decide to package the java/scala code in
> the python package. (BTW, PySpark only bundle the jars of scala 2.11).
> >
> > Regards,
> > Dian
> >
> >> 在 2019年7月3日，下午7:13，Chesnay Schepler  写道：
> >>
> >> The existing artifact in the pyflink project was neither released by
> the Flink project / anyone affiliated with it nor approved by the Flink PMC.
> >>
> >> As such, if we were to use this account I believe we should delete it
> to not mislead users that this is in any way an apache-provided
> distribution. Since this goes against the users wishes, I would be in favor
> of creating a separate account, and giving back control over the pyflink
> account.
> >>
> >> My take on the raised points:
> >> 1.1) "apache-flink"
> >> 1.2)  option 2
> >> 2) Given that we only distribute python code there should be no reason
> to differentiate between scala versions. We should not be distributing any
> java/scala code and/or modules to PyPi. Currently, I'm a bit confused about
> this question and wonder what exactly we are trying to publish here.
> >> 3) The should be treated as any other source release; i.e., it needs a
> LICENSE and NOTICE file, signatures and a PMC vote. My suggestion would be
> to make this part of our normal release process. There will be _one_ source
> release on dist.apache.org encompassing everything, and a separate python
> of focused source release that we push to PyPi. The LICENSE and NOTICE
> contained in the python source release must also be present in the source
> release of Flink; so basically the python source release is just the
> contents of flink-python module the maven pom.xml, with no other special
> sauce added during the release process.
> >>
> >> On 02/07/2019 05:42, jincheng sun wrote:
> >>> Hi all,
> >>>
> >>> With the effort of FLIP-38 [1], the Python Table API(without UDF
> support
> >>> for now) will be supported in the coming release-1.9.
> >>> As described in "Build PyFlink"[2], if users want to use the Python
> Table
> >>> API, they can manually install it using the command:
> >>> "cd flink-python && python3 setup.py sdist && pip install
> dist/*.tar.gz".
> >>>
> >>> This is non-trivial for users and it will be better if we can follow
> the
> >>> Python way to publish PyFlink to PyPI
> >>> which is a repository of software for the Python programming language.
> Then
> >>> users can use the standard Python package
> >>> manager "pip" to install PyFlink: "pip install pyflink". So, there are
> some
> >>> topic need to be discussed as follows:
> >>>
> >>> 1. How to publish PyFlink to PyPI
> >>>
> >>> 1.1 Project Name
> >>>   We need to decide the project name of PyPI to use, for example,
> >>> apache-flink,  pyflink, etc.
> >>>
> >>>  Regarding to the name "pyflink", it has already been registered by
> >>> @ueqt and there is already a package '1.0' released under this project
> >>> which is based on flink-libraries/flink-python.
> >>>
> >>> @ueqt has kindly agreed to give this project back to the
> community. And
> >>> he has requested that the released package '1.0' should not be removed
> as
> >>> it has already been used in their company.
> >>>
> >>>  So we need to decide whether to use the name '

JIRA Contributor apply

[Nuist_Hook]

Hi, I want to contribute to Apache Flink. Would you please give me the 
contributor permission? My JIRA ID is geyx881105.

[jira] [Created] (FLINK-13096) Typo in Utils

[Charles Xu (JIRA)]

Charles Xu created FLINK-13096:
--

 Summary: Typo in Utils
 Key: FLINK-13096
 URL: https://issues.apache.org/jira/browse/FLINK-13096
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Affects Versions: 1.8.1, 1.8.0
Reporter: Charles Xu


Typo in Utils.getEnvironmentVariables() method's comment : defintion  -> 
definition 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: RE: [DISCUSS] Improve Queryable State and introduce aQueryServerProxy component

[Yu Li]

Hi Vino,

Thanks for the write up. I've left some comments on google doc, please
check. Thanks.

Best Regards,
Yu


On Thu, 4 Jul 2019 at 15:37, vino yang  wrote:

> Hi Jiayi,
>
> Thanks for your comments.
>
> It's valuable. I have accepted it and refined my design document.
>
> You can have another review when you have time.
>
> Best,
> Vino
>
> bupt_ljy  于2019年7月3日周三 下午2:48写道：
>
> > Hi vino,
> > Big +1 for this.
> >
> > Glad to see new progress on this topic! I’ve left some comments on it.
> >
> >
> > Best Regards,
> >
> > Jiayi Liao
> >
> >  Original Message
> > *Sender:* vino yang
> > *Recipient:* Georgi Stoyanov
> > *Cc:* dev; user; Stefan
> > Richter; Aljoscha Krettek;
> > kklou...@gmail.com; Stephan Ewen;
> > l...@apache.org; Tzu-Li (Gordon) Tai<
> tzuli...@apache.org>
> > *Date:* Tuesday, Jul 2, 2019 16:45
> > *Subject:* Re: RE: [DISCUSS] Improve Queryable State and introduce
> > aQueryServerProxy component
> >
> > Hi all,
> >
> > In the past, I have tried to further refine the design of this topic
> > thread and wrote a design document to give more detailed design images
> and
> > text description, so that it is more conducive to discussion.[1]
> >
> > Note: The document is not yet completed, for example, the
> "Implementation"
> > section is missing. Therefore, it is still in an open discussion state. I
> > will improve the rest while listening to the opinions of the community.
> >
> > Welcome and appreciate more discussions and feedback.
> >
> > Best,
> > Vino
> >
> > [1]:
> >
> https://docs.google.com/document/d/181qYVIiHQGrc3hCj3QBn1iEHF4bUztdw4XO8VSaf_uI/edit?usp=sharing
> >
> >
> > yanghua1127  于2019年6月7日周五 下午11:32写道：
> >
> >> Hi Georgi,
> >>
> >> Thanks for your feedback. And glad to hear you are using queryable
> state.
> >>
> >> I agree that implementation of option 1 is easier than others. However,
> >> when we design the new architecture we need to consider more aspects
> .e.g.
> >> scalability. So it seems option 3 is more suitable. Actually, some
> >> committers such as Stefan, Gordon and Aljoscha have given me feedback
> and
> >> direction.
> >>
> >> Currently, I am writing the design document. If it is ready to be
> >> presented. I will copy to this thread and we can discuss further
> details.
> >>
> >> 
> >> Best,
> >> Vino
> >>
> >>
> >> On 2019-06-07 19:03 , Georgi Stoyanov  Wrote:
> >>
> >> Hi Vino,
> >>
> >>
> >>
> >> I was investigating the current architecture and AFAIK the first
> proposal
> >> will be a lot easier to implement, cause currently JM has the
> information
> >> about the states (where, which etc thanks to KvStateLocationRegistry.
> >> Correct me if I’m wrong)
> >>
> >> We are using the feature and it’s indeed not very cool to iterate trough
> >> ports, check which TM is the responsible one etc etc.
> >>
> >>
> >>
> >> It will be very useful if someone from the committers joins the topic
> and
> >> give us some insights what’s going to happen with that feature.
> >>
> >>
> >>
> >>
> >>
> >> Kind Regards,
> >>
> >> Georgi
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> *From:* vino yang 
> >> *Sent:* Thursday, April 25, 2019 5:18 PM
> >> *To:* dev ; user 
> >> *Cc:* Stefan Richter ; Aljoscha Krettek <
> >> aljos...@apache.org>; kklou...@gmail.com
> >> *Subject:* [DISCUSS] Improve Queryable State and introduce a
> >> QueryServerProxy component
> >>
> >>
> >>
> >> Hi all,
> >>
> >>
> >>
> >> I want to share my thought with you about improving the queryable state
> >> and introducing a QueryServerProxy component.
> >>
> >>
> >>
> >> I think the current queryable state's client is hard to use. Because it
> >> needs users to know the TaskManager's address and proxy's port.
> Actually,
> >> some business users who do not have good knowledge about the Flink's
> inner
> >> or runtime in production. However, sometimes they need to query the
> values
> >> of states.
> >>
> >>
> >>
> >> IMO, the reason caused this problem is because of the queryable state's
> >> architecture. Currently, the queryable state clients interact with
> >> query state client proxy components which host on each TaskManager. This
> >> design is difficult to encapsulate the point of change and exposes too
> much
> >> detail to the user.
> >>
> >>
> >>
> >> My personal idea is that we could introduce a really queryable state
> >> server, named e.g. *QueryStateProxyServer* which would delegate all the
> >> query state request and query the local registry then redirect the
> request
> >> to the specific *QueryStateClientProxy*(runs on each TaskManager). The
> >> server is the users really want to care about. And it would make the
> users
> >> ignorant to the TaskManagers' address and proxies' port. The current
> >> *QueryStateClientProxy* would become *QueryStateProxyClient*.
> >>
> >>
> >>
> >> Generally speaking, the roles of the QueryStateProxyServer list below:
> >>
> >>
> >>
> >>- works as all the query client's proxy to receive all the request
> >>and send response;
> >>- a router to redire

[jira] [Created] (FLINK-13097) Buffer depletion in SimpleCollectingOutputView throws non-obvious EOFException

[JIRA]

Cyrille Chépélov created FLINK-13097:


 Summary: Buffer depletion in SimpleCollectingOutputView throws 
non-obvious EOFException 
 Key: FLINK-13097
 URL: https://issues.apache.org/jira/browse/FLINK-13097
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.8.0, 1.7.2
 Environment: SCIO + BEAM + Flink under scala 2.12, java 8

Reporter: Cyrille Chépélov


When SimpleCollectingOutputView is used, records are collected into a 
pre-allocated MemorySegmentSource. 

In case of depletion, the SimpleCollectingOutputView#nextSegment method throws 
EOFException without a message.

This can be non-obvious to diagnose as a newcomer, as 
* the Java SDK documentation strongly suggests EOFException is related to an 
inability to read further (whereas in this context, the exception materializes 
an inability to _write_ further)
* awareness than pre-allocated, fixed-size buffers are in play may not 
necessarily be expected of a newcomer to flink.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Application to Flink Contributor

[Yi Huang]

Hi,

I would like to contribute to Apache Flink and my Jira ID is huangyi345.

Thanks.
Regards

Re: [DISCUSS]Support Upsert mode for Streaming Non-window FlatAggregate

[jincheng sun]

Hi All,

@Kurt Young  one user-defined table aggregate function
can be used in both with(out) keys case, and we do not introduce any other
aggregations. just like the explanation from @Hequn.

@Hequn Cheng  thanks for your explanation!

One thing should be mentioned here:

When a user uses a User-defined table aggregate function (UDTAGG), he must
understand the behavior of the UDTAGG, including the return type and the
characteristics of the returned data. such as the key fields.  So although
using `withKeys` approach it is not rigorous enough(we do not need) but
intuitive enough, considering that if `flatAggregate` is followed by an
`upsertSink`, then the user must know the keys of the output of UDTAGG
clearly, otherwise the keys of `upsertSink` cannot be defined. So I still
prefer the `withKeys` solution by now.

Looking forward to any feedback from all of you!

Best,
Jincheng



Hequn Cheng  于2019年7月1日周一 下午5:35写道：

> Hi Kurt,
>
> Thanks for your questions. Here are my thoughts.
>
> > if I want to write such kind function, should I make sure that this
> function is used with some keys?
> The key information may not be used. We can also use RetractSink to emit
> the table directly.
>
> >  If I need a use case to calculate topn without key, should I write
> another function or I can reuse previous one.
> For the TopN example, you can reuse the previous function if you don't care
> about the key information.
>
> So, the key information is only an indicator(or a description), not an
> operator, as Jincheng mentioned above.
> We do not need to change the function logic and it will not add any other
> aggregations.
>
> BTW, we have three approaches in the document. Approach 1 defines keys on
> API level as we think it's more common to define keys on Table.
> While approach 3 defines keys in the TableAggregateFunction which is more
> precise but it is not very clear for Table users. So, we should take all
> these into consideration, and make the decision in this discussion thread.
>
> You can take a look at the document and welcome any suggestions or other
> better solutions.
>
> Best, Hequn
>
>
> On Mon, Jul 1, 2019 at 12:13 PM Kurt Young  wrote:
>
> > Hi Jincheng,
> >
> > Thanks for the clarification. Take 'TopN' for example, if I want to write
> > such kind function,
> > should I make sure that this function is used with some keys? If I need a
> > use case to calculate
> > topn without key, should I write another function or I can reuse previous
> > one.
> >
> > I'm not sure about the idea of this does not involve semantic changes. To
> > me, it sounds like
> > we are doing another nested aggregation inside the table
> > which TableAggregateFunction emits.
> >
> > Maybe I'm not familiar with this function enough, hope you can help me to
> > understand.
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, Jul 1, 2019 at 11:59 AM jincheng sun 
> > wrote:
> >
> > > Hi Kurt,
> > >
> > > Thanks for your questions, I am glad to share my thoughts here:
> > >
> > > My question is, will that effect the logic ofTableAggregateFunction
> user
> > > > wrote? Should the user know that there will a key and make some
> changes
> > > to
> > > > this function?
> > >
> > >
> > > No, the keys information depends on the implementation of the
> > > TableAggregateFunction.
> > > For example, for a `topN` user defined TableAggregateFunction, we can
> > only
> > > use the `keys` if the `topN` contains `rankid` in the output. You can
> > > treat the
> > > `keys` like an indicator.
> > >
> > > If not, how will framework deal with the output of user's
> > > > TableAggregateFunction.  if user output multiple rows with the same
> > key,
> > > > should be latter one replace previous ones?
> > >
> > >
> > > If a TableAggregateFunction outputs multiple rows with the same key,
> the
> > > latter one should replace the previous one, either with upsert mode or
> > > retract mode. i.e., Whether the user defines the Key or not, the Flink
> > > framework should ensure the correctness of the semantics.
> > >
> > > At present, the problem we are discussing does not involve semantic
> > > changes. The definition of keys is to support non-window flatAggregate
> on
> > > upsert mode. (The upsert mode is already supported in the flink
> > framework.
> > > The current discussion only needs to inform the framework that the keys
> > > information, which is the `withKeys` API we discussing.)
> > >
> > > Welcome any other feedbacks :)
> > >
> > > Best,
> > > Jincheng
> > >
> > > Kurt Young  于2019年7月1日周一 上午9:23写道：
> > >
> > > > Hi,
> > > >
> > > > I have a question about the key information of
> TableAggregateFunction.
> > > > IIUC, you need to define
> > > > something like primary key or unique key in the result table of
> > > > TableAggregateFunction, and also
> > > > need a way to let user configure this through the API. My question
> is,
> > > will
> > > > that effect the logic of
> > > > TableAggregateFunction user wrote? Should the user know that there
> > will a

[VOTE] Migrate to sponsored Travis account

[Chesnay Schepler]

I've raised a JIRA 
with INFRA to inquire 
whether it would be possible to switch to a different Travis account, 
and if so what steps would need to be taken.
We need a proper confirmation from INFRA since we are not in full 
control of the flink repository (for example, we cannot access the 
settings page).


If this is indeed possible, Ververica is willing sponsor a Travis 
account for the Flink project.

This would provide us with more than enough resources than we need.

Since this makes the project more reliant on resources provided by 
external companies I would like to vote on this.


Please vote on this proposal, as follows:
[ ] +1, Approve the migration to a Ververica-sponsored Travis account, 
provided that INFRA approves
[ ] -1, Do not approach the migration to a Ververica-sponsored Travis 
account


The vote will be open for at least 24h, and until we have confirmation 
from INFRA. The voting period may be shorter than the usual 3 days since 
our current is effectively not working.


On 04/07/2019 06:51, Bowen Li wrote:
Re: > Are they using their own Travis CI pool, or did the switch to an 
entirely different CI service?


I reached out to Wes and Krisztián from Apache Arrow PMC. They are 
currently moving away from ASF's Travis to their own in-house metal 
machines at [1] with custom CI application at [2]. They've seen 
significant improvement w.r.t both much higher performance and 
basically no resource waiting time, "night-and-day" difference quoting 
Wes.


Re: > If we can just switch to our own Travis pool, just for our 
project, then this might be something we can do fairly quickly?


I believe so, according to [3] and [4]


[1] https://ci.ursalabs.org/ 
[2] https://github.com/ursa-labs/ursabot
[3] 
https://docs.travis-ci.com/user/migrate/open-source-repository-migration

[4] https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com



On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > wrote:


Are they using their own Travis CI pool, or did the switch to an
entirely different CI service?

If we can just switch to our own Travis pool, just for our
project, then
this might be something we can do fairly quickly?

On 03/07/2019 05:55, Bowen Li wrote:
> I responded in the INFRA ticket [1] that I believe they are
using a wrong
> metric against Flink and the total build time is a completely
different
> thing than guaranteed build capacity.
>
> My response:
>
> "As mentioned above, since I started to pay attention to Flink's
build
> queue a few tens of days ago, I'm in Seattle and I saw no build
was kicking
> off in PST daytime in weekdays for Flink. Our teammates in China
and Europe
> have also reported similar observations. So we need to evaluate
how the
> large total build time came from - if 1) your number and 2) our
> observations from three locations that cover pretty much a full
day, are
> all true, I **guess** one reason can be that - highly likely the
extra
> build time came from weekends when other Apache projects may be
idle and
> Flink just drains hard its congested queue.
>
> Please be aware of that we're not complaining about the lack of
resources
> in general, I'm complaining about the lack of **stable, dedicated**
> resources. An example for the latter one is, currently even if
no build is
> in Flink's queue and I submit a request to be the queue head in PST
> morning, my build won't even start in 6-8+h. That is an absurd
amount of
> waiting time.
>
> That's saying, if ASF INFRA decides to adopt a quota system and
grants
> Flink five DEDICATED servers that runs all the time only for
Flink, that'll
> be PERFECT and can totally solve our problem now.
>
> Please be aware of that we're not complaining about the lack of
resources
> in general, I'm complaining about the lack of **stable, dedicated**
> resources. An example for the latter one is, currently even if
no build is
> in Flink's queue and I submit a request to be the queue head in PST
> morning, my build won't even start in 6-8+h. That is an absurd
amount of
> waiting time.
>
>
> That's saying, if ASF INFRA decides to adopt a quota system and
grants
> Flink five DEDICATED servers that runs all the time only for
Flink, that'll
> be PERFECT and can totally solve our problem now.
>
> I feel what's missing in the ASF INFRA's Travis resource pool is
some level
> of build capacity SLAs and certainty"
>
>
> Again, I believe there are differences in nature of these two
problems,
> long build time v.s. lack of dedicated build resource. That's
saying,
> shortening build time may relieve the situation, and may not.
I'm sightly
 

Re: [DISCUSS] A more restrictive JIRA workflow

[Chesnay Schepler]

@Robert what's the state here?

On 24/06/2019 16:16, Robert Metzger wrote:

Hey all,

I would like to drive this discussion to an end soon.
I've just merged the updated contribution guide to the Flink website:
https://flink.apache.org/contributing/contribute-code.html

I will now ask Apache IINFRA to change the permissions in our Jira.

Here's the updated TODO list:

1. I update the contribution guide DONE
2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
unassigned JIRAs IN PROGRESS
3. We ask Infra to change the permissions of our JIRA so that: IN PROGRESS
   a) only committers can assign users to tickets
   b) contributors can't assign users to tickets
   c) Every registered JIRA user is an assignable user in FLINK





On Fri, May 24, 2019 at 9:18 AM Robert Metzger  wrote:


Hey,

I started working on step 1 and proposed some changes to the Flink
website: https://github.com/apache/flink-web/pull/217



On Tue, Apr 30, 2019 at 4:08 PM Robert Metzger 
wrote:


Hi Fabian,
You are right, I made a mistake. I don't think it makes sense to
introduce a new permission class. This will make the life of JIRA admins
unnecessarily complicated.
I updated the task list:

1. I update the contribution guide
2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
unassigned JIRAs
3. We ask Infra to change the permissions of our JIRA so that:
   a) only committers can assign users to tickets
   b) contributors can't assign users to tickets
   c) Every registered JIRA user is an assignable user in FLINK
4. We remove all existing contributors


On Tue, Apr 30, 2019 at 12:00 PM Fabian Hueske  wrote:


Hi Robert,

If I understood the decision correctly, we also need to ask Infra to make
everybody an assignable user, right?
Or do we want to add a new permission class "Assignable User" such that
everyone still needs to ask for the right Jira permissions?

Fabian


Am Di., 30. Apr. 2019 um 10:46 Uhr schrieb Timo Walther <
twal...@apache.org

:
Hi Robert,

thanks for taking care of this. +1 to your suggested steps.

Regards,
Timo


Am 30.04.19 um 10:42 schrieb Robert Metzger:

@Stephan: I agree. Auto-closing PRs is quite aggressive.
I will only do that for PRs without JIRA ID or "[hotfix]" in the

title.

We can always revisit this at a later stage.


I'm proposing the following steps:

1. I update the contribution guide
2. Update Flinkbot to close invalid PRs, and show warnings on PRs

with

unassigned JIRAs
3. We ask Infra to change the permissions of our JIRA so that:
a) only committers can assign users to tickets
b) contributors can't assign users to tickets
4. We remove all existing contributors




On Wed, Apr 24, 2019 at 11:17 AM vino yang 

wrote:

also +1 for option 2.

I think auto-close a PR sometimes a bit impertinency.
The reasons just like Stephan mentioned.

Stephan Ewen  于2019年4月24日周三 下午4:08写道：


About auto-closing PRs from unassigned issues, consider the

following

case

that has happened quite a bit.

- a user reports a small bug and immediately wants to provide a

fix

for

it
- it makes sense to not stall the user for a few days until a

committer

assigned the issue
- not auto-closing the PR would at least allow the user to

provide

their

patch.

On Wed, Apr 24, 2019 at 10:00 AM Stephan Ewen 

wrote:

+1 for option #2

Seems to me that this does not contradict option #1, it only

extends

this

a bit. I think there is a good case for that, to help frequent

contributors

on a way to committership.

@Konstantin: Trivial fixes (typos, docs, javadocs, ...) should

still

be

possible as "hotfixes".

On Mon, Apr 15, 2019 at 3:14 PM Timo Walther 

wrote:

I think this really depends on the contribution.

Sometimes "triviality" means that people just want to fix a typo

in

some

docs. For this, a hotfix PR is sufficient and does not need a

JIRA

issue.

However, sometimes "triviality" is only trivial at first glance

but

introduces side effects. In any case, any contribution needs to

be

reviewed and merged by a committer so follow-up responses and

follow-up

work might always be required. But you are right, committers

need to

respond quicker in any case.

Timo


Am 15.04.19 um 14:54 schrieb Konstantin Knauf:

Hi everyone,

just my two cents:  as a non-committer I appreciate a

lightweight,

frictionless process for trivial changes or small fixes without

the

need to

approach a committer beforehand. If it takes 5 days, so that I

can

start

with a triviality, I might not bother in the first place. So, in

order

for

this not to backfire by making the community more exclusive, we

need

more

timely responses & follow ups by committers after the change to

the

workflow. Having said that, I am slightly leaning towards

Andrey's

interpretation of option 2.

Cheers,

Konstantin



On Mon, Apr 15, 2019 at 1:39 PM Andrey Zagrebin <

and...@ververica.com

wrote:


@Robert thanks for pointing out and sorry for confusion. The

correct

text:

+1 for op

Re: [VOTE] Migrate to sponsored Travis account

[Stephan Ewen]

+1 to move to a private Travis account.

I can confirm that Ververica will sponsor a Travis CI plan that is
equivalent or a bit higher than the previous ASF quota (10 concurrent build
queues)

Best,
Stephan

On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler  wrote:

> I've raised a JIRA
> with INFRA to inquire
> whether it would be possible to switch to a different Travis account,
> and if so what steps would need to be taken.
> We need a proper confirmation from INFRA since we are not in full
> control of the flink repository (for example, we cannot access the
> settings page).
>
> If this is indeed possible, Ververica is willing sponsor a Travis
> account for the Flink project.
> This would provide us with more than enough resources than we need.
>
> Since this makes the project more reliant on resources provided by
> external companies I would like to vote on this.
>
> Please vote on this proposal, as follows:
> [ ] +1, Approve the migration to a Ververica-sponsored Travis account,
> provided that INFRA approves
> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
> account
>
> The vote will be open for at least 24h, and until we have confirmation
> from INFRA. The voting period may be shorter than the usual 3 days since
> our current is effectively not working.
>
> On 04/07/2019 06:51, Bowen Li wrote:
> > Re: > Are they using their own Travis CI pool, or did the switch to an
> > entirely different CI service?
> >
> > I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> > currently moving away from ASF's Travis to their own in-house metal
> > machines at [1] with custom CI application at [2]. They've seen
> > significant improvement w.r.t both much higher performance and
> > basically no resource waiting time, "night-and-day" difference quoting
> > Wes.
> >
> > Re: > If we can just switch to our own Travis pool, just for our
> > project, then this might be something we can do fairly quickly?
> >
> > I believe so, according to [3] and [4]
> >
> >
> > [1] https://ci.ursalabs.org/ 
> > [2] https://github.com/ursa-labs/ursabot
> > [3]
> > https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> > [4] https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> >
> >
> >
> > On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler  > > wrote:
> >
> > Are they using their own Travis CI pool, or did the switch to an
> > entirely different CI service?
> >
> > If we can just switch to our own Travis pool, just for our
> > project, then
> > this might be something we can do fairly quickly?
> >
> > On 03/07/2019 05:55, Bowen Li wrote:
> > > I responded in the INFRA ticket [1] that I believe they are
> > using a wrong
> > > metric against Flink and the total build time is a completely
> > different
> > > thing than guaranteed build capacity.
> > >
> > > My response:
> > >
> > > "As mentioned above, since I started to pay attention to Flink's
> > build
> > > queue a few tens of days ago, I'm in Seattle and I saw no build
> > was kicking
> > > off in PST daytime in weekdays for Flink. Our teammates in China
> > and Europe
> > > have also reported similar observations. So we need to evaluate
> > how the
> > > large total build time came from - if 1) your number and 2) our
> > > observations from three locations that cover pretty much a full
> > day, are
> > > all true, I **guess** one reason can be that - highly likely the
> > extra
> > > build time came from weekends when other Apache projects may be
> > idle and
> > > Flink just drains hard its congested queue.
> > >
> > > Please be aware of that we're not complaining about the lack of
> > resources
> > > in general, I'm complaining about the lack of **stable, dedicated**
> > > resources. An example for the latter one is, currently even if
> > no build is
> > > in Flink's queue and I submit a request to be the queue head in PST
> > > morning, my build won't even start in 6-8+h. That is an absurd
> > amount of
> > > waiting time.
> > >
> > > That's saying, if ASF INFRA decides to adopt a quota system and
> > grants
> > > Flink five DEDICATED servers that runs all the time only for
> > Flink, that'll
> > > be PERFECT and can totally solve our problem now.
> > >
> > > Please be aware of that we're not complaining about the lack of
> > resources
> > > in general, I'm complaining about the lack of **stable, dedicated**
> > > resources. An example for the latter one is, currently even if
> > no build is
> > > in Flink's queue and I submit a request to be the queue head in PST
> > > morning, my build won't even start in 6-8+h. That is an absurd
> > amount of
> > > waiting time.
> > >
> > 

Re: [VOTE] Migrate to sponsored Travis account

[Aljoscha Krettek]

+1

Aljoscha

> On 4. Jul 2019, at 11:09, Stephan Ewen  wrote:
> 
> +1 to move to a private Travis account.
> 
> I can confirm that Ververica will sponsor a Travis CI plan that is
> equivalent or a bit higher than the previous ASF quota (10 concurrent build
> queues)
> 
> Best,
> Stephan
> 
> On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler  wrote:
> 
>> I've raised a JIRA
>> with INFRA to inquire
>> whether it would be possible to switch to a different Travis account,
>> and if so what steps would need to be taken.
>> We need a proper confirmation from INFRA since we are not in full
>> control of the flink repository (for example, we cannot access the
>> settings page).
>> 
>> If this is indeed possible, Ververica is willing sponsor a Travis
>> account for the Flink project.
>> This would provide us with more than enough resources than we need.
>> 
>> Since this makes the project more reliant on resources provided by
>> external companies I would like to vote on this.
>> 
>> Please vote on this proposal, as follows:
>> [ ] +1, Approve the migration to a Ververica-sponsored Travis account,
>> provided that INFRA approves
>> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
>> account
>> 
>> The vote will be open for at least 24h, and until we have confirmation
>> from INFRA. The voting period may be shorter than the usual 3 days since
>> our current is effectively not working.
>> 
>> On 04/07/2019 06:51, Bowen Li wrote:
>>> Re: > Are they using their own Travis CI pool, or did the switch to an
>>> entirely different CI service?
>>> 
>>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
>>> currently moving away from ASF's Travis to their own in-house metal
>>> machines at [1] with custom CI application at [2]. They've seen
>>> significant improvement w.r.t both much higher performance and
>>> basically no resource waiting time, "night-and-day" difference quoting
>>> Wes.
>>> 
>>> Re: > If we can just switch to our own Travis pool, just for our
>>> project, then this might be something we can do fairly quickly?
>>> 
>>> I believe so, according to [3] and [4]
>>> 
>>> 
>>> [1] https://ci.ursalabs.org/ 
>>> [2] https://github.com/ursa-labs/ursabot
>>> [3]
>>> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
>>> [4] https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
>>> 
>>> 
>>> 
>>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler >> > wrote:
>>> 
>>>Are they using their own Travis CI pool, or did the switch to an
>>>entirely different CI service?
>>> 
>>>If we can just switch to our own Travis pool, just for our
>>>project, then
>>>this might be something we can do fairly quickly?
>>> 
>>>On 03/07/2019 05:55, Bowen Li wrote:
 I responded in the INFRA ticket [1] that I believe they are
>>>using a wrong
 metric against Flink and the total build time is a completely
>>>different
 thing than guaranteed build capacity.
 
 My response:
 
 "As mentioned above, since I started to pay attention to Flink's
>>>build
 queue a few tens of days ago, I'm in Seattle and I saw no build
>>>was kicking
 off in PST daytime in weekdays for Flink. Our teammates in China
>>>and Europe
 have also reported similar observations. So we need to evaluate
>>>how the
 large total build time came from - if 1) your number and 2) our
 observations from three locations that cover pretty much a full
>>>day, are
 all true, I **guess** one reason can be that - highly likely the
>>>extra
 build time came from weekends when other Apache projects may be
>>>idle and
 Flink just drains hard its congested queue.
 
 Please be aware of that we're not complaining about the lack of
>>>resources
 in general, I'm complaining about the lack of **stable, dedicated**
 resources. An example for the latter one is, currently even if
>>>no build is
 in Flink's queue and I submit a request to be the queue head in PST
 morning, my build won't even start in 6-8+h. That is an absurd
>>>amount of
 waiting time.
 
 That's saying, if ASF INFRA decides to adopt a quota system and
>>>grants
 Flink five DEDICATED servers that runs all the time only for
>>>Flink, that'll
 be PERFECT and can totally solve our problem now.
 
 Please be aware of that we're not complaining about the lack of
>>>resources
 in general, I'm complaining about the lack of **stable, dedicated**
 resources. An example for the latter one is, currently even if
>>>no build is
 in Flink's queue and I submit a request to be the queue head in PST
 morning, my build won't even start in 6-8+h. That is an absurd
>>>amount of
 waiting time.
 
 
 That's saying, if ASF INFRA decides to adopt a quota s

[jira] [Created] (FLINK-13098) Add a new type UNDEFINED of shuffle mode

[Biao Liu (JIRA)]

Biao Liu created FLINK-13098:


 Summary: Add a new type UNDEFINED of shuffle mode
 Key: FLINK-13098
 URL: https://issues.apache.org/jira/browse/FLINK-13098
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Biao Liu
Assignee: Biao Liu
 Fix For: 1.9.0


The {{UNDEFINED}} type is the default value of shuffle mode. If there is no 
specific {{PartitionTransformation}}, the shuffle mode would be {{UNDEFINED}}.
 This new shuffle type leaves some space for optimization later. The 
optimization might be based on resources or some global settings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

When publish the flink 1.9 Doc

[LakeShen]

Hi community，I want to know when publish the flink 1.9 document? Thanks.

[jira] [Created] (FLINK-13099) Add a new type UNDEFINED of shuffle mode

[Biao Liu (JIRA)]

Biao Liu created FLINK-13099:


 Summary: Add a new type UNDEFINED of shuffle mode
 Key: FLINK-13099
 URL: https://issues.apache.org/jira/browse/FLINK-13099
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Biao Liu
Assignee: Biao Liu
 Fix For: 1.9.0


The {{UNDEFINED}} type is the default value of shuffle mode. If there is no 
specific {{PartitionTransformation}}, the shuffle mode would be {{UNDEFINED}}.
 This new shuffle type leaves some space for optimization later. The 
optimization might be based on resources or some global settings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: When publish the flink 1.9 Doc

[Tzu-Li (Gordon) Tai]

Hi,

There is already document available for 1.9-SNAPSHOT [1].
The documentation for the official stable release will not be finalized
until 1.9 is actually released. Feature freeze for 1.9.0 is happening this
week, so ideally this should be available near end of July.

Cheers,
Gordon

[1] https://ci.apache.org/projects/flink/flink-docs-master/

On Thu, Jul 4, 2019 at 5:49 PM LakeShen  wrote:

> Hi community，I want to know when publish the flink 1.9 document? Thanks.
>

Re: [VOTE] Migrate to sponsored Travis account

[Kurt Young]

+1 and great thanks Chesnay for pushing this.

Best,
Kurt


On Thu, Jul 4, 2019 at 5:44 PM Aljoscha Krettek  wrote:

> +1
>
> Aljoscha
>
> > On 4. Jul 2019, at 11:09, Stephan Ewen  wrote:
> >
> > +1 to move to a private Travis account.
> >
> > I can confirm that Ververica will sponsor a Travis CI plan that is
> > equivalent or a bit higher than the previous ASF quota (10 concurrent
> build
> > queues)
> >
> > Best,
> > Stephan
> >
> > On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler 
> wrote:
> >
> >> I've raised a JIRA
> >> with INFRA to
> inquire
> >> whether it would be possible to switch to a different Travis account,
> >> and if so what steps would need to be taken.
> >> We need a proper confirmation from INFRA since we are not in full
> >> control of the flink repository (for example, we cannot access the
> >> settings page).
> >>
> >> If this is indeed possible, Ververica is willing sponsor a Travis
> >> account for the Flink project.
> >> This would provide us with more than enough resources than we need.
> >>
> >> Since this makes the project more reliant on resources provided by
> >> external companies I would like to vote on this.
> >>
> >> Please vote on this proposal, as follows:
> >> [ ] +1, Approve the migration to a Ververica-sponsored Travis account,
> >> provided that INFRA approves
> >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
> >> account
> >>
> >> The vote will be open for at least 24h, and until we have confirmation
> >> from INFRA. The voting period may be shorter than the usual 3 days since
> >> our current is effectively not working.
> >>
> >> On 04/07/2019 06:51, Bowen Li wrote:
> >>> Re: > Are they using their own Travis CI pool, or did the switch to an
> >>> entirely different CI service?
> >>>
> >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> >>> currently moving away from ASF's Travis to their own in-house metal
> >>> machines at [1] with custom CI application at [2]. They've seen
> >>> significant improvement w.r.t both much higher performance and
> >>> basically no resource waiting time, "night-and-day" difference quoting
> >>> Wes.
> >>>
> >>> Re: > If we can just switch to our own Travis pool, just for our
> >>> project, then this might be something we can do fairly quickly?
> >>>
> >>> I believe so, according to [3] and [4]
> >>>
> >>>
> >>> [1] https://ci.ursalabs.org/ 
> >>> [2] https://github.com/ursa-labs/ursabot
> >>> [3]
> >>>
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> >>> [4]
> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> >>>
> >>>
> >>>
> >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler  >>> > wrote:
> >>>
> >>>Are they using their own Travis CI pool, or did the switch to an
> >>>entirely different CI service?
> >>>
> >>>If we can just switch to our own Travis pool, just for our
> >>>project, then
> >>>this might be something we can do fairly quickly?
> >>>
> >>>On 03/07/2019 05:55, Bowen Li wrote:
>  I responded in the INFRA ticket [1] that I believe they are
> >>>using a wrong
>  metric against Flink and the total build time is a completely
> >>>different
>  thing than guaranteed build capacity.
> 
>  My response:
> 
>  "As mentioned above, since I started to pay attention to Flink's
> >>>build
>  queue a few tens of days ago, I'm in Seattle and I saw no build
> >>>was kicking
>  off in PST daytime in weekdays for Flink. Our teammates in China
> >>>and Europe
>  have also reported similar observations. So we need to evaluate
> >>>how the
>  large total build time came from - if 1) your number and 2) our
>  observations from three locations that cover pretty much a full
> >>>day, are
>  all true, I **guess** one reason can be that - highly likely the
> >>>extra
>  build time came from weekends when other Apache projects may be
> >>>idle and
>  Flink just drains hard its congested queue.
> 
>  Please be aware of that we're not complaining about the lack of
> >>>resources
>  in general, I'm complaining about the lack of **stable, dedicated**
>  resources. An example for the latter one is, currently even if
> >>>no build is
>  in Flink's queue and I submit a request to be the queue head in PST
>  morning, my build won't even start in 6-8+h. That is an absurd
> >>>amount of
>  waiting time.
> 
>  That's saying, if ASF INFRA decides to adopt a quota system and
> >>>grants
>  Flink five DEDICATED servers that runs all the time only for
> >>>Flink, that'll
>  be PERFECT and can totally solve our problem now.
> 
>  Please be aware of that we're not complaining about the lack of
> >>>resources
>  in general, I'm complaining about the lack of **stable, dedi

Re: [VOTE] Migrate to sponsored Travis account

[Chesnay Schepler]

Small update with mostly bad news:

INFRA doesn't know whether it is possible, and referred my to Travis 
support.
They did point out that it could be problematic in regards to read/write 
permissions for the repository.


From my own findings /so far/ with a test repo/organization, it does 
not appear possible to configure the Travis account used for a specific 
repository.


So yeah, if we go down this route we may have to pimp the Flinkbot to 
trigger builds through the Travis REST API.


On 04/07/2019 10:46, Chesnay Schepler wrote:
I've raised a JIRA 
with INFRA to 
inquire whether it would be possible to switch to a different Travis 
account, and if so what steps would need to be taken.
We need a proper confirmation from INFRA since we are not in full 
control of the flink repository (for example, we cannot access the 
settings page).


If this is indeed possible, Ververica is willing sponsor a Travis 
account for the Flink project.

This would provide us with more than enough resources than we need.

Since this makes the project more reliant on resources provided by 
external companies I would like to vote on this.


Please vote on this proposal, as follows:
[ ] +1, Approve the migration to a Ververica-sponsored Travis account, 
provided that INFRA approves
[ ] -1, Do not approach the migration to a Ververica-sponsored Travis 
account


The vote will be open for at least 24h, and until we have confirmation 
from INFRA. The voting period may be shorter than the usual 3 days 
since our current is effectively not working.


On 04/07/2019 06:51, Bowen Li wrote:
Re: > Are they using their own Travis CI pool, or did the switch to 
an entirely different CI service?


I reached out to Wes and Krisztián from Apache Arrow PMC. They are 
currently moving away from ASF's Travis to their own in-house metal 
machines at [1] with custom CI application at [2]. They've seen 
significant improvement w.r.t both much higher performance and 
basically no resource waiting time, "night-and-day" difference 
quoting Wes.


Re: > If we can just switch to our own Travis pool, just for our 
project, then this might be something we can do fairly quickly?


I believe so, according to [3] and [4]


[1] https://ci.ursalabs.org/ 
[2] https://github.com/ursa-labs/ursabot
[3] 
https://docs.travis-ci.com/user/migrate/open-source-repository-migration

[4] https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com



On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > wrote:


    Are they using their own Travis CI pool, or did the switch to an
    entirely different CI service?

    If we can just switch to our own Travis pool, just for our
    project, then
    this might be something we can do fairly quickly?

    On 03/07/2019 05:55, Bowen Li wrote:
    > I responded in the INFRA ticket [1] that I believe they are
    using a wrong
    > metric against Flink and the total build time is a completely
    different
    > thing than guaranteed build capacity.
    >
    > My response:
    >
    > "As mentioned above, since I started to pay attention to Flink's
    build
    > queue a few tens of days ago, I'm in Seattle and I saw no build
    was kicking
    > off in PST daytime in weekdays for Flink. Our teammates in China
    and Europe
    > have also reported similar observations. So we need to evaluate
    how the
    > large total build time came from - if 1) your number and 2) our
    > observations from three locations that cover pretty much a full
    day, are
    > all true, I **guess** one reason can be that - highly likely the
    extra
    > build time came from weekends when other Apache projects may be
    idle and
    > Flink just drains hard its congested queue.
    >
    > Please be aware of that we're not complaining about the lack of
    resources
    > in general, I'm complaining about the lack of **stable, 
dedicated**

    > resources. An example for the latter one is, currently even if
    no build is
    > in Flink's queue and I submit a request to be the queue head in 
PST

    > morning, my build won't even start in 6-8+h. That is an absurd
    amount of
    > waiting time.
    >
    > That's saying, if ASF INFRA decides to adopt a quota system and
    grants
    > Flink five DEDICATED servers that runs all the time only for
    Flink, that'll
    > be PERFECT and can totally solve our problem now.
    >
    > Please be aware of that we're not complaining about the lack of
    resources
    > in general, I'm complaining about the lack of **stable, 
dedicated**

    > resources. An example for the latter one is, currently even if
    no build is
    > in Flink's queue and I submit a request to be the queue head in 
PST

    > morning, my build won't even start in 6-8+h. That is an absurd
    amount of
    > waiting time.
    >
    >
    > That's saying, if ASF INFRA decides to adopt a quota system a

[jira] [Created] (FLINK-13100) Fix the unexpected IOException during FileBufferReader#nextBuffer

[zhijiang (JIRA)]

zhijiang created FLINK-13100:


 Summary: Fix the unexpected IOException during 
FileBufferReader#nextBuffer
 Key: FLINK-13100
 URL: https://issues.apache.org/jira/browse/FLINK-13100
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: zhijiang


In the implementation of FileBufferReader#nextBuffer, we expect the next memory 
segment always available based on the assumption that the nextBuffer call could 
only happen when the previous buffer was recycled before. Otherwise it would 
throw an IOException in current implementation.

In fact, the above assumption is not making sense based on the credit-based and 
zero-copy features in network. The detail processes are as follows:
 * The netty thread finishes calling the channel.writeAndFlush() in 
PartitionRequestQueue and adds a listener to handle the ChannelFuture later. 
Before future done, the corresponding buffer is not recycled because of 
zero-copy improvement.

 * Before the previous future done, the netty thread could trigger next 
writeAndFlush via processing addCredit message, then 
FileBufferReader#nextBuffer would throw exception because of previous buffer 
not recycled.

We thought of several ways for solving this potential bug:
 * It does not trigger the next writeAndFlush before the previous future done. 
To do so it has to maintain the future state and check it in relevant actions. 
I wonder it might bring performance regression in network throughput and bring 
extra state management.

 * Adjust the implementation of current FileBufferReader. We ever regarded the 
blocking partition view as always available based on the next buffer read 
ahead, so it would be always added into available queue in 
PartitionRequestQueue. Actually this next buffer ahead only simplifies the 
process of BoundedBlockingSubpartitionReader#notifyDataAvailable. The view 
availability could be judged based on available buffers in FileBufferReader 
instead of next buffer ahead. When the buffer is recycled into FileBufferReader 
after writeAndFlush done, it could call notifyDataAvailable to add this view 
into available queue in PartitionRequestQueue.

I prefer the second way because it would not bring any bad impacts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS]Support Upsert mode for Streaming Non-window FlatAggregate

[Kurt Young]

Hi Jincheng,

Thanks for the clarification. I think you just pointed out my concern by
yourself:

> When a user uses a User-defined table aggregate function (UDTAGG), he
must understand the behavior of the UDTAGG, including the return type and
the characteristics of the returned data. such as the key fields.

This indicates that the UDTAGG is somehow be classified to different types,
one will no key, one with key information. So the developer of the UDTAGG
should choose which type of this function should be. In this case,
my question would be, why don't we have explicit information about keys
such as we split UDTAGG to keyed UDTAGG and non-keyed UDTAGG. So the user
and the framework will have a better understanding of
this UDTAGG. `withKeys` solution is letting user to choose the key and it
seems it will only work correctly only if the user choose the *right* key
this UDTAGG has.

Let me know if this makes sense to you.

Best,
Kurt


On Thu, Jul 4, 2019 at 4:32 PM jincheng sun 
wrote:

> Hi All,
>
> @Kurt Young  one user-defined table aggregate function
> can be used in both with(out) keys case, and we do not introduce any other
> aggregations. just like the explanation from @Hequn.
>
> @Hequn Cheng  thanks for your explanation!
>
> One thing should be mentioned here:
>
> When a user uses a User-defined table aggregate function (UDTAGG), he must
> understand the behavior of the UDTAGG, including the return type and the
> characteristics of the returned data. such as the key fields.  So although
> using `withKeys` approach it is not rigorous enough(we do not need) but
> intuitive enough, considering that if `flatAggregate` is followed by an
> `upsertSink`, then the user must know the keys of the output of UDTAGG
> clearly, otherwise the keys of `upsertSink` cannot be defined. So I still
> prefer the `withKeys` solution by now.
>
> Looking forward to any feedback from all of you!
>
> Best,
> Jincheng
>
>
>
> Hequn Cheng  于2019年7月1日周一 下午5:35写道：
>
>> Hi Kurt,
>>
>> Thanks for your questions. Here are my thoughts.
>>
>> > if I want to write such kind function, should I make sure that this
>> function is used with some keys?
>> The key information may not be used. We can also use RetractSink to emit
>> the table directly.
>>
>> >  If I need a use case to calculate topn without key, should I write
>> another function or I can reuse previous one.
>> For the TopN example, you can reuse the previous function if you don't
>> care
>> about the key information.
>>
>> So, the key information is only an indicator(or a description), not an
>> operator, as Jincheng mentioned above.
>> We do not need to change the function logic and it will not add any other
>> aggregations.
>>
>> BTW, we have three approaches in the document. Approach 1 defines keys on
>> API level as we think it's more common to define keys on Table.
>> While approach 3 defines keys in the TableAggregateFunction which is more
>> precise but it is not very clear for Table users. So, we should take all
>> these into consideration, and make the decision in this discussion thread.
>>
>> You can take a look at the document and welcome any suggestions or other
>> better solutions.
>>
>> Best, Hequn
>>
>>
>> On Mon, Jul 1, 2019 at 12:13 PM Kurt Young  wrote:
>>
>> > Hi Jincheng,
>> >
>> > Thanks for the clarification. Take 'TopN' for example, if I want to
>> write
>> > such kind function,
>> > should I make sure that this function is used with some keys? If I need
>> a
>> > use case to calculate
>> > topn without key, should I write another function or I can reuse
>> previous
>> > one.
>> >
>> > I'm not sure about the idea of this does not involve semantic changes.
>> To
>> > me, it sounds like
>> > we are doing another nested aggregation inside the table
>> > which TableAggregateFunction emits.
>> >
>> > Maybe I'm not familiar with this function enough, hope you can help me
>> to
>> > understand.
>> >
>> > Best,
>> > Kurt
>> >
>> >
>> > On Mon, Jul 1, 2019 at 11:59 AM jincheng sun 
>> > wrote:
>> >
>> > > Hi Kurt,
>> > >
>> > > Thanks for your questions, I am glad to share my thoughts here:
>> > >
>> > > My question is, will that effect the logic ofTableAggregateFunction
>> user
>> > > > wrote? Should the user know that there will a key and make some
>> changes
>> > > to
>> > > > this function?
>> > >
>> > >
>> > > No, the keys information depends on the implementation of the
>> > > TableAggregateFunction.
>> > > For example, for a `topN` user defined TableAggregateFunction, we can
>> > only
>> > > use the `keys` if the `topN` contains `rankid` in the output. You can
>> > > treat the
>> > > `keys` like an indicator.
>> > >
>> > > If not, how will framework deal with the output of user's
>> > > > TableAggregateFunction.  if user output multiple rows with the same
>> > key,
>> > > > should be latter one replace previous ones?
>> > >
>> > >
>> > > If a TableAggregateFunction outputs multiple rows with the same key,
>> the
>> > > latter one should replace the previ

Re:Re: [VOTE] Migrate to sponsored Travis account

[Haibo Sun]

+1. Thank Chesnay for pushing this forward.

Best,
Haibo


At 2019-07-04 17:58:28, "Kurt Young"  wrote:
>+1 and great thanks Chesnay for pushing this.
>
>Best,
>Kurt
>
>
>On Thu, Jul 4, 2019 at 5:44 PM Aljoscha Krettek  wrote:
>
>> +1
>>
>> Aljoscha
>>
>> > On 4. Jul 2019, at 11:09, Stephan Ewen  wrote:
>> >
>> > +1 to move to a private Travis account.
>> >
>> > I can confirm that Ververica will sponsor a Travis CI plan that is
>> > equivalent or a bit higher than the previous ASF quota (10 concurrent
>> build
>> > queues)
>> >
>> > Best,
>> > Stephan
>> >
>> > On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler 
>> wrote:
>> >
>> >> I've raised a JIRA
>> >> with INFRA to
>> inquire
>> >> whether it would be possible to switch to a different Travis account,
>> >> and if so what steps would need to be taken.
>> >> We need a proper confirmation from INFRA since we are not in full
>> >> control of the flink repository (for example, we cannot access the
>> >> settings page).
>> >>
>> >> If this is indeed possible, Ververica is willing sponsor a Travis
>> >> account for the Flink project.
>> >> This would provide us with more than enough resources than we need.
>> >>
>> >> Since this makes the project more reliant on resources provided by
>> >> external companies I would like to vote on this.
>> >>
>> >> Please vote on this proposal, as follows:
>> >> [ ] +1, Approve the migration to a Ververica-sponsored Travis account,
>> >> provided that INFRA approves
>> >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
>> >> account
>> >>
>> >> The vote will be open for at least 24h, and until we have confirmation
>> >> from INFRA. The voting period may be shorter than the usual 3 days since
>> >> our current is effectively not working.
>> >>
>> >> On 04/07/2019 06:51, Bowen Li wrote:
>> >>> Re: > Are they using their own Travis CI pool, or did the switch to an
>> >>> entirely different CI service?
>> >>>
>> >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
>> >>> currently moving away from ASF's Travis to their own in-house metal
>> >>> machines at [1] with custom CI application at [2]. They've seen
>> >>> significant improvement w.r.t both much higher performance and
>> >>> basically no resource waiting time, "night-and-day" difference quoting
>> >>> Wes.
>> >>>
>> >>> Re: > If we can just switch to our own Travis pool, just for our
>> >>> project, then this might be something we can do fairly quickly?
>> >>>
>> >>> I believe so, according to [3] and [4]
>> >>>
>> >>>
>> >>> [1] https://ci.ursalabs.org/ 
>> >>> [2] https://github.com/ursa-labs/ursabot
>> >>> [3]
>> >>>
>> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
>> >>> [4]
>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > >>> > wrote:
>> >>>
>> >>>Are they using their own Travis CI pool, or did the switch to an
>> >>>entirely different CI service?
>> >>>
>> >>>If we can just switch to our own Travis pool, just for our
>> >>>project, then
>> >>>this might be something we can do fairly quickly?
>> >>>
>> >>>On 03/07/2019 05:55, Bowen Li wrote:
>>  I responded in the INFRA ticket [1] that I believe they are
>> >>>using a wrong
>>  metric against Flink and the total build time is a completely
>> >>>different
>>  thing than guaranteed build capacity.
>> 
>>  My response:
>> 
>>  "As mentioned above, since I started to pay attention to Flink's
>> >>>build
>>  queue a few tens of days ago, I'm in Seattle and I saw no build
>> >>>was kicking
>>  off in PST daytime in weekdays for Flink. Our teammates in China
>> >>>and Europe
>>  have also reported similar observations. So we need to evaluate
>> >>>how the
>>  large total build time came from - if 1) your number and 2) our
>>  observations from three locations that cover pretty much a full
>> >>>day, are
>>  all true, I **guess** one reason can be that - highly likely the
>> >>>extra
>>  build time came from weekends when other Apache projects may be
>> >>>idle and
>>  Flink just drains hard its congested queue.
>> 
>>  Please be aware of that we're not complaining about the lack of
>> >>>resources
>>  in general, I'm complaining about the lack of **stable, dedicated**
>>  resources. An example for the latter one is, currently even if
>> >>>no build is
>>  in Flink's queue and I submit a request to be the queue head in PST
>>  morning, my build won't even start in 6-8+h. That is an absurd
>> >>>amount of
>>  waiting time.
>> 
>>  That's saying, if ASF INFRA decides to adopt a quota system and
>> >>>grants
>>  Flink five DEDICATED servers that runs all the time only for
>> >>>Fl

Re: Re: [VOTE] Migrate to sponsored Travis account

[zhijiang]

+1 and thanks for Chesnay' work on this.

Best,
Zhijiang
 
--
From:Haibo Sun 
Send Time:2019年7月4日(星期四) 18:21
To:dev 
Cc:priv...@flink.apache.org 
Subject:Re:Re: [VOTE] Migrate to sponsored Travis account

+1. Thank Chesnay for pushing this forward.

Best,
Haibo


At 2019-07-04 17:58:28, "Kurt Young"  wrote:
>+1 and great thanks Chesnay for pushing this.
>
>Best,
>Kurt
>
>
>On Thu, Jul 4, 2019 at 5:44 PM Aljoscha Krettek  wrote:
>
>> +1
>>
>> Aljoscha
>>
>> > On 4. Jul 2019, at 11:09, Stephan Ewen  wrote:
>> >
>> > +1 to move to a private Travis account.
>> >
>> > I can confirm that Ververica will sponsor a Travis CI plan that is
>> > equivalent or a bit higher than the previous ASF quota (10 concurrent
>> build
>> > queues)
>> >
>> > Best,
>> > Stephan
>> >
>> > On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler 
>> wrote:
>> >
>> >> I've raised a JIRA
>> >> with INFRA to
>> inquire
>> >> whether it would be possible to switch to a different Travis account,
>> >> and if so what steps would need to be taken.
>> >> We need a proper confirmation from INFRA since we are not in full
>> >> control of the flink repository (for example, we cannot access the
>> >> settings page).
>> >>
>> >> If this is indeed possible, Ververica is willing sponsor a Travis
>> >> account for the Flink project.
>> >> This would provide us with more than enough resources than we need.
>> >>
>> >> Since this makes the project more reliant on resources provided by
>> >> external companies I would like to vote on this.
>> >>
>> >> Please vote on this proposal, as follows:
>> >> [ ] +1, Approve the migration to a Ververica-sponsored Travis account,
>> >> provided that INFRA approves
>> >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
>> >> account
>> >>
>> >> The vote will be open for at least 24h, and until we have confirmation
>> >> from INFRA. The voting period may be shorter than the usual 3 days since
>> >> our current is effectively not working.
>> >>
>> >> On 04/07/2019 06:51, Bowen Li wrote:
>> >>> Re: > Are they using their own Travis CI pool, or did the switch to an
>> >>> entirely different CI service?
>> >>>
>> >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
>> >>> currently moving away from ASF's Travis to their own in-house metal
>> >>> machines at [1] with custom CI application at [2]. They've seen
>> >>> significant improvement w.r.t both much higher performance and
>> >>> basically no resource waiting time, "night-and-day" difference quoting
>> >>> Wes.
>> >>>
>> >>> Re: > If we can just switch to our own Travis pool, just for our
>> >>> project, then this might be something we can do fairly quickly?
>> >>>
>> >>> I believe so, according to [3] and [4]
>> >>>
>> >>>
>> >>> [1] https://ci.ursalabs.org/ 
>> >>> [2] https://github.com/ursa-labs/ursabot
>> >>> [3]
>> >>>
>> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
>> >>> [4]
>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > >>> > wrote:
>> >>>
>> >>>Are they using their own Travis CI pool, or did the switch to an
>> >>>entirely different CI service?
>> >>>
>> >>>If we can just switch to our own Travis pool, just for our
>> >>>project, then
>> >>>this might be something we can do fairly quickly?
>> >>>
>> >>>On 03/07/2019 05:55, Bowen Li wrote:
>>  I responded in the INFRA ticket [1] that I believe they are
>> >>>using a wrong
>>  metric against Flink and the total build time is a completely
>> >>>different
>>  thing than guaranteed build capacity.
>> 
>>  My response:
>> 
>>  "As mentioned above, since I started to pay attention to Flink's
>> >>>build
>>  queue a few tens of days ago, I'm in Seattle and I saw no build
>> >>>was kicking
>>  off in PST daytime in weekdays for Flink. Our teammates in China
>> >>>and Europe
>>  have also reported similar observations. So we need to evaluate
>> >>>how the
>>  large total build time came from - if 1) your number and 2) our
>>  observations from three locations that cover pretty much a full
>> >>>day, are
>>  all true, I **guess** one reason can be that - highly likely the
>> >>>extra
>>  build time came from weekends when other Apache projects may be
>> >>>idle and
>>  Flink just drains hard its congested queue.
>> 
>>  Please be aware of that we're not complaining about the lack of
>> >>>resources
>>  in general, I'm complaining about the lack of **stable, dedicated**
>>  resources. An example for the latter one is, currently even if
>> >>>no build is
>>  in Flink's queue and I submit a request to be the queue head in PST
>>  morning, my bu

Re: [DISCUSS] Flink framework and user log separation

[Stephan Ewen]

Is that something that can just be done by the right logging framework and
configuration?

Like having a log framework with two targets, one filtered on
"org.apache.flink" and the other one filtered on "my.company.project" or so?

On Fri, Mar 1, 2019 at 3:44 AM vino yang  wrote:

> Hi Jamie Grier,
>
> Thank you for your reply, let me add some explanations to this design.
>
> First of all, as stated in "Goal", it is mainly for the "Standalone"
> cluster model, although we have implemented it for Flink on YARN, this does
> not mean that we can't turn off this feature by means of options. It should
> be noted that the separation is basically based on the "log configuration
> file", it is very scalable and even allows users to define the log pattern
> of the configuration file (of course this is an extension feature, not
> mentioned in the design documentation). In fact, "multiple files are a
> special case of a single file", we can provide an option to keep it still
> the default behavior, it should be the scene you expect in the container.
>
> According to Flink's official 2016 adjustment report [1], users using the
> standalone mode are quite close to the yarn mode (unfortunately there is no
> data support in 2017). Although we mainly use Flink on Yarn now, we have
> used standalone in depth (close to the daily processing volume of 20
> trillion messages). In this scenario, the user logs generated by different
> job's tasks are mixed together, and it is very difficult to locate the
> issue. Moreover, as we configure the log file scrolling policy, we have to
> log in to the server to view it. Therefore, we expect that for the same
> task manager, the user logs generated by the tasks from the same job can be
> distinguished.
>
> In addition, I have tried MDC technology, but it can not achieve the goal.
> The underlying Flink is log4j 1.x and logback. We need to be compatible
> with both frameworks at the same time, and we don't allow large-scale
> changes to the active code, and no sense to the user.
>
> Some other points:
>
> 1) Many of our users have experience using Storm and Spark, and they are
> more accustomed to that style in standalone mode;
> 2) We split the user log by Job, which will help to implement the "business
> log aggregation" feature based on the Job.
>
> Best,
> Vino
>
> [1]: https://www.ververica.com/blog/flink-user-survey-2016-part-1
>
> Jamie Grier  于2019年3月1日周五 上午7:32写道：
>
> > I think maybe if I understood this correctly this design is going in the
> > wrong direction.  The problem with Flink logging, when you are running
> > multiple jobs in the same TMs, is not just about separating out the
> > business level logging into separate files.  The Flink framework itself
> > logs many things where there is clearly a single job in context but that
> > all ends up in the same log file and with no clear separation amongst the
> > log lines.
> >
> > Also, I don't think shooting to have multiple log files is a very good
> idea
> > either.  It's common, especially on container-based deployments, that the
> > expectation is that a process (like Flink) logs everything to stdout and
> > the surrounding tooling takes care of routing that log data somewhere.  I
> > think we should stick with that model and expect that there will be a
> > single log stream coming out of each Flink process.
> >
> > Instead, I think it would be better to enhance Flink's logging capability
> > such that the appropriate context can be added to each log line with the
> > exact format controlled by the end user.  It might make sense to take a
> > look at MDC, for example, as a way to approach this.
> >
> >
> > On Thu, Feb 28, 2019 at 4:24 AM vino yang  wrote:
> >
> > > Dear devs,
> > >
> > > Currently, for log output, Flink does not explicitly distinguish
> between
> > > framework logs and user logs. In Task Manager, logs from the framework
> > are
> > > intermixed with the user's business logs. In some deployment models,
> such
> > > as Standalone or YARN session, there are different task instances of
> > > different jobs deployed in the same Task Manager. It makes the log
> event
> > > flow more confusing unless the users explicitly use tags to distinguish
> > > them and it makes locating problems more difficult and inefficient. For
> > > YARN job cluster deployment model, this problem will not be very
> serious,
> > > but we still need to artificially distinguish between the framework and
> > the
> > > business log. Overall, we found that Flink's existing log model has the
> > > following problems:
> > >
> > >
> > >-
> > >
> > >Framework log and business log are mixed in the same log file. There
> > >is no way to make a clear distinction, which is not conducive to
> > problem
> > >location and analysis;
> > >-
> > >
> > >Not conducive to the independent collection of business logs;
> > >
> > >
> > > Therefore, we propose a mechanism to separate the framework and
> business
> > > log. It can split 

Re: [DISCUSS] Flink framework and user log separation

[Chesnay Schepler]

From what I understand this isn't about logging Flink/user messages to 
different files, but log everything relevant to a specific job to a 
separate file (including what is being logged in runtime classes, i.e. 
Tasks, Operators etc.)


On 04/07/2019 12:37, Stephan Ewen wrote:

Is that something that can just be done by the right logging framework and
configuration?

Like having a log framework with two targets, one filtered on
"org.apache.flink" and the other one filtered on "my.company.project" or so?

On Fri, Mar 1, 2019 at 3:44 AM vino yang  wrote:


Hi Jamie Grier,

Thank you for your reply, let me add some explanations to this design.

First of all, as stated in "Goal", it is mainly for the "Standalone"
cluster model, although we have implemented it for Flink on YARN, this does
not mean that we can't turn off this feature by means of options. It should
be noted that the separation is basically based on the "log configuration
file", it is very scalable and even allows users to define the log pattern
of the configuration file (of course this is an extension feature, not
mentioned in the design documentation). In fact, "multiple files are a
special case of a single file", we can provide an option to keep it still
the default behavior, it should be the scene you expect in the container.

According to Flink's official 2016 adjustment report [1], users using the
standalone mode are quite close to the yarn mode (unfortunately there is no
data support in 2017). Although we mainly use Flink on Yarn now, we have
used standalone in depth (close to the daily processing volume of 20
trillion messages). In this scenario, the user logs generated by different
job's tasks are mixed together, and it is very difficult to locate the
issue. Moreover, as we configure the log file scrolling policy, we have to
log in to the server to view it. Therefore, we expect that for the same
task manager, the user logs generated by the tasks from the same job can be
distinguished.

In addition, I have tried MDC technology, but it can not achieve the goal.
The underlying Flink is log4j 1.x and logback. We need to be compatible
with both frameworks at the same time, and we don't allow large-scale
changes to the active code, and no sense to the user.

Some other points:

1) Many of our users have experience using Storm and Spark, and they are
more accustomed to that style in standalone mode;
2) We split the user log by Job, which will help to implement the "business
log aggregation" feature based on the Job.

Best,
Vino

[1]: https://www.ververica.com/blog/flink-user-survey-2016-part-1

Jamie Grier  于2019年3月1日周五 上午7:32写道：


I think maybe if I understood this correctly this design is going in the
wrong direction.  The problem with Flink logging, when you are running
multiple jobs in the same TMs, is not just about separating out the
business level logging into separate files.  The Flink framework itself
logs many things where there is clearly a single job in context but that
all ends up in the same log file and with no clear separation amongst the
log lines.

Also, I don't think shooting to have multiple log files is a very good

idea

either.  It's common, especially on container-based deployments, that the
expectation is that a process (like Flink) logs everything to stdout and
the surrounding tooling takes care of routing that log data somewhere.  I
think we should stick with that model and expect that there will be a
single log stream coming out of each Flink process.

Instead, I think it would be better to enhance Flink's logging capability
such that the appropriate context can be added to each log line with the
exact format controlled by the end user.  It might make sense to take a
look at MDC, for example, as a way to approach this.


On Thu, Feb 28, 2019 at 4:24 AM vino yang  wrote:


Dear devs,

Currently, for log output, Flink does not explicitly distinguish

between

framework logs and user logs. In Task Manager, logs from the framework

are

intermixed with the user's business logs. In some deployment models,

such

as Standalone or YARN session, there are different task instances of
different jobs deployed in the same Task Manager. It makes the log

event

flow more confusing unless the users explicitly use tags to distinguish
them and it makes locating problems more difficult and inefficient. For
YARN job cluster deployment model, this problem will not be very

serious,

but we still need to artificially distinguish between the framework and

the

business log. Overall, we found that Flink's existing log model has the
following problems:


-

Framework log and business log are mixed in the same log file. There
is no way to make a clear distinction, which is not conducive to

problem

location and analysis;
-

Not conducive to the independent collection of business logs;


Therefore, we propose a mechanism to separate the framework and

business

log. It can split existing log files for Task Manager.

Re: [VOTE] Migrate to sponsored Travis account

[Dian Fu]

+1. Thanks Chesnay and Bowen for pushing this forward.

Regards,
Dian

> 在 2019年7月4日，下午6:28，zhijiang  写道：
> 
> +1 and thanks for Chesnay' work on this.
> 
> Best,
> Zhijiang
> 
> --
> From:Haibo Sun 
> Send Time:2019年7月4日(星期四) 18:21
> To:dev 
> Cc:priv...@flink.apache.org 
> Subject:Re:Re: [VOTE] Migrate to sponsored Travis account
> 
> +1. Thank Chesnay for pushing this forward.
> 
> Best,
> Haibo
> 
> 
> At 2019-07-04 17:58:28, "Kurt Young"  wrote:
>> +1 and great thanks Chesnay for pushing this.
>> 
>> Best,
>> Kurt
>> 
>> 
>> On Thu, Jul 4, 2019 at 5:44 PM Aljoscha Krettek  wrote:
>> 
>>> +1
>>> 
>>> Aljoscha
>>> 
 On 4. Jul 2019, at 11:09, Stephan Ewen  wrote:
 
 +1 to move to a private Travis account.
 
 I can confirm that Ververica will sponsor a Travis CI plan that is
 equivalent or a bit higher than the previous ASF quota (10 concurrent
>>> build
 queues)
 
 Best,
 Stephan
 
 On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler 
>>> wrote:
 
> I've raised a JIRA
> with INFRA to
>>> inquire
> whether it would be possible to switch to a different Travis account,
> and if so what steps would need to be taken.
> We need a proper confirmation from INFRA since we are not in full
> control of the flink repository (for example, we cannot access the
> settings page).
> 
> If this is indeed possible, Ververica is willing sponsor a Travis
> account for the Flink project.
> This would provide us with more than enough resources than we need.
> 
> Since this makes the project more reliant on resources provided by
> external companies I would like to vote on this.
> 
> Please vote on this proposal, as follows:
> [ ] +1, Approve the migration to a Ververica-sponsored Travis account,
> provided that INFRA approves
> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
> account
> 
> The vote will be open for at least 24h, and until we have confirmation
> from INFRA. The voting period may be shorter than the usual 3 days since
> our current is effectively not working.
> 
> On 04/07/2019 06:51, Bowen Li wrote:
>> Re: > Are they using their own Travis CI pool, or did the switch to an
>> entirely different CI service?
>> 
>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
>> currently moving away from ASF's Travis to their own in-house metal
>> machines at [1] with custom CI application at [2]. They've seen
>> significant improvement w.r.t both much higher performance and
>> basically no resource waiting time, "night-and-day" difference quoting
>> Wes.
>> 
>> Re: > If we can just switch to our own Travis pool, just for our
>> project, then this might be something we can do fairly quickly?
>> 
>> I believe so, according to [3] and [4]
>> 
>> 
>> [1] https://ci.ursalabs.org/ 
>> [2] https://github.com/ursa-labs/ursabot
>> [3]
>> 
>>> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
>> [4]
>>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
>> 
>> 
>> 
>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > > wrote:
>> 
>>   Are they using their own Travis CI pool, or did the switch to an
>>   entirely different CI service?
>> 
>>   If we can just switch to our own Travis pool, just for our
>>   project, then
>>   this might be something we can do fairly quickly?
>> 
>>   On 03/07/2019 05:55, Bowen Li wrote:
>>> I responded in the INFRA ticket [1] that I believe they are
>>   using a wrong
>>> metric against Flink and the total build time is a completely
>>   different
>>> thing than guaranteed build capacity.
>>> 
>>> My response:
>>> 
>>> "As mentioned above, since I started to pay attention to Flink's
>>   build
>>> queue a few tens of days ago, I'm in Seattle and I saw no build
>>   was kicking
>>> off in PST daytime in weekdays for Flink. Our teammates in China
>>   and Europe
>>> have also reported similar observations. So we need to evaluate
>>   how the
>>> large total build time came from - if 1) your number and 2) our
>>> observations from three locations that cover pretty much a full
>>   day, are
>>> all true, I **guess** one reason can be that - highly likely the
>>   extra
>>> build time came from weekends when other Apache projects may be
>>   idle and
>>> Flink just drains hard its congested queue.
>>> 
>>> Please be aware of that we're not complaining about the lack of
>>   resources
>>> in general, I'm complaining about the lack of **stable, dedicated**
>>> r

Re: When publish the flink 1.9 Doc

[LakeShen]

Thank you

Tzu-Li (Gordon) Tai  于2019年7月4日周四 下午5:54写道：

> Hi,
>
> There is already document available for 1.9-SNAPSHOT [1].
> The documentation for the official stable release will not be finalized
> until 1.9 is actually released. Feature freeze for 1.9.0 is happening this
> week, so ideally this should be available near end of July.
>
> Cheers,
> Gordon
>
> [1] https://ci.apache.org/projects/flink/flink-docs-master/
>
> On Thu, Jul 4, 2019 at 5:49 PM LakeShen  wrote:
>
> > Hi community，I want to know when publish the flink 1.9 document? Thanks.
> >
>

Flink 1.9 blink planner

[kaka chen]

Hi All:

We found TableEnvironment in Flink 1.9 blink planner didn't support
functions such as sqlUpdate() and  registerTableSink(), when I used Scala
api, it will throw error.

*Error:(61, 20) value sqlUpdate is not a member of
org.apache.flink.table.api.scala.StreamTableEnvironment.*

Is it normal? Could someone help, thanks.

Thanks,
Kaka Chen

Re: [DISCUSS] A more restrictive JIRA workflow

[Robert Metzger]

This is the Jira ticket I opened
https://issues.apache.org/jira/browse/INFRA-18644 a long time ago :)

On Thu, Jul 4, 2019 at 10:47 AM Chesnay Schepler  wrote:

> @Robert what's the state here?
>
> On 24/06/2019 16:16, Robert Metzger wrote:
> > Hey all,
> >
> > I would like to drive this discussion to an end soon.
> > I've just merged the updated contribution guide to the Flink website:
> > https://flink.apache.org/contributing/contribute-code.html
> >
> > I will now ask Apache IINFRA to change the permissions in our Jira.
> >
> > Here's the updated TODO list:
> >
> > 1. I update the contribution guide DONE
> > 2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
> > unassigned JIRAs IN PROGRESS
> > 3. We ask Infra to change the permissions of our JIRA so that: IN
> PROGRESS
> >a) only committers can assign users to tickets
> >b) contributors can't assign users to tickets
> >c) Every registered JIRA user is an assignable user in FLINK
> >
> >
> >
> >
> >
> > On Fri, May 24, 2019 at 9:18 AM Robert Metzger 
> wrote:
> >
> >> Hey,
> >>
> >> I started working on step 1 and proposed some changes to the Flink
> >> website: https://github.com/apache/flink-web/pull/217
> >>
> >>
> >>
> >> On Tue, Apr 30, 2019 at 4:08 PM Robert Metzger 
> >> wrote:
> >>
> >>> Hi Fabian,
> >>> You are right, I made a mistake. I don't think it makes sense to
> >>> introduce a new permission class. This will make the life of JIRA
> admins
> >>> unnecessarily complicated.
> >>> I updated the task list:
> >>>
> >>> 1. I update the contribution guide
> >>> 2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
> >>> unassigned JIRAs
> >>> 3. We ask Infra to change the permissions of our JIRA so that:
> >>>a) only committers can assign users to tickets
> >>>b) contributors can't assign users to tickets
> >>>c) Every registered JIRA user is an assignable user in FLINK
> >>> 4. We remove all existing contributors
> >>>
> >>>
> >>> On Tue, Apr 30, 2019 at 12:00 PM Fabian Hueske 
> wrote:
> >>>
>  Hi Robert,
> 
>  If I understood the decision correctly, we also need to ask Infra to
> make
>  everybody an assignable user, right?
>  Or do we want to add a new permission class "Assignable User" such
> that
>  everyone still needs to ask for the right Jira permissions?
> 
>  Fabian
> 
> 
>  Am Di., 30. Apr. 2019 um 10:46 Uhr schrieb Timo Walther <
>  twal...@apache.org
> > :
> > Hi Robert,
> >
> > thanks for taking care of this. +1 to your suggested steps.
> >
> > Regards,
> > Timo
> >
> >
> > Am 30.04.19 um 10:42 schrieb Robert Metzger:
> >> @Stephan: I agree. Auto-closing PRs is quite aggressive.
> >> I will only do that for PRs without JIRA ID or "[hotfix]" in the
>  title.
> >> We can always revisit this at a later stage.
> >>
> >>
> >> I'm proposing the following steps:
> >>
> >> 1. I update the contribution guide
> >> 2. Update Flinkbot to close invalid PRs, and show warnings on PRs
>  with
> >> unassigned JIRAs
> >> 3. We ask Infra to change the permissions of our JIRA so that:
> >> a) only committers can assign users to tickets
> >> b) contributors can't assign users to tickets
> >> 4. We remove all existing contributors
> >>
> >>
> >>
> >>
> >> On Wed, Apr 24, 2019 at 11:17 AM vino yang 
> > wrote:
> >>> also +1 for option 2.
> >>>
> >>> I think auto-close a PR sometimes a bit impertinency.
> >>> The reasons just like Stephan mentioned.
> >>>
> >>> Stephan Ewen  于2019年4月24日周三 下午4:08写道：
> >>>
>  About auto-closing PRs from unassigned issues, consider the
>  following
> >>> case
>  that has happened quite a bit.
> 
>  - a user reports a small bug and immediately wants to provide
> a
>  fix
> > for
>  it
>  - it makes sense to not stall the user for a few days until a
> > committer
>  assigned the issue
>  - not auto-closing the PR would at least allow the user to
>  provide
> >>> their
>  patch.
> 
>  On Wed, Apr 24, 2019 at 10:00 AM Stephan Ewen 
> > wrote:
> > +1 for option #2
> >
> > Seems to me that this does not contradict option #1, it only
>  extends
> >>> this
> > a bit. I think there is a good case for that, to help frequent
>  contributors
> > on a way to committership.
> >
> > @Konstantin: Trivial fixes (typos, docs, javadocs, ...) should
>  still
> > be
> > possible as "hotfixes".
> >
> > On Mon, Apr 15, 2019 at 3:14 PM Timo Walther  >
> >>> wrote:
> >> I think this really depends on the contribution.
> >>
> >> Sometimes "triviality" means that people just want to fix a typo
>  in
> >>> some
> >>

[jira] [Created] (FLINK-13101) Introduce "blocking after chaining off" property of StreamGraph

[Biao Liu (JIRA)]

Biao Liu created FLINK-13101:


 Summary: Introduce "blocking after chaining off" property of 
StreamGraph
 Key: FLINK-13101
 URL: https://issues.apache.org/jira/browse/FLINK-13101
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Biao Liu
Assignee: Biao Liu
 Fix For: 1.9.0


The property "blocking after chaining off" means, if there are some stream 
edges that can not be chained and the shuffle mode of edge is not specified, 
translate these edges into {{BLOCKING}} result partition type.

The reason of introducing it is to satisfy the requirement of Blink batch 
planner. Because the current scheduling strategy is a bit simple. It can not 
support some complex scenarios, like a batch job with resources limited.

To be honest, it's probably a work-around solution. However it's an internal 
implementation, we can replace it when we are able to support batch job by 
scheduling strategy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Flink framework and user log separation

[vino yang]

Hi Stephan,

Thanks for your reply.

In some cases, your solution can take effects.

However, in some scenarios, it does not meet the requirement:


   - One program has multiple job instances;
   - If we make Flink as a platform, we can not know the package of the
   users' program to config the log profiles before starting the cluster

Chesnay's understanding is right, we need to split business logs based on
the job.

Recently, a user also feedbacked this requirement.[1]

[1]: https://issues.apache.org/jira/browse/FLINK-12953

Stephan Ewen  于2019年7月4日周四 下午6:38写道：

> Is that something that can just be done by the right logging framework and
> configuration?
>
> Like having a log framework with two targets, one filtered on
> "org.apache.flink" and the other one filtered on "my.company.project" or
> so?
>
> On Fri, Mar 1, 2019 at 3:44 AM vino yang  wrote:
>
> > Hi Jamie Grier,
> >
> > Thank you for your reply, let me add some explanations to this design.
> >
> > First of all, as stated in "Goal", it is mainly for the "Standalone"
> > cluster model, although we have implemented it for Flink on YARN, this
> does
> > not mean that we can't turn off this feature by means of options. It
> should
> > be noted that the separation is basically based on the "log configuration
> > file", it is very scalable and even allows users to define the log
> pattern
> > of the configuration file (of course this is an extension feature, not
> > mentioned in the design documentation). In fact, "multiple files are a
> > special case of a single file", we can provide an option to keep it still
> > the default behavior, it should be the scene you expect in the container.
> >
> > According to Flink's official 2016 adjustment report [1], users using the
> > standalone mode are quite close to the yarn mode (unfortunately there is
> no
> > data support in 2017). Although we mainly use Flink on Yarn now, we have
> > used standalone in depth (close to the daily processing volume of 20
> > trillion messages). In this scenario, the user logs generated by
> different
> > job's tasks are mixed together, and it is very difficult to locate the
> > issue. Moreover, as we configure the log file scrolling policy, we have
> to
> > log in to the server to view it. Therefore, we expect that for the same
> > task manager, the user logs generated by the tasks from the same job can
> be
> > distinguished.
> >
> > In addition, I have tried MDC technology, but it can not achieve the
> goal.
> > The underlying Flink is log4j 1.x and logback. We need to be compatible
> > with both frameworks at the same time, and we don't allow large-scale
> > changes to the active code, and no sense to the user.
> >
> > Some other points:
> >
> > 1) Many of our users have experience using Storm and Spark, and they are
> > more accustomed to that style in standalone mode;
> > 2) We split the user log by Job, which will help to implement the
> "business
> > log aggregation" feature based on the Job.
> >
> > Best,
> > Vino
> >
> > [1]: https://www.ververica.com/blog/flink-user-survey-2016-part-1
> >
> > Jamie Grier  于2019年3月1日周五 上午7:32写道：
> >
> > > I think maybe if I understood this correctly this design is going in
> the
> > > wrong direction.  The problem with Flink logging, when you are running
> > > multiple jobs in the same TMs, is not just about separating out the
> > > business level logging into separate files.  The Flink framework itself
> > > logs many things where there is clearly a single job in context but
> that
> > > all ends up in the same log file and with no clear separation amongst
> the
> > > log lines.
> > >
> > > Also, I don't think shooting to have multiple log files is a very good
> > idea
> > > either.  It's common, especially on container-based deployments, that
> the
> > > expectation is that a process (like Flink) logs everything to stdout
> and
> > > the surrounding tooling takes care of routing that log data
> somewhere.  I
> > > think we should stick with that model and expect that there will be a
> > > single log stream coming out of each Flink process.
> > >
> > > Instead, I think it would be better to enhance Flink's logging
> capability
> > > such that the appropriate context can be added to each log line with
> the
> > > exact format controlled by the end user.  It might make sense to take a
> > > look at MDC, for example, as a way to approach this.
> > >
> > >
> > > On Thu, Feb 28, 2019 at 4:24 AM vino yang 
> wrote:
> > >
> > > > Dear devs,
> > > >
> > > > Currently, for log output, Flink does not explicitly distinguish
> > between
> > > > framework logs and user logs. In Task Manager, logs from the
> framework
> > > are
> > > > intermixed with the user's business logs. In some deployment models,
> > such
> > > > as Standalone or YARN session, there are different task instances of
> > > > different jobs deployed in the same Task Manager. It makes the log
> > event
> > > > flow more confusing unless the users explicitly use tags to
> dist

[jira] [Created] (FLINK-13102) Travis build optimization

[Alex (JIRA)]

Alex created FLINK-13102:


 Summary: Travis build optimization
 Key: FLINK-13102
 URL: https://issues.apache.org/jira/browse/FLINK-13102
 Project: Flink
  Issue Type: Improvement
  Components: Travis
Reporter: Alex


For some types of PRs, it's relatively easy to tell that the changes in a PR 
don't affect other parts of the system. Namely, it's documentation changes and 
maybe Web UI changes (although, the letter is not that frequently and regularly 
changes).

In such cases, it's possible to run more "lightweight" build steps, skipping 
the whole build and tests suite.

 

*Note* (from [~Zentol]): in that case, the check at least should run license 
checks.

 

 

The idea is that for documentation only changes, we can detect such PRs by 
testing that the list of changed files are only in {{docs/}} subdir (excluding 
{{docs/_includes/generated/}}). Detecting list of changed files during a Travis 
build is described in:

 * [https://twitter.com/keeskluskens/status/996005160089477120]

 * [https://reflectoring.io/skip-ci-build/]

 

*Additional improvement:* if we know that a PR is docs only change, we can also 
early run links checks during the on PR CI run (currently, such check is 
running after merge, by a scheduled run).

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Migrate to sponsored Travis account

[vino yang]

+1

Dian Fu  于2019年7月4日周四 下午7:09写道：

> +1. Thanks Chesnay and Bowen for pushing this forward.
>
> Regards,
> Dian
>
> > 在 2019年7月4日，下午6:28，zhijiang  写道：
> >
> > +1 and thanks for Chesnay' work on this.
> >
> > Best,
> > Zhijiang
> >
> > --
> > From:Haibo Sun 
> > Send Time:2019年7月4日(星期四) 18:21
> > To:dev 
> > Cc:priv...@flink.apache.org 
> > Subject:Re:Re: [VOTE] Migrate to sponsored Travis account
> >
> > +1. Thank Chesnay for pushing this forward.
> >
> > Best,
> > Haibo
> >
> >
> > At 2019-07-04 17:58:28, "Kurt Young"  wrote:
> >> +1 and great thanks Chesnay for pushing this.
> >>
> >> Best,
> >> Kurt
> >>
> >>
> >> On Thu, Jul 4, 2019 at 5:44 PM Aljoscha Krettek 
> wrote:
> >>
> >>> +1
> >>>
> >>> Aljoscha
> >>>
>  On 4. Jul 2019, at 11:09, Stephan Ewen  wrote:
> 
>  +1 to move to a private Travis account.
> 
>  I can confirm that Ververica will sponsor a Travis CI plan that is
>  equivalent or a bit higher than the previous ASF quota (10 concurrent
> >>> build
>  queues)
> 
>  Best,
>  Stephan
> 
>  On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler 
> >>> wrote:
> 
> > I've raised a JIRA
> > with INFRA to
> >>> inquire
> > whether it would be possible to switch to a different Travis account,
> > and if so what steps would need to be taken.
> > We need a proper confirmation from INFRA since we are not in full
> > control of the flink repository (for example, we cannot access the
> > settings page).
> >
> > If this is indeed possible, Ververica is willing sponsor a Travis
> > account for the Flink project.
> > This would provide us with more than enough resources than we need.
> >
> > Since this makes the project more reliant on resources provided by
> > external companies I would like to vote on this.
> >
> > Please vote on this proposal, as follows:
> > [ ] +1, Approve the migration to a Ververica-sponsored Travis
> account,
> > provided that INFRA approves
> > [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
> > account
> >
> > The vote will be open for at least 24h, and until we have
> confirmation
> > from INFRA. The voting period may be shorter than the usual 3 days
> since
> > our current is effectively not working.
> >
> > On 04/07/2019 06:51, Bowen Li wrote:
> >> Re: > Are they using their own Travis CI pool, or did the switch to
> an
> >> entirely different CI service?
> >>
> >> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> >> currently moving away from ASF's Travis to their own in-house metal
> >> machines at [1] with custom CI application at [2]. They've seen
> >> significant improvement w.r.t both much higher performance and
> >> basically no resource waiting time, "night-and-day" difference
> quoting
> >> Wes.
> >>
> >> Re: > If we can just switch to our own Travis pool, just for our
> >> project, then this might be something we can do fairly quickly?
> >>
> >> I believe so, according to [3] and [4]
> >>
> >>
> >> [1] https://ci.ursalabs.org/ 
> >> [2] https://github.com/ursa-labs/ursabot
> >> [3]
> >>
> >>>
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> >> [4]
> >>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> >>
> >>
> >>
> >> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler <
> ches...@apache.org
> >> > wrote:
> >>
> >>   Are they using their own Travis CI pool, or did the switch to an
> >>   entirely different CI service?
> >>
> >>   If we can just switch to our own Travis pool, just for our
> >>   project, then
> >>   this might be something we can do fairly quickly?
> >>
> >>   On 03/07/2019 05:55, Bowen Li wrote:
> >>> I responded in the INFRA ticket [1] that I believe they are
> >>   using a wrong
> >>> metric against Flink and the total build time is a completely
> >>   different
> >>> thing than guaranteed build capacity.
> >>>
> >>> My response:
> >>>
> >>> "As mentioned above, since I started to pay attention to Flink's
> >>   build
> >>> queue a few tens of days ago, I'm in Seattle and I saw no build
> >>   was kicking
> >>> off in PST daytime in weekdays for Flink. Our teammates in China
> >>   and Europe
> >>> have also reported similar observations. So we need to evaluate
> >>   how the
> >>> large total build time came from - if 1) your number and 2) our
> >>> observations from three locations that cover pretty much a full
> >>   day, are
> >>> all true, I **guess** one reason can be that - highly likely the
> >>   extra
> >>> build

Re: [VOTE] Migrate to sponsored Travis account

[Xu Forward]

+1

vino yang  于2019年7月4日周四 下午7:55写道：

> +1
>
> Dian Fu  于2019年7月4日周四 下午7:09写道：
>
> > +1. Thanks Chesnay and Bowen for pushing this forward.
> >
> > Regards,
> > Dian
> >
> > > 在 2019年7月4日，下午6:28，zhijiang  写道：
> > >
> > > +1 and thanks for Chesnay' work on this.
> > >
> > > Best,
> > > Zhijiang
> > >
> > > --
> > > From:Haibo Sun 
> > > Send Time:2019年7月4日(星期四) 18:21
> > > To:dev 
> > > Cc:priv...@flink.apache.org 
> > > Subject:Re:Re: [VOTE] Migrate to sponsored Travis account
> > >
> > > +1. Thank Chesnay for pushing this forward.
> > >
> > > Best,
> > > Haibo
> > >
> > >
> > > At 2019-07-04 17:58:28, "Kurt Young"  wrote:
> > >> +1 and great thanks Chesnay for pushing this.
> > >>
> > >> Best,
> > >> Kurt
> > >>
> > >>
> > >> On Thu, Jul 4, 2019 at 5:44 PM Aljoscha Krettek 
> > wrote:
> > >>
> > >>> +1
> > >>>
> > >>> Aljoscha
> > >>>
> >  On 4. Jul 2019, at 11:09, Stephan Ewen  wrote:
> > 
> >  +1 to move to a private Travis account.
> > 
> >  I can confirm that Ververica will sponsor a Travis CI plan that is
> >  equivalent or a bit higher than the previous ASF quota (10
> concurrent
> > >>> build
> >  queues)
> > 
> >  Best,
> >  Stephan
> > 
> >  On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler <
> ches...@apache.org>
> > >>> wrote:
> > 
> > > I've raised a JIRA
> > > with INFRA to
> > >>> inquire
> > > whether it would be possible to switch to a different Travis
> account,
> > > and if so what steps would need to be taken.
> > > We need a proper confirmation from INFRA since we are not in full
> > > control of the flink repository (for example, we cannot access the
> > > settings page).
> > >
> > > If this is indeed possible, Ververica is willing sponsor a Travis
> > > account for the Flink project.
> > > This would provide us with more than enough resources than we need.
> > >
> > > Since this makes the project more reliant on resources provided by
> > > external companies I would like to vote on this.
> > >
> > > Please vote on this proposal, as follows:
> > > [ ] +1, Approve the migration to a Ververica-sponsored Travis
> > account,
> > > provided that INFRA approves
> > > [ ] -1, Do not approach the migration to a Ververica-sponsored
> Travis
> > > account
> > >
> > > The vote will be open for at least 24h, and until we have
> > confirmation
> > > from INFRA. The voting period may be shorter than the usual 3 days
> > since
> > > our current is effectively not working.
> > >
> > > On 04/07/2019 06:51, Bowen Li wrote:
> > >> Re: > Are they using their own Travis CI pool, or did the switch
> to
> > an
> > >> entirely different CI service?
> > >>
> > >> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> > >> currently moving away from ASF's Travis to their own in-house
> metal
> > >> machines at [1] with custom CI application at [2]. They've seen
> > >> significant improvement w.r.t both much higher performance and
> > >> basically no resource waiting time, "night-and-day" difference
> > quoting
> > >> Wes.
> > >>
> > >> Re: > If we can just switch to our own Travis pool, just for our
> > >> project, then this might be something we can do fairly quickly?
> > >>
> > >> I believe so, according to [3] and [4]
> > >>
> > >>
> > >> [1] https://ci.ursalabs.org/ 
> > >> [2] https://github.com/ursa-labs/ursabot
> > >> [3]
> > >>
> > >>>
> > https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> > >> [4]
> > >>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> > >>
> > >>
> > >>
> > >> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler <
> > ches...@apache.org
> > >> > wrote:
> > >>
> > >>   Are they using their own Travis CI pool, or did the switch to an
> > >>   entirely different CI service?
> > >>
> > >>   If we can just switch to our own Travis pool, just for our
> > >>   project, then
> > >>   this might be something we can do fairly quickly?
> > >>
> > >>   On 03/07/2019 05:55, Bowen Li wrote:
> > >>> I responded in the INFRA ticket [1] that I believe they are
> > >>   using a wrong
> > >>> metric against Flink and the total build time is a completely
> > >>   different
> > >>> thing than guaranteed build capacity.
> > >>>
> > >>> My response:
> > >>>
> > >>> "As mentioned above, since I started to pay attention to Flink's
> > >>   build
> > >>> queue a few tens of days ago, I'm in Seattle and I saw no build
> > >>   was kicking
> > >>> off in PST daytime in weekdays for Flink. Our teammates in China
> > >>   and Europe
> > >>> 

Re: [VOTE] Migrate to sponsored Travis account

[Chesnay Schepler]

Note that the Flinkbot approach isn't that trivial either; we can't 
_just_ trigger builds for a branch in the apache repo, but would first 
have to clone the branch/pr into a separate repository (that is owned by 
the github account that the travis account would be tied to).


One roadblock after the next showing up...

On 04/07/2019 11:59, Chesnay Schepler wrote:

Small update with mostly bad news:

INFRA doesn't know whether it is possible, and referred my to Travis 
support.
They did point out that it could be problematic in regards to 
read/write permissions for the repository.


From my own findings /so far/ with a test repo/organization, it does 
not appear possible to configure the Travis account used for a 
specific repository.


So yeah, if we go down this route we may have to pimp the Flinkbot to 
trigger builds through the Travis REST API.


On 04/07/2019 10:46, Chesnay Schepler wrote:
I've raised a JIRA 
with INFRA to 
inquire whether it would be possible to switch to a different Travis 
account, and if so what steps would need to be taken.
We need a proper confirmation from INFRA since we are not in full 
control of the flink repository (for example, we cannot access the 
settings page).


If this is indeed possible, Ververica is willing sponsor a Travis 
account for the Flink project.

This would provide us with more than enough resources than we need.

Since this makes the project more reliant on resources provided by 
external companies I would like to vote on this.


Please vote on this proposal, as follows:
[ ] +1, Approve the migration to a Ververica-sponsored Travis 
account, provided that INFRA approves
[ ] -1, Do not approach the migration to a Ververica-sponsored Travis 
account


The vote will be open for at least 24h, and until we have 
confirmation from INFRA. The voting period may be shorter than the 
usual 3 days since our current is effectively not working.


On 04/07/2019 06:51, Bowen Li wrote:
Re: > Are they using their own Travis CI pool, or did the switch to 
an entirely different CI service?


I reached out to Wes and Krisztián from Apache Arrow PMC. They are 
currently moving away from ASF's Travis to their own in-house metal 
machines at [1] with custom CI application at [2]. They've seen 
significant improvement w.r.t both much higher performance and 
basically no resource waiting time, "night-and-day" difference 
quoting Wes.


Re: > If we can just switch to our own Travis pool, just for our 
project, then this might be something we can do fairly quickly?


I believe so, according to [3] and [4]


[1] https://ci.ursalabs.org/ 
[2] https://github.com/ursa-labs/ursabot
[3] 
https://docs.travis-ci.com/user/migrate/open-source-repository-migration 

[4] 
https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com




On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > wrote:


    Are they using their own Travis CI pool, or did the switch to an
    entirely different CI service?

    If we can just switch to our own Travis pool, just for our
    project, then
    this might be something we can do fairly quickly?

    On 03/07/2019 05:55, Bowen Li wrote:
    > I responded in the INFRA ticket [1] that I believe they are
    using a wrong
    > metric against Flink and the total build time is a completely
    different
    > thing than guaranteed build capacity.
    >
    > My response:
    >
    > "As mentioned above, since I started to pay attention to Flink's
    build
    > queue a few tens of days ago, I'm in Seattle and I saw no build
    was kicking
    > off in PST daytime in weekdays for Flink. Our teammates in China
    and Europe
    > have also reported similar observations. So we need to evaluate
    how the
    > large total build time came from - if 1) your number and 2) our
    > observations from three locations that cover pretty much a full
    day, are
    > all true, I **guess** one reason can be that - highly likely the
    extra
    > build time came from weekends when other Apache projects may be
    idle and
    > Flink just drains hard its congested queue.
    >
    > Please be aware of that we're not complaining about the lack of
    resources
    > in general, I'm complaining about the lack of **stable, 
dedicated**

    > resources. An example for the latter one is, currently even if
    no build is
    > in Flink's queue and I submit a request to be the queue head 
in PST

    > morning, my build won't even start in 6-8+h. That is an absurd
    amount of
    > waiting time.
    >
    > That's saying, if ASF INFRA decides to adopt a quota system and
    grants
    > Flink five DEDICATED servers that runs all the time only for
    Flink, that'll
    > be PERFECT and can totally solve our problem now.
    >
    > Please be aware of that we're not complaining about the lack of
    resources
    > in general, I'm complaining about the lack

Re: [DISCUSS]Support Upsert mode for Streaming Non-window FlatAggregate

[Jark Wu]

Hi jingcheng,

I agree with Kurt's point. As you said "the user must know the keys of the
output of UDTAGG clearly".
If I understand correctly, the key information is strongly relative to the
UDTAGG implementation.
Users may call `flatAggregate` on a UDTAGG instance with different keys
which may result in a wrong result.
So I think it would be better to couple key information with UDTAGG
interface (i.e. "Approach 3" in your design doc).

Regards,
Jark

On Thu, 4 Jul 2019 at 18:06, Kurt Young  wrote:

> Hi Jincheng,
>
> Thanks for the clarification. I think you just pointed out my concern by
> yourself:
>
> > When a user uses a User-defined table aggregate function (UDTAGG), he
> must understand the behavior of the UDTAGG, including the return type and
> the characteristics of the returned data. such as the key fields.
>
> This indicates that the UDTAGG is somehow be classified to different types,
> one will no key, one with key information. So the developer of the UDTAGG
> should choose which type of this function should be. In this case,
> my question would be, why don't we have explicit information about keys
> such as we split UDTAGG to keyed UDTAGG and non-keyed UDTAGG. So the user
> and the framework will have a better understanding of
> this UDTAGG. `withKeys` solution is letting user to choose the key and it
> seems it will only work correctly only if the user choose the *right* key
> this UDTAGG has.
>
> Let me know if this makes sense to you.
>
> Best,
> Kurt
>
>
> On Thu, Jul 4, 2019 at 4:32 PM jincheng sun 
> wrote:
>
> > Hi All,
> >
> > @Kurt Young  one user-defined table aggregate function
> > can be used in both with(out) keys case, and we do not introduce any
> other
> > aggregations. just like the explanation from @Hequn.
> >
> > @Hequn Cheng  thanks for your explanation!
> >
> > One thing should be mentioned here:
> >
> > When a user uses a User-defined table aggregate function (UDTAGG), he
> must
> > understand the behavior of the UDTAGG, including the return type and the
> > characteristics of the returned data. such as the key fields.  So
> although
> > using `withKeys` approach it is not rigorous enough(we do not need) but
> > intuitive enough, considering that if `flatAggregate` is followed by an
> > `upsertSink`, then the user must know the keys of the output of UDTAGG
> > clearly, otherwise the keys of `upsertSink` cannot be defined. So I still
> > prefer the `withKeys` solution by now.
> >
> > Looking forward to any feedback from all of you!
> >
> > Best,
> > Jincheng
> >
> >
> >
> > Hequn Cheng  于2019年7月1日周一 下午5:35写道：
> >
> >> Hi Kurt,
> >>
> >> Thanks for your questions. Here are my thoughts.
> >>
> >> > if I want to write such kind function, should I make sure that this
> >> function is used with some keys?
> >> The key information may not be used. We can also use RetractSink to emit
> >> the table directly.
> >>
> >> >  If I need a use case to calculate topn without key, should I write
> >> another function or I can reuse previous one.
> >> For the TopN example, you can reuse the previous function if you don't
> >> care
> >> about the key information.
> >>
> >> So, the key information is only an indicator(or a description), not an
> >> operator, as Jincheng mentioned above.
> >> We do not need to change the function logic and it will not add any
> other
> >> aggregations.
> >>
> >> BTW, we have three approaches in the document. Approach 1 defines keys
> on
> >> API level as we think it's more common to define keys on Table.
> >> While approach 3 defines keys in the TableAggregateFunction which is
> more
> >> precise but it is not very clear for Table users. So, we should take all
> >> these into consideration, and make the decision in this discussion
> thread.
> >>
> >> You can take a look at the document and welcome any suggestions or other
> >> better solutions.
> >>
> >> Best, Hequn
> >>
> >>
> >> On Mon, Jul 1, 2019 at 12:13 PM Kurt Young  wrote:
> >>
> >> > Hi Jincheng,
> >> >
> >> > Thanks for the clarification. Take 'TopN' for example, if I want to
> >> write
> >> > such kind function,
> >> > should I make sure that this function is used with some keys? If I
> need
> >> a
> >> > use case to calculate
> >> > topn without key, should I write another function or I can reuse
> >> previous
> >> > one.
> >> >
> >> > I'm not sure about the idea of this does not involve semantic changes.
> >> To
> >> > me, it sounds like
> >> > we are doing another nested aggregation inside the table
> >> > which TableAggregateFunction emits.
> >> >
> >> > Maybe I'm not familiar with this function enough, hope you can help me
> >> to
> >> > understand.
> >> >
> >> > Best,
> >> > Kurt
> >> >
> >> >
> >> > On Mon, Jul 1, 2019 at 11:59 AM jincheng sun <
> sunjincheng...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi Kurt,
> >> > >
> >> > > Thanks for your questions, I am glad to share my thoughts here:
> >> > >
> >> > > My question is, will that effect the logic ofTableAggregateFunction
> >> user
> >>

Re: Flink 1.9 blink planner

[Jark Wu]

Hi Kaka,

Thanks for trying to use blink planner.

We are working on bridging blink planner to the public API (i.e.
TableEnvironment) and this is on a good way.
Until then, we will remove blink's own TableEnvironment and use API's
TableEnvironment and your problem will be fixed.
Ideally, we can finish this at the end of this week.

Best,
Jark

On Thu, 4 Jul 2019 at 19:20, kaka chen  wrote:

> Hi All:
>
> We found TableEnvironment in Flink 1.9 blink planner didn't support
> functions such as sqlUpdate() and  registerTableSink(), when I used Scala
> api, it will throw error.
>
> *Error:(61, 20) value sqlUpdate is not a member of
> org.apache.flink.table.api.scala.StreamTableEnvironment.*
>
> Is it normal? Could someone help, thanks.
>
> Thanks,
> Kaka Chen
>

Re: Flink 1.9 blink planner

[kaka chen]

Hi Jark,

Got it. Thanks.

Thanks,
Kaka

Jark Wu  于2019年7月4日周四 下午8:30写道：

> Hi Kaka,
>
> Thanks for trying to use blink planner.
>
> We are working on bridging blink planner to the public API (i.e.
> TableEnvironment) and this is on a good way.
> Until then, we will remove blink's own TableEnvironment and use API's
> TableEnvironment and your problem will be fixed.
> Ideally, we can finish this at the end of this week.
>
> Best,
> Jark
>
> On Thu, 4 Jul 2019 at 19:20, kaka chen  wrote:
>
> > Hi All:
> >
> > We found TableEnvironment in Flink 1.9 blink planner didn't support
> > functions such as sqlUpdate() and  registerTableSink(), when I used Scala
> > api, it will throw error.
> >
> > *Error:(61, 20) value sqlUpdate is not a member of
> > org.apache.flink.table.api.scala.StreamTableEnvironment.*
> >
> > Is it normal? Could someone help, thanks.
> >
> > Thanks,
> > Kaka Chen
> >
>

[jira] [Created] (FLINK-13103) Graceful Shutdown Handling by UDFs

[Kostas Kloudas (JIRA)]

Kostas Kloudas created FLINK-13103:
--

 Summary: Graceful Shutdown Handling by UDFs
 Key: FLINK-13103
 URL: https://issues.apache.org/jira/browse/FLINK-13103
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Affects Versions: 1.8.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas


This is an umbrella issue for 
[FLIP-46|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs]]
 and it will be broken down into more fine grained JIRAs as the discussion on 
the FLIP evolves.

 

For more details on what the FLIP is about, please refer to the link above.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] Graceful Shutdown Handling by UDFs.

[Kostas Kloudas]

Hi all,

In many cases, UDFs (User Defined Functions) need to be able to perform
application-specific actions when they stop in an orderly manner.
Currently, Flink's UDFs, and more specifically the RichFunction which
exposes lifecycle-related hooks, only have a close() method which is called
in any case of job termination. This includes any form of orderly
termination (STOP or End-Of-Stream) and termination due to an error.


The FLIP in [1] and the design document in [2] propose the addition of an
interface that will allow UDFs that implement it to perform application
specific logic in the case of graceful termination. These cases include
DRAIN and SUSPEND for streaming jobs (see FLIP-34), but also reaching the
End-Of-Stream for jobs with finite sources.

Let's have a lively discussion to solve this issue that has been around for
quite some time.

Cheers,
Kostas

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs

[2]
https://docs.google.com/document/d/1SXfhmeiJfWqi2ITYgCgAoSDUv5PNq1T8Zu01nR5Ebog/edit?usp=sharing

Re: [VOTE] Migrate to sponsored Travis account

[Hequn Cheng]

+1.

And thanks a lot to Chesnay for pushing this.

Best, Hequn

On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler  wrote:

> Note that the Flinkbot approach isn't that trivial either; we can't
> _just_ trigger builds for a branch in the apache repo, but would first
> have to clone the branch/pr into a separate repository (that is owned by
> the github account that the travis account would be tied to).
>
> One roadblock after the next showing up...
>
> On 04/07/2019 11:59, Chesnay Schepler wrote:
> > Small update with mostly bad news:
> >
> > INFRA doesn't know whether it is possible, and referred my to Travis
> > support.
> > They did point out that it could be problematic in regards to
> > read/write permissions for the repository.
> >
> > From my own findings /so far/ with a test repo/organization, it does
> > not appear possible to configure the Travis account used for a
> > specific repository.
> >
> > So yeah, if we go down this route we may have to pimp the Flinkbot to
> > trigger builds through the Travis REST API.
> >
> > On 04/07/2019 10:46, Chesnay Schepler wrote:
> >> I've raised a JIRA
> >> with INFRA to
> >> inquire whether it would be possible to switch to a different Travis
> >> account, and if so what steps would need to be taken.
> >> We need a proper confirmation from INFRA since we are not in full
> >> control of the flink repository (for example, we cannot access the
> >> settings page).
> >>
> >> If this is indeed possible, Ververica is willing sponsor a Travis
> >> account for the Flink project.
> >> This would provide us with more than enough resources than we need.
> >>
> >> Since this makes the project more reliant on resources provided by
> >> external companies I would like to vote on this.
> >>
> >> Please vote on this proposal, as follows:
> >> [ ] +1, Approve the migration to a Ververica-sponsored Travis
> >> account, provided that INFRA approves
> >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
> >> account
> >>
> >> The vote will be open for at least 24h, and until we have
> >> confirmation from INFRA. The voting period may be shorter than the
> >> usual 3 days since our current is effectively not working.
> >>
> >> On 04/07/2019 06:51, Bowen Li wrote:
> >>> Re: > Are they using their own Travis CI pool, or did the switch to
> >>> an entirely different CI service?
> >>>
> >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> >>> currently moving away from ASF's Travis to their own in-house metal
> >>> machines at [1] with custom CI application at [2]. They've seen
> >>> significant improvement w.r.t both much higher performance and
> >>> basically no resource waiting time, "night-and-day" difference
> >>> quoting Wes.
> >>>
> >>> Re: > If we can just switch to our own Travis pool, just for our
> >>> project, then this might be something we can do fairly quickly?
> >>>
> >>> I believe so, according to [3] and [4]
> >>>
> >>>
> >>> [1] https://ci.ursalabs.org/ 
> >>> [2] https://github.com/ursa-labs/ursabot
> >>> [3]
> >>>
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> >>>
> >>> [4]
> >>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> >>>
> >>>
> >>>
> >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler  >>> > wrote:
> >>>
> >>> Are they using their own Travis CI pool, or did the switch to an
> >>> entirely different CI service?
> >>>
> >>> If we can just switch to our own Travis pool, just for our
> >>> project, then
> >>> this might be something we can do fairly quickly?
> >>>
> >>> On 03/07/2019 05:55, Bowen Li wrote:
> >>> > I responded in the INFRA ticket [1] that I believe they are
> >>> using a wrong
> >>> > metric against Flink and the total build time is a completely
> >>> different
> >>> > thing than guaranteed build capacity.
> >>> >
> >>> > My response:
> >>> >
> >>> > "As mentioned above, since I started to pay attention to Flink's
> >>> build
> >>> > queue a few tens of days ago, I'm in Seattle and I saw no build
> >>> was kicking
> >>> > off in PST daytime in weekdays for Flink. Our teammates in China
> >>> and Europe
> >>> > have also reported similar observations. So we need to evaluate
> >>> how the
> >>> > large total build time came from - if 1) your number and 2) our
> >>> > observations from three locations that cover pretty much a full
> >>> day, are
> >>> > all true, I **guess** one reason can be that - highly likely the
> >>> extra
> >>> > build time came from weekends when other Apache projects may be
> >>> idle and
> >>> > Flink just drains hard its congested queue.
> >>> >
> >>> > Please be aware of that we're not complaining about the lack of
> >>> resources
> >>> > in general, I'm complaining about the lack of **s

Re: [DISCUSS] Make window state queryable

[mayozhang]

It’s a good idea to get the process information of large ongoing window.
+1 from my side.

> 在 2019年7月4日，11:41，vino yang  写道：
> 
> Hi folks,
> 
> Currently, the queryable state is not widely used in production. IMO, there
> are two key reasons caused this result. 1) the client of the queryable
> state is hard to use. Because it requires users to know the address of
> TaskManager and the port of the proxy. Actually, most business users who do
> not have good knowledge about the Flink's inner and runtime in production.
> 2) The benefit of this feature has not been excavated. In Flink DataStream
> API, State is the first level citizen, it’s Flink key advantage compared
> with other compute engines. Because the queryable state is the most
> effective way to pry the latest computing progress.
> 
> Three months ago, I started a discussion about improving the queryable
> state and introducing a proxy component.[1] It brings a lot of attention
> and discussion. Recently, I have submitted a design document about the
> proposal.[2] These efforts try to process the first problem.
> 
> About the second question, the most essential solution is that we should
> really make the queryable state work. The window operator is one of the
> most valuable and most frequently used operators of all Flink operators.
> And it also uses keyed state which is queryable. So we propose to let the
> state of the window operator be queried. This is not only for increasing
> the value of the queryable state but also for the real business needs.
> 
> IMO, allowing window state to be queried will provide great value. In many
> scenarios, we often use large windows for aggregate calculations. A very
> common example is a day-level window that counts the PV of a day. But
> usually, the user is not only satisfied to wait until the end of the window
> to get the result. They want to get "intermediate results" at a smaller
> time granularity to analyze trends. Because Flink does not provide periodic
> triggers for fixed windows. We have extended this and implemented an
> "incremental window". It can trigger a fixed window with a smaller interval
> period and feedback intermediate results. However, we believe that this
> approach is still not flexible enough. We should let the user query the
> current calculation result of the window through the API at any time.
> 
> However, I know that if we want to implement it, we still have some details
> that need to be discussed, such as how to let users know the state
> descriptors in the window, namespace and so on.
> 
> This discussion thread is mainly to listen to the community's opinion on
> this proposal.
> 
> Any feedback and ideas are welcome and appreciated.
> 
> Best,
> Vino
> 
> [1]:
> http://mail-archives.apache.org/mod_mbox/flink-dev/201907.mbox/%3ctencent_35a56d6858408be2e2064...@qq.com%3E
> [2]:
> https://docs.google.com/document/d/181qYVIiHQGrc3hCj3QBn1iEHF4bUztdw4XO8VSaf_uI/edit?usp=sharing

Re: [DISCUSS]Support Upsert mode for Streaming Non-window FlatAggregate

[Hequn Cheng]

Hi Kurt and Jark,

Thanks a lot for your great inputs!

The keys of the query may not strongly be related to the UDTAGG.
It may also be related to the corresponding scenarios that a user wants to
achieve.

For example, take TopN again as an example. If the TopN table aggregate
function
outputs three columns(rankid, time, value), either rankid or rankid+time
could be
used as the key. Which one to be chosen is more likely to be decided by the
user
according to his business.

Best, Hequn

On Thu, Jul 4, 2019 at 8:11 PM Jark Wu  wrote:

> Hi jingcheng,
>
> I agree with Kurt's point. As you said "the user must know the keys of the
> output of UDTAGG clearly".
> If I understand correctly, the key information is strongly relative to the
> UDTAGG implementation.
> Users may call `flatAggregate` on a UDTAGG instance with different keys
> which may result in a wrong result.
> So I think it would be better to couple key information with UDTAGG
> interface (i.e. "Approach 3" in your design doc).
>
> Regards,
> Jark
>
> On Thu, 4 Jul 2019 at 18:06, Kurt Young  wrote:
>
>> Hi Jincheng,
>>
>> Thanks for the clarification. I think you just pointed out my concern by
>> yourself:
>>
>> > When a user uses a User-defined table aggregate function (UDTAGG), he
>> must understand the behavior of the UDTAGG, including the return type and
>> the characteristics of the returned data. such as the key fields.
>>
>> This indicates that the UDTAGG is somehow be classified to different
>> types,
>> one will no key, one with key information. So the developer of the UDTAGG
>> should choose which type of this function should be. In this case,
>> my question would be, why don't we have explicit information about keys
>> such as we split UDTAGG to keyed UDTAGG and non-keyed UDTAGG. So the user
>> and the framework will have a better understanding of
>> this UDTAGG. `withKeys` solution is letting user to choose the key and it
>> seems it will only work correctly only if the user choose the *right* key
>> this UDTAGG has.
>>
>> Let me know if this makes sense to you.
>>
>> Best,
>> Kurt
>>
>>
>> On Thu, Jul 4, 2019 at 4:32 PM jincheng sun 
>> wrote:
>>
>> > Hi All,
>> >
>> > @Kurt Young  one user-defined table aggregate
>> function
>> > can be used in both with(out) keys case, and we do not introduce any
>> other
>> > aggregations. just like the explanation from @Hequn.
>> >
>> > @Hequn Cheng  thanks for your explanation!
>> >
>> > One thing should be mentioned here:
>> >
>> > When a user uses a User-defined table aggregate function (UDTAGG), he
>> must
>> > understand the behavior of the UDTAGG, including the return type and the
>> > characteristics of the returned data. such as the key fields.  So
>> although
>> > using `withKeys` approach it is not rigorous enough(we do not need) but
>> > intuitive enough, considering that if `flatAggregate` is followed by an
>> > `upsertSink`, then the user must know the keys of the output of UDTAGG
>> > clearly, otherwise the keys of `upsertSink` cannot be defined. So I
>> still
>> > prefer the `withKeys` solution by now.
>> >
>> > Looking forward to any feedback from all of you!
>> >
>> > Best,
>> > Jincheng
>> >
>> >
>> >
>> > Hequn Cheng  于2019年7月1日周一 下午5:35写道：
>> >
>> >> Hi Kurt,
>> >>
>> >> Thanks for your questions. Here are my thoughts.
>> >>
>> >> > if I want to write such kind function, should I make sure that this
>> >> function is used with some keys?
>> >> The key information may not be used. We can also use RetractSink to
>> emit
>> >> the table directly.
>> >>
>> >> >  If I need a use case to calculate topn without key, should I write
>> >> another function or I can reuse previous one.
>> >> For the TopN example, you can reuse the previous function if you don't
>> >> care
>> >> about the key information.
>> >>
>> >> So, the key information is only an indicator(or a description), not an
>> >> operator, as Jincheng mentioned above.
>> >> We do not need to change the function logic and it will not add any
>> other
>> >> aggregations.
>> >>
>> >> BTW, we have three approaches in the document. Approach 1 defines keys
>> on
>> >> API level as we think it's more common to define keys on Table.
>> >> While approach 3 defines keys in the TableAggregateFunction which is
>> more
>> >> precise but it is not very clear for Table users. So, we should take
>> all
>> >> these into consideration, and make the decision in this discussion
>> thread.
>> >>
>> >> You can take a look at the document and welcome any suggestions or
>> other
>> >> better solutions.
>> >>
>> >> Best, Hequn
>> >>
>> >>
>> >> On Mon, Jul 1, 2019 at 12:13 PM Kurt Young  wrote:
>> >>
>> >> > Hi Jincheng,
>> >> >
>> >> > Thanks for the clarification. Take 'TopN' for example, if I want to
>> >> write
>> >> > such kind function,
>> >> > should I make sure that this function is used with some keys? If I
>> need
>> >> a
>> >> > use case to calculate
>> >> > topn without key, should I write another function or I can reuse
>> >> previous
>> >> 

[VOTE] How to Deal with Split/Select in DataStream API

[Xingcan Cui]

Hi folks,

Two weeks ago, I started a thread [1] discussing whether we should discard
the split/select methods (which have been marked as deprecation since v1.7)
in DataStream API.

The fact is, these methods will cause "unexpected" results when using
consecutively (e.g., ds.split(a).select(b).split(c).select(d)) or
multi-times on the same target (e.g., ds.split(a).select(b),
ds.split(c).select(d)). The reason is that following the initial design,
the new split/select logic will always override the existing one on the
same target operator, rather than append to it. Some users may not be aware
of that, but if you do, a current solution would be to use the more
powerful side output feature [2].

FLINK-11084  added some
restrictions to the existing split/select logic and suggest to replace it with
side output in the future. However, considering that the side output is
currently only available in the process function layer and the split/select
could have been widely used in many real-world applications, we'd like to start
a vote andlisten to the community on how to deal with them.

In the discussion thread [1], we proposed three solutions as follows. All
of them are feasible but have different impacts on the public API.

1) Port the side output feature to DataStream API's flatMap and replace
split/select with it.

2) Introduce a dedicated function in DataStream API (with the "correct"
behavior but a different name) that can be used to replace the existing
split/select.

3) Keep split/select but change the behavior/semantic to be "correct".

Note that this is just a vote for gathering information, so feel free to
participate and share your opinions.

The voting time will end on *July 7th 17:00 EDT*.

Thanks,
Xingcan

[1]
https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d400cb7c61859e1f4@%3Cdev.flink.apache.org%3E
[2]
https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/side_output.html

Re: [VOTE] Migrate to sponsored Travis account

[Bowen Li]

+1 on approval of the migration to our own Travis account. The foreseeable
benefits of the whole community's productivity and iteration speed would be
significant!

I think using Flinkbot or Travis REST API would be an implementation
details. Once we determine the overall direction, details can be figured
out.

Good news is that, upon my research on how Arrow and Spark integrate their
own in-house CI services with github repo, they are both using bots with
Github API. See a typical PR check for those projects at [1] and [2]. Thus,
we are **not alone** on this path.

Specifically for Apache Arrow, they have 'Ursabot', similar to our
Flinkbot, as I shared the link in the discussion. [3] lays out how Usrabot
works and integrates with Github API to trigger build. I think their
documentations is a bit outdated though - the doc says it cannot report
back build status to github, but from [1] we can see that the build status
are actually reported.

@Chesnay thanks for taking actions on this. Though I don't have access to
settings of Flink's github repo, I will continue to help push this
initiative in whichever way I can. Wes and Krisztián from Arrow are also
very friendly and helpful, and I can connect you to them to learn their
experience.

[1] https://github.com/apache/arrow/pull/4809
[2] https://github.com/apache/spark/pull/25053
[3] https://github.com/ursa-labs/ursabot#driving-ursabot


On Thu, Jul 4, 2019 at 6:42 AM Hequn Cheng  wrote:

> +1.
>
> And thanks a lot to Chesnay for pushing this.
>
> Best, Hequn
>
> On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler 
> wrote:
>
>> Note that the Flinkbot approach isn't that trivial either; we can't
>> _just_ trigger builds for a branch in the apache repo, but would first
>> have to clone the branch/pr into a separate repository (that is owned by
>> the github account that the travis account would be tied to).
>>
>> One roadblock after the next showing up...
>>
>> On 04/07/2019 11:59, Chesnay Schepler wrote:
>> > Small update with mostly bad news:
>> >
>> > INFRA doesn't know whether it is possible, and referred my to Travis
>> > support.
>> > They did point out that it could be problematic in regards to
>> > read/write permissions for the repository.
>> >
>> > From my own findings /so far/ with a test repo/organization, it does
>> > not appear possible to configure the Travis account used for a
>> > specific repository.
>> >
>> > So yeah, if we go down this route we may have to pimp the Flinkbot to
>> > trigger builds through the Travis REST API.
>> >
>> > On 04/07/2019 10:46, Chesnay Schepler wrote:
>> >> I've raised a JIRA
>> >> with INFRA to
>> >> inquire whether it would be possible to switch to a different Travis
>> >> account, and if so what steps would need to be taken.
>> >> We need a proper confirmation from INFRA since we are not in full
>> >> control of the flink repository (for example, we cannot access the
>> >> settings page).
>> >>
>> >> If this is indeed possible, Ververica is willing sponsor a Travis
>> >> account for the Flink project.
>> >> This would provide us with more than enough resources than we need.
>> >>
>> >> Since this makes the project more reliant on resources provided by
>> >> external companies I would like to vote on this.
>> >>
>> >> Please vote on this proposal, as follows:
>> >> [ ] +1, Approve the migration to a Ververica-sponsored Travis
>> >> account, provided that INFRA approves
>> >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
>> >> account
>> >>
>> >> The vote will be open for at least 24h, and until we have
>> >> confirmation from INFRA. The voting period may be shorter than the
>> >> usual 3 days since our current is effectively not working.
>> >>
>> >> On 04/07/2019 06:51, Bowen Li wrote:
>> >>> Re: > Are they using their own Travis CI pool, or did the switch to
>> >>> an entirely different CI service?
>> >>>
>> >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
>> >>> currently moving away from ASF's Travis to their own in-house metal
>> >>> machines at [1] with custom CI application at [2]. They've seen
>> >>> significant improvement w.r.t both much higher performance and
>> >>> basically no resource waiting time, "night-and-day" difference
>> >>> quoting Wes.
>> >>>
>> >>> Re: > If we can just switch to our own Travis pool, just for our
>> >>> project, then this might be something we can do fairly quickly?
>> >>>
>> >>> I believe so, according to [3] and [4]
>> >>>
>> >>>
>> >>> [1] https://ci.ursalabs.org/ 
>> >>> [2] https://github.com/ursa-labs/ursabot
>> >>> [3]
>> >>>
>> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
>> >>>
>> >>> [4]
>> >>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > >>> > wrote:
>> >>>
>> >>> Are they using their o

[jira] [Created] (FLINK-13104) Flink Datadog metrics client callback does not check for errors on posting and fails silently

[Mans Singh (JIRA)]

Mans Singh created FLINK-13104:
--

 Summary: Flink Datadog metrics client callback does not check for 
errors on posting and fails silently
 Key: FLINK-13104
 URL: https://issues.apache.org/jira/browse/FLINK-13104
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics
Affects Versions: 1.8.1
Reporter: Mans Singh
Assignee: Mans Singh
 Fix For: 1.9.0


Flink DatadogHttpClient's callback does not check if the request was 
successful. In case of non-successful posting request it should log a warning 
so the the error can be resolved.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] How to Deal with Split/Select in DataStream API

[Hao Sun]

Personally I prefer 3) to keep split/select and correct the behavior. I
feel side output is kind of overkill for such a primitive function, and I
prefer simple APIs like split/select.

Hao Sun


On Thu, Jul 4, 2019 at 11:20 AM Xingcan Cui  wrote:

> Hi folks,
>
> Two weeks ago, I started a thread [1] discussing whether we should discard
> the split/select methods (which have been marked as deprecation since v1.7)
> in DataStream API.
>
> The fact is, these methods will cause "unexpected" results when using
> consecutively (e.g., ds.split(a).select(b).split(c).select(d)) or
> multi-times on the same target (e.g., ds.split(a).select(b),
> ds.split(c).select(d)). The reason is that following the initial design,
> the new split/select logic will always override the existing one on the
> same target operator, rather than append to it. Some users may not be
> aware of that, but if you do, a current solution would be to use the more
> powerful side output feature [2].
>
> FLINK-11084  added
> some restrictions to the existing split/select logic and suggest to
> replace it with side output in the future. However, considering that the
> side output is currently only available in the process function layer and
> the split/select could have been widely used in many real-world
> applications, we'd like to start a vote andlisten to the community on how
> to deal with them.
>
> In the discussion thread [1], we proposed three solutions as follows. All
> of them are feasible but have different impacts on the public API.
>
> 1) Port the side output feature to DataStream API's flatMap and replace
> split/select with it.
>
> 2) Introduce a dedicated function in DataStream API (with the "correct"
> behavior but a different name) that can be used to replace the existing
> split/select.
>
> 3) Keep split/select but change the behavior/semantic to be "correct".
>
> Note that this is just a vote for gathering information, so feel free to
> participate and share your opinions.
>
> The voting time will end on *July 7th 17:00 EDT*.
>
> Thanks,
> Xingcan
>
> [1]
> https://lists.apache.org/thread.html/f94ea5c97f96c705527dcc809b0e2b69e87a4c5d400cb7c61859e1f4@%3Cdev.flink.apache.org%3E
> 
> [2]
> https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/side_output.html
> 
>

Re: [VOTE] Migrate to sponsored Travis account

[Congxian Qiu]

+1 for the migration.

Best,
Congxian


Hequn Cheng  于2019年7月4日周四 下午9:42写道：

> +1.
>
> And thanks a lot to Chesnay for pushing this.
>
> Best, Hequn
>
> On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler 
> wrote:
>
> > Note that the Flinkbot approach isn't that trivial either; we can't
> > _just_ trigger builds for a branch in the apache repo, but would first
> > have to clone the branch/pr into a separate repository (that is owned by
> > the github account that the travis account would be tied to).
> >
> > One roadblock after the next showing up...
> >
> > On 04/07/2019 11:59, Chesnay Schepler wrote:
> > > Small update with mostly bad news:
> > >
> > > INFRA doesn't know whether it is possible, and referred my to Travis
> > > support.
> > > They did point out that it could be problematic in regards to
> > > read/write permissions for the repository.
> > >
> > > From my own findings /so far/ with a test repo/organization, it does
> > > not appear possible to configure the Travis account used for a
> > > specific repository.
> > >
> > > So yeah, if we go down this route we may have to pimp the Flinkbot to
> > > trigger builds through the Travis REST API.
> > >
> > > On 04/07/2019 10:46, Chesnay Schepler wrote:
> > >> I've raised a JIRA
> > >> with INFRA to
> > >> inquire whether it would be possible to switch to a different Travis
> > >> account, and if so what steps would need to be taken.
> > >> We need a proper confirmation from INFRA since we are not in full
> > >> control of the flink repository (for example, we cannot access the
> > >> settings page).
> > >>
> > >> If this is indeed possible, Ververica is willing sponsor a Travis
> > >> account for the Flink project.
> > >> This would provide us with more than enough resources than we need.
> > >>
> > >> Since this makes the project more reliant on resources provided by
> > >> external companies I would like to vote on this.
> > >>
> > >> Please vote on this proposal, as follows:
> > >> [ ] +1, Approve the migration to a Ververica-sponsored Travis
> > >> account, provided that INFRA approves
> > >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
> > >> account
> > >>
> > >> The vote will be open for at least 24h, and until we have
> > >> confirmation from INFRA. The voting period may be shorter than the
> > >> usual 3 days since our current is effectively not working.
> > >>
> > >> On 04/07/2019 06:51, Bowen Li wrote:
> > >>> Re: > Are they using their own Travis CI pool, or did the switch to
> > >>> an entirely different CI service?
> > >>>
> > >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> > >>> currently moving away from ASF's Travis to their own in-house metal
> > >>> machines at [1] with custom CI application at [2]. They've seen
> > >>> significant improvement w.r.t both much higher performance and
> > >>> basically no resource waiting time, "night-and-day" difference
> > >>> quoting Wes.
> > >>>
> > >>> Re: > If we can just switch to our own Travis pool, just for our
> > >>> project, then this might be something we can do fairly quickly?
> > >>>
> > >>> I believe so, according to [3] and [4]
> > >>>
> > >>>
> > >>> [1] https://ci.ursalabs.org/ 
> > >>> [2] https://github.com/ursa-labs/ursabot
> > >>> [3]
> > >>>
> > https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> > >>>
> > >>> [4]
> > >>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler  > >>> > wrote:
> > >>>
> > >>> Are they using their own Travis CI pool, or did the switch to an
> > >>> entirely different CI service?
> > >>>
> > >>> If we can just switch to our own Travis pool, just for our
> > >>> project, then
> > >>> this might be something we can do fairly quickly?
> > >>>
> > >>> On 03/07/2019 05:55, Bowen Li wrote:
> > >>> > I responded in the INFRA ticket [1] that I believe they are
> > >>> using a wrong
> > >>> > metric against Flink and the total build time is a completely
> > >>> different
> > >>> > thing than guaranteed build capacity.
> > >>> >
> > >>> > My response:
> > >>> >
> > >>> > "As mentioned above, since I started to pay attention to
> Flink's
> > >>> build
> > >>> > queue a few tens of days ago, I'm in Seattle and I saw no build
> > >>> was kicking
> > >>> > off in PST daytime in weekdays for Flink. Our teammates in
> China
> > >>> and Europe
> > >>> > have also reported similar observations. So we need to evaluate
> > >>> how the
> > >>> > large total build time came from - if 1) your number and 2) our
> > >>> > observations from three locations that cover pretty much a full
> > >>> day, are
> > >>> > all true, I **guess** one reason can be that - highly likely
> the
> > >>> extra
> > >>>

[jira] [Created] (FLINK-13105) Add documentation for blink planner's built-in functions

[Zhenghua Gao (JIRA)]

Zhenghua Gao created FLINK-13105:


 Summary: Add documentation for blink planner's built-in functions
 Key: FLINK-13105
 URL: https://issues.apache.org/jira/browse/FLINK-13105
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: Zhenghua Gao
Assignee: Zhenghua Gao
 Fix For: 1.9.0


Blink planner intros some built-in functions which need to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Migrate to sponsored Travis account

[Jark Wu]

+1 for the migration and great thanks to Chesnay and Bowen for pushing this!

Cheers,
Jark

On Fri, 5 Jul 2019 at 09:34, Congxian Qiu  wrote:

> +1 for the migration.
>
> Best,
> Congxian
>
>
> Hequn Cheng  于2019年7月4日周四 下午9:42写道：
>
> > +1.
> >
> > And thanks a lot to Chesnay for pushing this.
> >
> > Best, Hequn
> >
> > On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler 
> > wrote:
> >
> > > Note that the Flinkbot approach isn't that trivial either; we can't
> > > _just_ trigger builds for a branch in the apache repo, but would first
> > > have to clone the branch/pr into a separate repository (that is owned
> by
> > > the github account that the travis account would be tied to).
> > >
> > > One roadblock after the next showing up...
> > >
> > > On 04/07/2019 11:59, Chesnay Schepler wrote:
> > > > Small update with mostly bad news:
> > > >
> > > > INFRA doesn't know whether it is possible, and referred my to Travis
> > > > support.
> > > > They did point out that it could be problematic in regards to
> > > > read/write permissions for the repository.
> > > >
> > > > From my own findings /so far/ with a test repo/organization, it does
> > > > not appear possible to configure the Travis account used for a
> > > > specific repository.
> > > >
> > > > So yeah, if we go down this route we may have to pimp the Flinkbot to
> > > > trigger builds through the Travis REST API.
> > > >
> > > > On 04/07/2019 10:46, Chesnay Schepler wrote:
> > > >> I've raised a JIRA
> > > >> with INFRA to
> > > >> inquire whether it would be possible to switch to a different Travis
> > > >> account, and if so what steps would need to be taken.
> > > >> We need a proper confirmation from INFRA since we are not in full
> > > >> control of the flink repository (for example, we cannot access the
> > > >> settings page).
> > > >>
> > > >> If this is indeed possible, Ververica is willing sponsor a Travis
> > > >> account for the Flink project.
> > > >> This would provide us with more than enough resources than we need.
> > > >>
> > > >> Since this makes the project more reliant on resources provided by
> > > >> external companies I would like to vote on this.
> > > >>
> > > >> Please vote on this proposal, as follows:
> > > >> [ ] +1, Approve the migration to a Ververica-sponsored Travis
> > > >> account, provided that INFRA approves
> > > >> [ ] -1, Do not approach the migration to a Ververica-sponsored
> Travis
> > > >> account
> > > >>
> > > >> The vote will be open for at least 24h, and until we have
> > > >> confirmation from INFRA. The voting period may be shorter than the
> > > >> usual 3 days since our current is effectively not working.
> > > >>
> > > >> On 04/07/2019 06:51, Bowen Li wrote:
> > > >>> Re: > Are they using their own Travis CI pool, or did the switch to
> > > >>> an entirely different CI service?
> > > >>>
> > > >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> > > >>> currently moving away from ASF's Travis to their own in-house metal
> > > >>> machines at [1] with custom CI application at [2]. They've seen
> > > >>> significant improvement w.r.t both much higher performance and
> > > >>> basically no resource waiting time, "night-and-day" difference
> > > >>> quoting Wes.
> > > >>>
> > > >>> Re: > If we can just switch to our own Travis pool, just for our
> > > >>> project, then this might be something we can do fairly quickly?
> > > >>>
> > > >>> I believe so, according to [3] and [4]
> > > >>>
> > > >>>
> > > >>> [1] https://ci.ursalabs.org/ 
> > > >>> [2] https://github.com/ursa-labs/ursabot
> > > >>> [3]
> > > >>>
> > >
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> > > >>>
> > > >>> [4]
> > > >>>
> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler <
> ches...@apache.org
> > > >>> > wrote:
> > > >>>
> > > >>> Are they using their own Travis CI pool, or did the switch to
> an
> > > >>> entirely different CI service?
> > > >>>
> > > >>> If we can just switch to our own Travis pool, just for our
> > > >>> project, then
> > > >>> this might be something we can do fairly quickly?
> > > >>>
> > > >>> On 03/07/2019 05:55, Bowen Li wrote:
> > > >>> > I responded in the INFRA ticket [1] that I believe they are
> > > >>> using a wrong
> > > >>> > metric against Flink and the total build time is a completely
> > > >>> different
> > > >>> > thing than guaranteed build capacity.
> > > >>> >
> > > >>> > My response:
> > > >>> >
> > > >>> > "As mentioned above, since I started to pay attention to
> > Flink's
> > > >>> build
> > > >>> > queue a few tens of days ago, I'm in Seattle and I saw no
> build
> > > >>> was kicking
> > > >>> > off in PST daytime in weekdays for Flink. Our teammates in
>

Re: [VOTE] Migrate to sponsored Travis account

[Yun Tang]

I noticed that switching to a separate Travis account to run CI is actually 
impossible from what https://issues.apache.org/jira/browse/INFRA-18703 said. 
Hope another option from Chesnay to speed up the CI progress would work soon.


Best
Yun Tang

From: Jark Wu 
Sent: Friday, July 5, 2019 10:34
To: dev
Cc: priv...@flink.apache.org; Bowen Li
Subject: Re: [VOTE] Migrate to sponsored Travis account

+1 for the migration and great thanks to Chesnay and Bowen for pushing this!

Cheers,
Jark

On Fri, 5 Jul 2019 at 09:34, Congxian Qiu  wrote:

> +1 for the migration.
>
> Best,
> Congxian
>
>
> Hequn Cheng  于2019年7月4日周四 下午9:42写道：
>
> > +1.
> >
> > And thanks a lot to Chesnay for pushing this.
> >
> > Best, Hequn
> >
> > On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler 
> > wrote:
> >
> > > Note that the Flinkbot approach isn't that trivial either; we can't
> > > _just_ trigger builds for a branch in the apache repo, but would first
> > > have to clone the branch/pr into a separate repository (that is owned
> by
> > > the github account that the travis account would be tied to).
> > >
> > > One roadblock after the next showing up...
> > >
> > > On 04/07/2019 11:59, Chesnay Schepler wrote:
> > > > Small update with mostly bad news:
> > > >
> > > > INFRA doesn't know whether it is possible, and referred my to Travis
> > > > support.
> > > > They did point out that it could be problematic in regards to
> > > > read/write permissions for the repository.
> > > >
> > > > From my own findings /so far/ with a test repo/organization, it does
> > > > not appear possible to configure the Travis account used for a
> > > > specific repository.
> > > >
> > > > So yeah, if we go down this route we may have to pimp the Flinkbot to
> > > > trigger builds through the Travis REST API.
> > > >
> > > > On 04/07/2019 10:46, Chesnay Schepler wrote:
> > > >> I've raised a JIRA
> > > >> with INFRA to
> > > >> inquire whether it would be possible to switch to a different Travis
> > > >> account, and if so what steps would need to be taken.
> > > >> We need a proper confirmation from INFRA since we are not in full
> > > >> control of the flink repository (for example, we cannot access the
> > > >> settings page).
> > > >>
> > > >> If this is indeed possible, Ververica is willing sponsor a Travis
> > > >> account for the Flink project.
> > > >> This would provide us with more than enough resources than we need.
> > > >>
> > > >> Since this makes the project more reliant on resources provided by
> > > >> external companies I would like to vote on this.
> > > >>
> > > >> Please vote on this proposal, as follows:
> > > >> [ ] +1, Approve the migration to a Ververica-sponsored Travis
> > > >> account, provided that INFRA approves
> > > >> [ ] -1, Do not approach the migration to a Ververica-sponsored
> Travis
> > > >> account
> > > >>
> > > >> The vote will be open for at least 24h, and until we have
> > > >> confirmation from INFRA. The voting period may be shorter than the
> > > >> usual 3 days since our current is effectively not working.
> > > >>
> > > >> On 04/07/2019 06:51, Bowen Li wrote:
> > > >>> Re: > Are they using their own Travis CI pool, or did the switch to
> > > >>> an entirely different CI service?
> > > >>>
> > > >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> > > >>> currently moving away from ASF's Travis to their own in-house metal
> > > >>> machines at [1] with custom CI application at [2]. They've seen
> > > >>> significant improvement w.r.t both much higher performance and
> > > >>> basically no resource waiting time, "night-and-day" difference
> > > >>> quoting Wes.
> > > >>>
> > > >>> Re: > If we can just switch to our own Travis pool, just for our
> > > >>> project, then this might be something we can do fairly quickly?
> > > >>>
> > > >>> I believe so, according to [3] and [4]
> > > >>>
> > > >>>
> > > >>> [1] https://ci.ursalabs.org/ 
> > > >>> [2] https://github.com/ursa-labs/ursabot
> > > >>> [3]
> > > >>>
> > >
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> > > >>>
> > > >>> [4]
> > > >>>
> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler <
> ches...@apache.org
> > > >>> > wrote:
> > > >>>
> > > >>> Are they using their own Travis CI pool, or did the switch to
> an
> > > >>> entirely different CI service?
> > > >>>
> > > >>> If we can just switch to our own Travis pool, just for our
> > > >>> project, then
> > > >>> this might be something we can do fairly quickly?
> > > >>>
> > > >>> On 03/07/2019 05:55, Bowen Li wrote:
> > > >>> > I responded in the INFRA ticket [1] that I believe they are
> > > >>> using a wrong
> > > >>> > metric against Flink and the total build time is a compl

Re: [VOTE] Migrate to sponsored Travis account

[JingsongLee]

+1 for the migration

Best, JingsongLee


--
From:Jark Wu 
Send Time:2019年7月5日(星期五) 10:35
To:dev 
Cc:private ; Bowen Li 
Subject:Re: [VOTE] Migrate to sponsored Travis account

+1 for the migration and great thanks to Chesnay and Bowen for pushing this!

Cheers,
Jark

On Fri, 5 Jul 2019 at 09:34, Congxian Qiu  wrote:

> +1 for the migration.
>
> Best,
> Congxian
>
>
> Hequn Cheng  于2019年7月4日周四 下午9:42写道：
>
> > +1.
> >
> > And thanks a lot to Chesnay for pushing this.
> >
> > Best, Hequn
> >
> > On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler 
> > wrote:
> >
> > > Note that the Flinkbot approach isn't that trivial either; we can't
> > > _just_ trigger builds for a branch in the apache repo, but would first
> > > have to clone the branch/pr into a separate repository (that is owned
> by
> > > the github account that the travis account would be tied to).
> > >
> > > One roadblock after the next showing up...
> > >
> > > On 04/07/2019 11:59, Chesnay Schepler wrote:
> > > > Small update with mostly bad news:
> > > >
> > > > INFRA doesn't know whether it is possible, and referred my to Travis
> > > > support.
> > > > They did point out that it could be problematic in regards to
> > > > read/write permissions for the repository.
> > > >
> > > > From my own findings /so far/ with a test repo/organization, it does
> > > > not appear possible to configure the Travis account used for a
> > > > specific repository.
> > > >
> > > > So yeah, if we go down this route we may have to pimp the Flinkbot to
> > > > trigger builds through the Travis REST API.
> > > >
> > > > On 04/07/2019 10:46, Chesnay Schepler wrote:
> > > >> I've raised a JIRA
> > > >> with INFRA to
> > > >> inquire whether it would be possible to switch to a different Travis
> > > >> account, and if so what steps would need to be taken.
> > > >> We need a proper confirmation from INFRA since we are not in full
> > > >> control of the flink repository (for example, we cannot access the
> > > >> settings page).
> > > >>
> > > >> If this is indeed possible, Ververica is willing sponsor a Travis
> > > >> account for the Flink project.
> > > >> This would provide us with more than enough resources than we need.
> > > >>
> > > >> Since this makes the project more reliant on resources provided by
> > > >> external companies I would like to vote on this.
> > > >>
> > > >> Please vote on this proposal, as follows:
> > > >> [ ] +1, Approve the migration to a Ververica-sponsored Travis
> > > >> account, provided that INFRA approves
> > > >> [ ] -1, Do not approach the migration to a Ververica-sponsored
> Travis
> > > >> account
> > > >>
> > > >> The vote will be open for at least 24h, and until we have
> > > >> confirmation from INFRA. The voting period may be shorter than the
> > > >> usual 3 days since our current is effectively not working.
> > > >>
> > > >> On 04/07/2019 06:51, Bowen Li wrote:
> > > >>> Re: > Are they using their own Travis CI pool, or did the switch to
> > > >>> an entirely different CI service?
> > > >>>
> > > >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> > > >>> currently moving away from ASF's Travis to their own in-house metal
> > > >>> machines at [1] with custom CI application at [2]. They've seen
> > > >>> significant improvement w.r.t both much higher performance and
> > > >>> basically no resource waiting time, "night-and-day" difference
> > > >>> quoting Wes.
> > > >>>
> > > >>> Re: > If we can just switch to our own Travis pool, just for our
> > > >>> project, then this might be something we can do fairly quickly?
> > > >>>
> > > >>> I believe so, according to [3] and [4]
> > > >>>
> > > >>>
> > > >>> [1] https://ci.ursalabs.org/ 
> > > >>> [2] https://github.com/ursa-labs/ursabot
> > > >>> [3]
> > > >>>
> > >
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> > > >>>
> > > >>> [4]
> > > >>>
> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler <
> ches...@apache.org
> > > >>> > wrote:
> > > >>>
> > > >>> Are they using their own Travis CI pool, or did the switch to
> an
> > > >>> entirely different CI service?
> > > >>>
> > > >>> If we can just switch to our own Travis pool, just for our
> > > >>> project, then
> > > >>> this might be something we can do fairly quickly?
> > > >>>
> > > >>> On 03/07/2019 05:55, Bowen Li wrote:
> > > >>> > I responded in the INFRA ticket [1] that I believe they are
> > > >>> using a wrong
> > > >>> > metric against Flink and the total build time is a completely
> > > >>> different
> > > >>> > thing than guaranteed build capacity.
> > > >>> >
> > > >>> > My response:
> > > >>> >
> > > >>> > "As mentioned above, since I sta

Re: [DISCUSS]Support Upsert mode for Streaming Non-window FlatAggregate

[Jark Wu]

Hi Hequn,

> If the TopN table aggregate function
> outputs three columns(rankid, time, value), either rankid or rankid+time
could be
> used as the key. Which one to be chosen is more likely to be decided by
the user
> according to his business.
In this case, the TopN table aggregate function should return two sets of
unique key, one is "rankid", another is "rankid, time".
This will be more align with current TopN node in blink planner and let
optimizer to decide which key based on the downstream information (column
selection, sink's primary key).


Best,
Jark

On Fri, 5 Jul 2019 at 00:05, Hequn Cheng  wrote:

> Hi Kurt and Jark,
>
> Thanks a lot for your great inputs!
>
> The keys of the query may not strongly be related to the UDTAGG.
> It may also be related to the corresponding scenarios that a user wants to
> achieve.
>
> For example, take TopN again as an example. If the TopN table aggregate
> function
> outputs three columns(rankid, time, value), either rankid or rankid+time
> could be
> used as the key. Which one to be chosen is more likely to be decided by
> the user
> according to his business.
>
> Best, Hequn
>
> On Thu, Jul 4, 2019 at 8:11 PM Jark Wu  wrote:
>
>> Hi jingcheng,
>>
>> I agree with Kurt's point. As you said "the user must know the keys of
>> the output of UDTAGG clearly".
>> If I understand correctly, the key information is strongly relative to
>> the UDTAGG implementation.
>> Users may call `flatAggregate` on a UDTAGG instance with different keys
>> which may result in a wrong result.
>> So I think it would be better to couple key information with UDTAGG
>> interface (i.e. "Approach 3" in your design doc).
>>
>> Regards,
>> Jark
>>
>> On Thu, 4 Jul 2019 at 18:06, Kurt Young  wrote:
>>
>>> Hi Jincheng,
>>>
>>> Thanks for the clarification. I think you just pointed out my concern by
>>> yourself:
>>>
>>> > When a user uses a User-defined table aggregate function (UDTAGG), he
>>> must understand the behavior of the UDTAGG, including the return type and
>>> the characteristics of the returned data. such as the key fields.
>>>
>>> This indicates that the UDTAGG is somehow be classified to different
>>> types,
>>> one will no key, one with key information. So the developer of the UDTAGG
>>> should choose which type of this function should be. In this case,
>>> my question would be, why don't we have explicit information about keys
>>> such as we split UDTAGG to keyed UDTAGG and non-keyed UDTAGG. So the user
>>> and the framework will have a better understanding of
>>> this UDTAGG. `withKeys` solution is letting user to choose the key and it
>>> seems it will only work correctly only if the user choose the *right* key
>>> this UDTAGG has.
>>>
>>> Let me know if this makes sense to you.
>>>
>>> Best,
>>> Kurt
>>>
>>>
>>> On Thu, Jul 4, 2019 at 4:32 PM jincheng sun 
>>> wrote:
>>>
>>> > Hi All,
>>> >
>>> > @Kurt Young  one user-defined table aggregate
>>> function
>>> > can be used in both with(out) keys case, and we do not introduce any
>>> other
>>> > aggregations. just like the explanation from @Hequn.
>>> >
>>> > @Hequn Cheng  thanks for your explanation!
>>> >
>>> > One thing should be mentioned here:
>>> >
>>> > When a user uses a User-defined table aggregate function (UDTAGG), he
>>> must
>>> > understand the behavior of the UDTAGG, including the return type and
>>> the
>>> > characteristics of the returned data. such as the key fields.  So
>>> although
>>> > using `withKeys` approach it is not rigorous enough(we do not need) but
>>> > intuitive enough, considering that if `flatAggregate` is followed by an
>>> > `upsertSink`, then the user must know the keys of the output of UDTAGG
>>> > clearly, otherwise the keys of `upsertSink` cannot be defined. So I
>>> still
>>> > prefer the `withKeys` solution by now.
>>> >
>>> > Looking forward to any feedback from all of you!
>>> >
>>> > Best,
>>> > Jincheng
>>> >
>>> >
>>> >
>>> > Hequn Cheng  于2019年7月1日周一 下午5:35写道：
>>> >
>>> >> Hi Kurt,
>>> >>
>>> >> Thanks for your questions. Here are my thoughts.
>>> >>
>>> >> > if I want to write such kind function, should I make sure that this
>>> >> function is used with some keys?
>>> >> The key information may not be used. We can also use RetractSink to
>>> emit
>>> >> the table directly.
>>> >>
>>> >> >  If I need a use case to calculate topn without key, should I write
>>> >> another function or I can reuse previous one.
>>> >> For the TopN example, you can reuse the previous function if you don't
>>> >> care
>>> >> about the key information.
>>> >>
>>> >> So, the key information is only an indicator(or a description), not an
>>> >> operator, as Jincheng mentioned above.
>>> >> We do not need to change the function logic and it will not add any
>>> other
>>> >> aggregations.
>>> >>
>>> >> BTW, we have three approaches in the document. Approach 1 defines
>>> keys on
>>> >> API level as we think it's more common to define keys on Table.
>>> >> While approach 3 defines keys in the TableAggregateFun

[jira] [Created] (FLINK-13106) Translate "parallel" page into Chinese

[yelun (JIRA)]

yelun created FLINK-13106:
-

 Summary: Translate "parallel" page into Chinese
 Key: FLINK-13106
 URL: https://issues.apache.org/jira/browse/FLINK-13106
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation, Documentation
Reporter: yelun


The page url is 
https://ci.apache.org/projects/flink/flink-docs-master/dev/parallel.html
The markdown file is located in flink/docs/dev/parallel.zh.md



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13107) Port TableApi IT and UT

[Jing Zhang (JIRA)]

Jing Zhang created FLINK-13107:
--

 Summary: Port TableApi IT and UT
 Key: FLINK-13107
 URL: https://issues.apache.org/jira/browse/FLINK-13107
 Project: Flink
  Issue Type: Task
  Components: Table SQL / Planner
Reporter: Jing Zhang
Assignee: Jing Zhang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Make window state queryable

[vino yang]

Hi all,

Thanks to Kostas for reminding me that as early as March 2017, the
community had a thread called "Future of Queryable State Feature". [1]

It has already discussed the queryable state and how to make the window
state queryable. I still think it can offer many advantages, especially for
Ad-Hoc.

Best,
Vino

[1]:
http://mail-archives.apache.org/mod_mbox/flink-dev/201703.mbox/%3C362C780C-9672-4DBD-B3F1-4EE7D1DB4CA6%40apache.org%3E

mayozhang <18124766...@163.com> 于2019年7月4日周四 下午10:21写道：

> It’s a good idea to get the process information of large ongoing window.
> +1 from my side.
>
> > 在 2019年7月4日，11:41，vino yang  写道：
> >
> > Hi folks,
> >
> > Currently, the queryable state is not widely used in production. IMO,
> there
> > are two key reasons caused this result. 1) the client of the queryable
> > state is hard to use. Because it requires users to know the address of
> > TaskManager and the port of the proxy. Actually, most business users who
> do
> > not have good knowledge about the Flink's inner and runtime in
> production.
> > 2) The benefit of this feature has not been excavated. In Flink
> DataStream
> > API, State is the first level citizen, it’s Flink key advantage compared
> > with other compute engines. Because the queryable state is the most
> > effective way to pry the latest computing progress.
> >
> > Three months ago, I started a discussion about improving the queryable
> > state and introducing a proxy component.[1] It brings a lot of attention
> > and discussion. Recently, I have submitted a design document about the
> > proposal.[2] These efforts try to process the first problem.
> >
> > About the second question, the most essential solution is that we should
> > really make the queryable state work. The window operator is one of the
> > most valuable and most frequently used operators of all Flink operators.
> > And it also uses keyed state which is queryable. So we propose to let the
> > state of the window operator be queried. This is not only for increasing
> > the value of the queryable state but also for the real business needs.
> >
> > IMO, allowing window state to be queried will provide great value. In
> many
> > scenarios, we often use large windows for aggregate calculations. A very
> > common example is a day-level window that counts the PV of a day. But
> > usually, the user is not only satisfied to wait until the end of the
> window
> > to get the result. They want to get "intermediate results" at a smaller
> > time granularity to analyze trends. Because Flink does not provide
> periodic
> > triggers for fixed windows. We have extended this and implemented an
> > "incremental window". It can trigger a fixed window with a smaller
> interval
> > period and feedback intermediate results. However, we believe that this
> > approach is still not flexible enough. We should let the user query the
> > current calculation result of the window through the API at any time.
> >
> > However, I know that if we want to implement it, we still have some
> details
> > that need to be discussed, such as how to let users know the state
> > descriptors in the window, namespace and so on.
> >
> > This discussion thread is mainly to listen to the community's opinion on
> > this proposal.
> >
> > Any feedback and ideas are welcome and appreciated.
> >
> > Best,
> > Vino
> >
> > [1]:
> >
> http://mail-archives.apache.org/mod_mbox/flink-dev/201907.mbox/%3ctencent_35a56d6858408be2e2064...@qq.com%3E
> > [2]:
> >
> https://docs.google.com/document/d/181qYVIiHQGrc3hCj3QBn1iEHF4bUztdw4XO8VSaf_uI/edit?usp=sharing
>
>
>

Re: [DISCUSS] FLIP-45: Reinforce Job Stop Semantic

[Congxian Qiu]

 Thanks for the effort, Yu, I've shared the link on the jira side[1] for
those who may be interested in this topic.

[1] https://issues.apache.org/jira/browse/FLINK-12619
Best,
Congxian


Yu Li  于2019年7月3日周三 下午10:56写道：

> Hi Flink devs,
>
> Through recent discussion on job stop processing, we found its semantic is
> incomplete, mainly reflected in two aspects:
>
> 1. In our released versions [1], the stop process is as follows:
>
>- A “stop” call is a graceful way of stopping a running streaming job.
>When the user requests to stop a job, all sources will receive a stop()
>method call. The job will keep running until all sources properly shut
>down. This allows the job to finish processing all inflight data.
>
> However, for stateful operators with retained checkpointing, the stop
> call won’t take any checkpoint, thus when resuming the job it needs to
> recover from the latest checkpoint with source rewinding, which causes the
> wait for processing all inflight data meaningless (all need to be processed
> again).
>
> 2. In the latest master branch after FLIP-34 [2], job stop will always be
> accompanied by a savepoint, which has below problems:
>
>- It's an unexpected behavior change from user perspective, that old
>stop command will fail on jobs w/o savepoint configuration.
>- It slows down the job stop procedure and might block launching new
>jobs when resource is contended.
>
> To resolve the above issues and reinforce job stop semantic, we have
> opened FLIP-45. In the document we carefully compared the Flink concepts
> with database, clarified what's missing in Flink job stop process, and
> proposed changes to complete it.
>
> Since this involves conception level clarification and user-facing
> changes, we would like to collect feedback on the proposal before
> continuing efforts.
>
> This is the FLIP:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-45%3A+Reinforce+Job+Stop+Semantic
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/cli.html
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212
>
> Best Regards,
> Yu
>

[jira] [Created] (FLINK-13108) Remove duplicated type cast in generated code

[Liya Fan (JIRA)]

Liya Fan created FLINK-13108:


 Summary: Remove duplicated type cast in generated code
 Key: FLINK-13108
 URL: https://issues.apache.org/jira/browse/FLINK-13108
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Liya Fan
Assignee: Liya Fan


There are duplicated cast operations in the generated code. For example, to run 
org.apache.flink.table.runtime.batch.sql.join.JoinITCase#testJoin, the 
generated code looks like this:

@Override
public void 
processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord 
element) throws Exception {
  org.apache.flink.table.dataformat.BaseRow in1 = 
(org.apache.flink.table.dataformat.BaseRow) 
(org.apache.flink.table.dataformat.BaseRow) 
converter$0.toInternal((org.apache.flink.types.Row) element.getValue());

This issue remove the duplicated type cast.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

flink jira contributor apply

[Nuist_Hook]

Hi,


I want to contribute to Apache Flink.
Would you please give me the contributor permission?
My JIRA ID is geyx881105