Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[Hequn Cheng]

+1 (binding)

Thanks,
Hequn

On Fri, Jan 12, 2024 at 2:19 PM godfrey he  wrote:

> +1 (binding)
>
> Thanks,
> Godfrey
>
> Zhu Zhu  于2024年1月12日周五 14:10写道：
> >
> > +1 (binding)
> >
> > Thanks,
> > Zhu
> >
> > Hangxiang Yu  于2024年1月11日周四 14:26写道：
> >
> > > +1 (non-binding)
> > >
> > > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Xuannan
> > > >
> > > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang  wrote:
> > > > >
> > > > > +1 (non-binding)--
> > > > >
> > > > > Best！
> > > > > Xuyang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 在 2024-01-11 10:00:11，"Yang Wang"  写道：
> > > > > >+1 (binding)
> > > > > >
> > > > > >
> > > > > >Best,
> > > > > >Yang
> > > > > >
> > > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron 
> wrote:
> > > > > >
> > > > > >> +1 non-binding
> > > > > >>
> > > > > >> Best
> > > > > >> Ron
> > > > > >>
> > > > > >> Matthias Pohl  于2024年1月10日周三
> > > 23:05写道：
> > > > > >>
> > > > > >> > +1 (binding)
> > > > > >> >
> > > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam <
> jam.gz...@gmail.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > > +1 non-binding
> > > > > >> > >
> > > > > >> > > Dawid Wysakowicz  于2024年1月10日周三
> > > 21:06写道：
> > > > > >> > >
> > > > > >> > > > +1 (binding)
> > > > > >> > > > Best,
> > > > > >> > > > Dawid
> > > > > >> > > >
> > > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski <
> > > > pnowoj...@apache.org>
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > > +1 (binding)
> > > > > >> > > > >
> > > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser <
> > > > martijnvis...@apache.org>
> > > > > >> > > > > napisał(a):
> > > > > >> > > > >
> > > > > >> > > > > > +1 (binding)
> > > > > >> > > > > >
> > > > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo Huang <
> > > > hxbks...@gmail.com
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > +1 (binding)
> > > > > >> > > > > > >
> > > > > >> > > > > > > Best,
> > > > > >> > > > > > > Xingbo
> > > > > >> > > > > > >
> > > > > >> > > > > > > Dian Fu  于2024年1月10日周三
> 11:35写道：
> > > > > >> > > > > > >
> > > > > >> > > > > > > > +1 (binding)
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Regards,
> > > > > >> > > > > > > > Dian
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Jan 10, 2024 at 5:09 AM Sharath <
> > > > > >> dsaishar...@gmail.com
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Best,
> > > > > >> > > > > > > > > Sharath
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Tue, Jan 9, 2024 at 1:02 PM Venkata Sanath
> > > > Muppalla <
> > > > > >> > > > > > > > sanath...@gmail.com>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > Sanath
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Tue, Jan 9, 2024 at 11:16 AM Peter Huang <
> > > > > >> > > > > > > > huangzhenqiu0...@gmail.com>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Best Regards
> > > > > >> > > > > > > > > > > Peter Huang
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Tue, Jan 9, 2024 at 5:26 AM Jane Chan <
> > > > > >> > > > > qingyue@gmail.com>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > Jane
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Tue, Jan 9, 2024 at 8:41 PM Lijie Wang
> <
> > > > > >> > > > > > > > wangdachui9...@gmail.com>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > Lijie
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Jiabao Sun  > > .invalid>
> > > > > >> > > > 于2024年1月9日周二
> > > > > >> > > > > > > > 19:28写道：
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > > Jiabao
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On 2024/01/09 09:58:04 xiangyu feng
> wrote:
> > > > > >> > > > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> 

Re: [VOTE] FLIP-139: General Python User-Defined Aggregate Function on Table API

[Hequn Cheng]

+1 (binding)

Best,
Hequn

On Thu, Sep 10, 2020 at 10:03 AM Dian Fu  wrote:

> +1(binding)
>
> Regards,
> Dian
>
> > 在 2020年9月8日，上午7:43，jincheng sun  写道：
> >
> > +1(binding)
> >
> > Best,
> > Jincheng
> >
> > Xingbo Huang  于2020年9月7日周一 下午5:45写道：
> >
> >> Hi,
> >>
> >> +1 (non-binding)
> >>
> >> Best,
> >> Xingbo
> >>
> >> Wei Zhong  于2020年9月7日周一 下午2:37写道：
> >>
> >>> Hi all,
> >>>
> >>> I would like to start the vote for FLIP-139[1] which is discussed and
> >>> reached consensus in the discussion thread[2].
> >>>
> >>> The vote will be open for at least 72 hours. I'll try to close it by
> >>> 2020-09-10 07:00 UTC, unless there is an objection or not enough votes.
> >>>
> >>> Best,
> >>> Wei
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-139%3A+General+Python+User-Defined+Aggregate+Function+Support+on+Table+API
> >>> [2]
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-139-General-Python-User-Defined-Aggregate-Function-on-Table-API-td44139.html
> >>>
> >>>
> >>
>
>

Re: [VOTE] FLIP-137: Support Pandas UDAF in PyFlink

[Hequn Cheng]

+1 (binding)


On Tue, Sep 8, 2020 at 7:43 AM jincheng sun 
wrote:

> +1(binding)
>
> Best,
> Jincheng
>
>
> Shuiqiang Chen  于2020年9月7日周一 下午3:50写道：
>
> > +1 (non-binding)
> >
> > > 在 2020年9月7日，下午2:38，Wei Zhong  写道：
> > >
> > > +1 (non-binding)
> > >
> > >> 在 2020年9月7日，10:00，Dian Fu  写道：
> > >>
> > >> +1
> > >>
> > >>> 在 2020年9月4日，上午11:12，Xingbo Huang  写道：
> > >>>
> > >>> Hi all,
> > >>> I would like to start the vote for FLIP-137[1], which is discussed
> and
> > >>> reached a consensus in the discussion thread[2].
> > >>>
> > >>> The vote will be open for at least 72h, unless there is an objection
> > or not
> > >>> enough votes.
> > >>>
> > >>> Best,
> > >>> Xingbo
> > >>>
> > >>> [1]
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-137%3A+Support+Pandas+UDAF+in+PyFlink
> > >>> [2]
> > >>>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-137-Support-Pandas-UDAF-in-PyFlink-tt44060.html
> > >>
> > >
> >
> >
>

[jira] [Created] (FLINK-18997) Rename type_info to result_type to make it more clear

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18997:
---

 Summary: Rename type_info to result_type to make it more clear
 Key: FLINK-18997
 URL: https://issues.apache.org/jira/browse/FLINK-18997
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18985) Update the Sphinx doc for Python DataStream API.

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18985:
---

 Summary: Update the Sphinx doc for Python DataStream API. 
 Key: FLINK-18985
 URL: https://issues.apache.org/jira/browse/FLINK-18985
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18984) Add tutorial documentation for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18984:
---

 Summary: Add tutorial documentation for Python DataStream API
 Key: FLINK-18984
 URL: https://issues.apache.org/jira/browse/FLINK-18984
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18966) Support key_by() on ConnectedStreams for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18966:
---

 Summary: Support key_by() on ConnectedStreams for Python 
DataStream API
 Key: FLINK-18966
 URL: https://issues.apache.org/jira/browse/FLINK-18966
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18954) Add documentation for Metrics in Python DataStream API.

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18954:
---

 Summary: Add documentation for Metrics in Python DataStream API.
 Key: FLINK-18954
 URL: https://issues.apache.org/jira/browse/FLINK-18954
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18953) Add documentation for DataTypes in Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18953:
---

 Summary: Add documentation for DataTypes in Python DataStream API
 Key: FLINK-18953
 URL: https://issues.apache.org/jira/browse/FLINK-18953
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18952) Add 10 minutes to DataStream API documentation

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18952:
---

 Summary: Add 10 minutes to DataStream API documentation
 Key: FLINK-18952
 URL: https://issues.apache.org/jira/browse/FLINK-18952
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18949) Support Streaming File Sink for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18949:
---

 Summary: Support Streaming File Sink for Python DataStream API
 Key: FLINK-18949
 URL: https://issues.apache.org/jira/browse/FLINK-18949
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18948) Add end to end test for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18948:
---

 Summary: Add end to end test for Python DataStream API
 Key: FLINK-18948
 URL: https://issues.apache.org/jira/browse/FLINK-18948
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18947) Support partitionCustom() operation for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18947:
---

 Summary: Support partitionCustom() operation for Python DataStream 
API
 Key: FLINK-18947
 URL: https://issues.apache.org/jira/browse/FLINK-18947
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18945) Support key_by() on ConnectedStreams for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18945:
---

 Summary: Support key_by() on ConnectedStreams for Python 
DataStream API
 Key: FLINK-18945
 URL: https://issues.apache.org/jira/browse/FLINK-18945
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18943) Support connect() operation for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18943:
---

 Summary: Support connect() operation for Python DataStream API
 Key: FLINK-18943
 URL: https://issues.apache.org/jira/browse/FLINK-18943
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-133: Rework PyFlink Documentation

[Hequn Cheng]

+1 (binding)

Best,
Hequn

On Wed, Aug 12, 2020 at 10:12 AM Wei Zhong  wrote:

> +1 (non-binding)
>
> Best,
> Wei
>
> > 在 2020年8月11日，22:41，Seth Wiesman  写道：
> >
> > +1 (binding)
> >
> > On Sun, Aug 9, 2020 at 8:15 PM jincheng sun 
> > wrote:
> >
> >> Hi Seth,
> >>
> >> Thank you for joining  the discussion and voting. Could you please have
> a
> >> look at the discussion thread.
> >> It would be great to know that did the follow-up discussion allay your
> >> concerns and any feedback is welcome.
> >>
> >> Best,
> >> Jincheng
> >>
> >>
> >>
> >> Seth Wiesman  于2020年8月3日周一 下午9:41写道：
> >>
> >>> -1
> >>>
> >>> I'm sorry to be late to the discussion but I have some concerns I've
> >>> brought up in the discussion thread.
> >>>
> >>> Seth
> >>>
> >>> On Mon, Aug 3, 2020 at 3:00 AM jincheng sun 
> >>> wrote:
> >>>
>  Hi everyone,
> 
>  I would like to start the vote for FLIP-133[1], which is discussed and
>  reached a consensus through the discussion thread[2].
> 
>  The vote will be open until 6th August (72h), unless there is an
> >>> objection
>  or not enough votes.
> 
>  Best,
>  Jincheng
> 
>  [1]:
> 
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-133%3A+Rework+PyFlink+Documentation
>  [2]:
> 
> 
> >>>
> >>
> https://lists.apache.org/thread.html/redebc9d1281edaa4a1fbf0d8c76a69fcff574b0496e78519840a5a61%40%3Cdev.flink.apache.org%3E
> 
> >>>
> >>
>
>

[jira] [Created] (FLINK-18883) Support reduce() operation for Python KeyedStream.

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18883:
---

 Summary: Support reduce() operation for Python KeyedStream.
 Key: FLINK-18883
 URL: https://issues.apache.org/jira/browse/FLINK-18883
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18817) 'Kerberized YARN per-job on Docker test' failed

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18817:
---

 Summary: 'Kerberized YARN per-job on Docker test' failed
 Key: FLINK-18817
 URL: https://issues.apache.org/jira/browse/FLINK-18817
 Project: Flink
  Issue Type: Test
  Components: Tests
Reporter: Hequn Cheng


The end-to-end test failed due to some AccessControlException:

https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/5169/logs/125

{code}
2020-08-04T13:13:10.2755424Z Failing this attempt.Diagnostics: Failed on local 
exception: java.io.IOException: 
org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
via:[TOKEN, KERBEROS]; Host Details : local host is: 
"worker1.docker-hadoop-cluster-network/172.19.0.5"; destination host is: 
"master.docker-hadoop-cluster-network":9000; 
2020-08-04T13:13:10.2757620Z java.io.IOException: Failed on local exception: 
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client 
cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: 
"worker1.docker-hadoop-cluster-network/172.19.0.5"; destination host is: 
"master.docker-hadoop-cluster-network":9000; 
2020-08-04T13:13:10.2758550Zat 
org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
2020-08-04T13:13:10.2758960Zat 
org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493)
2020-08-04T13:13:10.2759321Zat 
org.apache.hadoop.ipc.Client.call(Client.java:1435)
2020-08-04T13:13:10.2759676Zat 
org.apache.hadoop.ipc.Client.call(Client.java:1345)
2020-08-04T13:13:10.2760305Zat 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
2020-08-04T13:13:10.2760743Zat 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
2020-08-04T13:13:10.2761087Zat com.sun.proxy.$Proxy11.getFileInfo(Unknown 
Source)
2020-08-04T13:13:10.2761521Zat 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:796)
2020-08-04T13:13:10.2761964Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-08-04T13:13:10.2762310Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-08-04T13:13:10.2762741Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-08-04T13:13:10.2763105Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2020-08-04T13:13:10.2763503Zat 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
2020-08-04T13:13:10.2763979Zat 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
2020-08-04T13:13:10.2764474Zat 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
2020-08-04T13:13:10.2764944Zat 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
2020-08-04T13:13:10.2765417Zat 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
2020-08-04T13:13:10.2765770Zat com.sun.proxy.$Proxy12.getFileInfo(Unknown 
Source)
2020-08-04T13:13:10.2766093Zat 
org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1649)
2020-08-04T13:13:10.2766489Zat 
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1440)
2020-08-04T13:13:10.2767209Zat 
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1437)
2020-08-04T13:13:10.2767699Zat 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
2020-08-04T13:13:10.2768187Zat 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1437)
2020-08-04T13:13:10.2768646Zat 
org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
2020-08-04T13:13:10.2769051Zat 
org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
2020-08-04T13:13:10.2769470Zat 
org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
2020-08-04T13:13:10.2769988Zat 
org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
2020-08-04T13:13:10.2770438Zat 
java.security.AccessController.doPrivileged(Native Method)
2020-08-04T13:13:10.2770735Zat 
javax.security.auth.Subject.doAs(Subject.java:422)
2020-08-04T13:13:10.2771113Zat 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
2020-08-04T13:13:10.2771503Zat 
org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
2020-08-04T13:13:10.2771870Zat 
org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
2020-08-04T13:13:10.2772317Zat 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-08-04T13:13:10.2772732Zat 
java.util.concurrent.Executors

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

[Hequn Cheng]

Hi Jincheng,

Thanks a lot for raising the discussion. +1 for the FLIP.

I think this will bring big benefits for the PyFlink users. Currently, the
Python TableAPI document is hidden deeply under the TableAPI tab which
makes it quite unreadable. Also, the PyFlink documentation is mixed with
Java/Scala documentation. It is hard for users to have an overview of all
the PyFlink documents. As more and more functionalities are added into
PyFlink, I think it's time for us to refactor the document.

Best,
Hequn


On Fri, Jul 31, 2020 at 3:43 PM Marta Paes Moreira 
wrote:

> Hi, Jincheng!
>
> Thanks for creating this detailed FLIP, it will make a big difference in
> the experience of Python developers using Flink. I'm interested in
> contributing to this work, so I'll reach out to you offline!
>
> Also, thanks for sharing some information on the adoption of PyFlink, it's
> great to see that there are already production users.
>
> Marta
>
> On Fri, Jul 31, 2020 at 5:35 AM Xingbo Huang  wrote:
>
> > Hi Jincheng,
> >
> > Thanks a lot for bringing up this discussion and the proposal.
> >
> > Big +1 for improving the structure of PyFlink doc.
> >
> > It will be very friendly to give PyFlink users a unified entrance to
> learn
> > PyFlink documents.
> >
> > Best,
> > Xingbo
> >
> > Dian Fu  于2020年7月31日周五 上午11:00写道：
> >
> >> Hi Jincheng,
> >>
> >> Thanks a lot for bringing up this discussion and the proposal. +1 to
> >> improve the Python API doc.
> >>
> >> I have received many feedbacks from PyFlink beginners about
> >> the PyFlink doc, e.g. the materials are too few, the Python doc is mixed
> >> with the Java doc and it's not easy to find the docs he wants to know.
> >>
> >> I think it would greatly improve the user experience if we can have one
> >> place which includes most knowledges PyFlink users should know.
> >>
> >> Regards,
> >> Dian
> >>
> >> 在 2020年7月31日，上午10:14，jincheng sun  写道：
> >>
> >> Hi folks,
> >>
> >> Since the release of Flink 1.11, users of PyFlink have continued to
> grow.
> >> As far as I know there are many companies have used PyFlink for data
> >> analysis, operation and maintenance monitoring business has been put
> into
> >> production(Such as 聚美优品[1](Jumei),  浙江墨芷[2] (Mozhi) etc.).  According to
> >> the feedback we received, current documentation is not very friendly to
> >> PyFlink users. There are two shortcomings:
> >>
> >> - Python related content is mixed in the Java/Scala documentation, which
> >> makes it difficult for users who only focus on PyFlink to read.
> >> - There is already a "Python Table API" section in the Table API
> document
> >> to store PyFlink documents, but the number of articles is small and the
> >> content is fragmented. It is difficult for beginners to learn from it.
> >>
> >> In addition, FLIP-130 introduced the Python DataStream API. Many
> >> documents will be added for those new APIs. In order to increase the
> >> readability and maintainability of the PyFlink document, Wei Zhong and
> me
> >> have discussed offline and would like to rework it via this FLIP.
> >>
> >> We will rework the document around the following three objectives:
> >>
> >> - Add a separate section for Python API under the "Application
> >> Development" section.
> >> - Restructure current Python documentation to a brand new structure to
> >> ensure complete content and friendly to beginners.
> >> - Improve the documents shared by Python/Java/Scala to make it more
> >> friendly to Python users and without affecting Java/Scala users.
> >>
> >> More detail can be found in the FLIP-133:
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-133%3A+Rework+PyFlink+Documentation
> >>
> >> Best,
> >> Jincheng
> >>
> >> [1] https://mp.weixin.qq.com/s/zVsBIs1ZEFe4atYUYtZpRg
> >> [2] https://mp.weixin.qq.com/s/R4p_a2TWGpESBWr3pLtM2g
> >>
> >>
> >>
>

[jira] [Created] (FLINK-18766) Support add_sink() for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18766:
---

 Summary: Support add_sink() for Python DataStream API
 Key: FLINK-18766
 URL: https://issues.apache.org/jira/browse/FLINK-18766
 Project: Flink
  Issue Type: Sub-task
Reporter: Hequn Cheng






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18765) Support map() and flat_map() for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18765:
---

 Summary: Support map() and flat_map() for Python DataStream API
 Key: FLINK-18765
 URL: https://issues.apache.org/jira/browse/FLINK-18765
 Project: Flink
  Issue Type: Sub-task
Reporter: Hequn Cheng






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18764) Support from_collection for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18764:
---

 Summary: Support from_collection for Python DataStream API
 Key: FLINK-18764
 URL: https://issues.apache.org/jira/browse/FLINK-18764
 Project: Flink
  Issue Type: Sub-task
Reporter: Hequn Cheng






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18763) Support basic TypeInformation for Python DataStream API

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18763:
---

 Summary: Support basic TypeInformation for Python DataStream API
 Key: FLINK-18763
 URL: https://issues.apache.org/jira/browse/FLINK-18763
 Project: Flink
  Issue Type: Sub-task
Reporter: Hequn Cheng


Supports basic TypeInformation including BasicTypeInfo, LocalTimeTypeInfo, 
PrimitiveArrayTypeInfo, RowTypeInfo. 

Types.ROW()/Types.ROW_NAMED()/Types.PRIMITIVE_ARRAY() should also be supported.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18761) Support Python DataStream API (Stateless part)

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18761:
---

 Summary: Support Python DataStream API (Stateless part)
 Key: FLINK-18761
 URL: https://issues.apache.org/jira/browse/FLINK-18761
 Project: Flink
  Issue Type: New Feature
  Components: API / DataStream, API / Python
Reporter: Hequn Cheng


This is the umbrella Jira for FLIP-130, which intends to support Python 
DataStream API for the stateless part.

FLIP wiki page: 
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866298]

As we all know, Flink provides [three layered 
APIs|https://flink.apache.org/flink-applications.html#layered-apis]: the 
ProcessFunctions, the DataStream API and the SQL & Table API. Each API offers a 
different trade-off between conciseness and expressiveness and targets 
different use cases.

Currently, the SQL & Table API has already been supported in PyFlink. The API 
provides relational operations as well as user-defined functions to provide 
convenience for users who are familiar with python and relational programming.

Meanwhile, the DataStream API and ProcessFunctions provide more generic APIs to 
implement stream processing applications. The ProcessFunctions expose time and 
state which are the fundamental building blocks for any kind of streaming 
application. To cover more use cases, we are planning to cover all these APIs 
in PyFlink.

In this FLIP, we propose to support the Python DataStream API for the stateless 
part. For more detail, please refer to the [FLIP wiki 
page|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866298].
 As for the stateful part, it would come later after this FLIP. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-130: Support for Python DataStream API (Stateless Part)

[Hequn Cheng]

+1 (binding)

Best,
Hequn

On Tue, Jul 28, 2020 at 2:58 PM Aljoscha Krettek 
wrote:

> +1 (binding)
>
> Aljoscha
>
> On 28.07.20 04:12, Dian Fu wrote:
> > Thanks for driving this Shuiqiang.
> >
> > +1
> >
> > Regards,
> > Dian
> >
> >> 在 2020年7月27日，下午3:33，jincheng sun  写道：
> >>
> >> +1(binding)
> >>
> >> Best,
> >> Jincheng
> >>
> >> Shuiqiang Chen  于2020年7月24日周五 下午8:32写道：
> >>
> >>> Hi everyone,
> >>>
> >>> I would like to start the vote for FLIP-130[1], which is discussed and
> >>> reached a consensus through the discussion thread[2].
> >>>
> >>> The vote will be open until 29th July (72h), unless there is an
> objection
> >>> or not enough votes.
> >>>
> >>> Best,
> >>> Shuiqiang
> >>>
> >>> [1]:
> >>>
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866298
> >>> [2]:
> >>>
> >>>
> >>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-130-Support-for-Python-DataStream-API-Stateless-Part-td43035.html
> >>>
> >
>
>

Re: [DISCUSS] FLIP-130: Support for Python DataStream API (Stateless Part)

[Hequn Cheng]

Hi,

It's a good idea to start with a minimum size of API and add useful ones
when we find it is truly useful.
>From my side, I'm also ok with the partitionCustom() method. Thanks David
for your feedback!

Best,
Hequn

On Mon, Jul 27, 2020 at 8:57 PM Aljoscha Krettek 
wrote:

> Hi,
>
> I'm also not against adding that if it enables actual use cases. I don't
> think we need to spell out the whole API in the FLIP, though. We can add
> things as they come up.
>
> Best,
> Aljoscha
>
> On 24.07.20 14:43, Shuiqiang Chen wrote:
> > Hi David,
> >
> > Thank you for your reply! I have started the vote for this FLIP, but we
> can
> > keep the discussion on this thread.
> > In my perspective, I would not against adding the
> > DataStream.partitionCustom to Python DataStream API.  However, more
> inputs
> > are welcomed.
> >
> > Best,
> > Shuiqiang
> >
> >
> >
> > David Anderson  于2020年7月24日周五 下午7:52写道：
> >
> >> Sorry I'm coming to this rather late, but I would like to argue that
> >> DataStream.partitionCustom enables an important use case.
> >> What I have in mind is performing partitioned enrichment, where each
> >> instance can preload a slice of a static dataset that is being used for
> >> enrichment.
> >>
> >> For an example, consider
> >>
> https://github.com/knaufk/enrichments-with-flink/blob/master/src/main/java/com/github/knaufk/enrichments/CustomPartitionEnrichmenttJob.java
> >> .
> >>
> >> Regards,
> >> David
> >>
> >> On Fri, Jul 24, 2020 at 12:18 PM Shuiqiang Chen 
> >> wrote:
> >>
> >>> Hi Aljoscha, Thank you for your response.  I'll keep these two helper
> >>> methods in the Python DataStream implementation.
> >>>
> >>> And thank you all for joining in the discussion. It seems that we have
> >>> reached a consensus. I will start a vote for this FLIP later today.
> >>>
> >>> Best,
> >>> Shuiqiang
> >>>
> >>> Hequn Cheng  于2020年7月24日周五 下午5:29写道：
> >>>
> >>>> Thanks a lot for your valuable feedback and suggestions! @Aljoscha
> >>> Krettek
> >>>> 
> >>>> +1 to the vote.
> >>>>
> >>>> Best,
> >>>> Hequn
> >>>>
> >>>> On Fri, Jul 24, 2020 at 5:16 PM Aljoscha Krettek  >
> >>>> wrote:
> >>>>
> >>>>> Thanks for updating! And yes, I think it's ok to include the few
> >>> helper
> >>>>> methods such as "readFromFile" and "print".
> >>>>>
> >>>>> I think we can now proceed to a vote! Nice work, overall!
> >>>>>
> >>>>> Best,
> >>>>> Aljoscha
> >>>>>
> >>>>> On 16.07.20 17:16, Hequn Cheng wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Thanks a lot for your discussions.
> >>>>>> I think Aljoscha makes good suggestions here! Those problematic APIs
> >>>>> should
> >>>>>> not be added to the new Python DataStream API.
> >>>>>>
> >>>>>> Only one item I want to add based on the reply from Shuiqiang:
> >>>>>> I would also tend to keep the readTextFile() method. Apart from
> >>>> print(),
> >>>>>> the readTextFile() may also be very helpful and frequently used for
> >>>>> playing
> >>>>>> with Flink.
> >>>>>> For example, it is used in our WordCount example[1] which is almost
> >>> the
> >>>>>> first Flink program that every beginner runs.
> >>>>>> It is more efficient for reading multi-line data compared to
> >>>>>> fromCollection() meanwhile far more easier to be used compared to
> >>>> Kafka,
> >>>>>> Kinesis, RabbitMQ,etc., in
> >>>>>> cases for playing with Flink.
> >>>>>>
> >>>>>> What do you think?
> >>>>>>
> >>>>>> Best,
> >>>>>> Hequn
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java
&

Re: [DISCUSS] FLIP-130: Support for Python DataStream API (Stateless Part)

[Hequn Cheng]

Thanks a lot for your valuable feedback and suggestions! @Aljoscha Krettek

+1 to the vote.

Best,
Hequn

On Fri, Jul 24, 2020 at 5:16 PM Aljoscha Krettek 
wrote:

> Thanks for updating! And yes, I think it's ok to include the few helper
> methods such as "readFromFile" and "print".
>
> I think we can now proceed to a vote! Nice work, overall!
>
> Best,
> Aljoscha
>
> On 16.07.20 17:16, Hequn Cheng wrote:
> > Hi,
> >
> > Thanks a lot for your discussions.
> > I think Aljoscha makes good suggestions here! Those problematic APIs
> should
> > not be added to the new Python DataStream API.
> >
> > Only one item I want to add based on the reply from Shuiqiang:
> > I would also tend to keep the readTextFile() method. Apart from print(),
> > the readTextFile() may also be very helpful and frequently used for
> playing
> > with Flink.
> > For example, it is used in our WordCount example[1] which is almost the
> > first Flink program that every beginner runs.
> > It is more efficient for reading multi-line data compared to
> > fromCollection() meanwhile far more easier to be used compared to Kafka,
> > Kinesis, RabbitMQ,etc., in
> > cases for playing with Flink.
> >
> > What do you think?
> >
> > Best,
> > Hequn
> >
> > [1]
> >
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java
> >
> >
> > On Thu, Jul 16, 2020 at 3:37 PM Shuiqiang Chen 
> wrote:
> >
> >> Hi Aljoscha,
> >>
> >> Thank you for your valuable comments! I agree with you that there is
> some
> >> optimization space for existing API and can be applied to the python
> >> DataStream API implementation.
> >>
> >> According to your comments, I have concluded them into the following
> parts:
> >>
> >> 1. SingleOutputStreamOperator and DataStreamSource.
> >> Yes, the SingleOutputStreamOperator and DataStreamSource are a bit
> >> redundant, so we can unify their APIs into DataStream to make it more
> >> clear.
> >>
> >> 2. The internal or low-level methods.
> >>   - DataStream.get_id(): Has been removed in the FLIP wiki page.
> >>   - DataStream.partition_custom(): Has been removed in the FLIP wiki
> page.
> >>   - SingleOutputStreamOperator.can_be_parallel/forceNoParallel: Has been
> >> removed in the FLIP wiki page.
> >> Sorry for mistakenly making those internal methods public, we would not
> >> expose them to users in the Python API.
> >>
> >> 3. "declarative" Apis.
> >> - KeyedStream.sum/min/max/min_by/max_by: Has been removed in the FLIP
> wiki
> >> page. They could be well covered by Table API.
> >>
> >> 4. Spelling problems.
> >> - StreamExecutionEnvironment.from_collections. Should be
> from_collection().
> >> - StreamExecutionEnvironment.generate_sequenece. Should be
> >> generate_sequence().
> >> Sorry for the spelling error.
> >>
> >> 5. Predefined source and sink.
> >> As you said, most of the predefined sources are not suitable for
> >> production, we can ignore them in the new Python DataStream API.
> >> There is one exception that maybe I think we should add the print()
> since
> >> it is commonly used by users and it is very useful for debugging jobs.
> We
> >> can add comments for the API that it should never be used for
> production.
> >> Meanwhile, as you mentioned, a good alternative that always prints on
> the
> >> client should also be supported. For this case, maybe we can add the
> >> collect method and return an Iterator. With the iterator, uses can print
> >> the content on the client. This is also consistent with the behavior in
> >> Table API.
> >>
> >> 6. For Row.
> >> Do you mean that we should not expose the Row type in Python API? Maybe
> I
> >> haven't gotten your concerns well.
> >> We can use tuple type in Python DataStream to support Row. (I have
> updated
> >> the example section of the FLIP to reflect the design.)
> >>
> >> Highly appreciated for your suggestions again. Looking forward to your
> >> feedback.
> >>
> >> Best,
> >> Shuiqiang
> >>
> >> Aljoscha Krettek  于2020年7月15日周三 下午5:58写道：
> >>
> >>> Hi,
> >>>
> >>> thanks for the proposal! I have some comments about the API. We should
> >> not

Re: [ANNOUNCE] Apache Flink 1.11.1 released

[Hequn Cheng]

Thanks Dian for the great work and thanks to everyone who makes this
release possible!

Best, Hequn

On Wed, Jul 22, 2020 at 4:40 PM Jark Wu  wrote:

> Congratulations! Thanks Dian for the great work and to be the release
> manager!
>
> Best,
> Jark
>
> On Wed, 22 Jul 2020 at 15:45, Yangze Guo  wrote:
>
> > Congrats!
> >
> > Thanks Dian Fu for being release manager, and everyone involved!
> >
> > Best,
> > Yangze Guo
> >
> > On Wed, Jul 22, 2020 at 3:14 PM Wei Zhong 
> wrote:
> > >
> > > Congratulations! Thanks Dian for the great work!
> > >
> > > Best,
> > > Wei
> > >
> > > > 在 2020年7月22日，15:09，Leonard Xu  写道：
> > > >
> > > > Congratulations!
> > > >
> > > > Thanks Dian Fu for the great work as release manager, and thanks
> > everyone involved!
> > > >
> > > > Best
> > > > Leonard Xu
> > > >
> > > >> 在 2020年7月22日，14:52，Dian Fu  写道：
> > > >>
> > > >> The Apache Flink community is very happy to announce the release of
> > Apache Flink 1.11.1, which is the first bugfix release for the Apache
> Flink
> > 1.11 series.
> > > >>
> > > >> Apache Flink® is an open-source stream processing framework for
> > distributed, high-performing, always-available, and accurate data
> streaming
> > applications.
> > > >>
> > > >> The release is available for download at:
> > > >> https://flink.apache.org/downloads.html
> > > >>
> > > >> Please check out the release blog post for an overview of the
> > improvements for this bugfix release:
> > > >> https://flink.apache.org/news/2020/07/21/release-1.11.1.html
> > > >>
> > > >> The full release notes are available in Jira:
> > > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348323
> > > >>
> > > >> We would like to thank all contributors of the Apache Flink
> community
> > who made this release possible!
> > > >>
> > > >> Regards,
> > > >> Dian
> > > >
> > >
> >
>

[jira] [Created] (FLINK-18655) Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module flink-runtime

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18655:
---

 Summary: Set failOnUnableToExtractRepoInfo to false for 
git-commit-id-plugin in module flink-runtime
 Key: FLINK-18655
 URL: https://issues.apache.org/jira/browse/FLINK-18655
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Configuration
Affects Versions: 1.11.0
Reporter: Hequn Cheng
Assignee: Hequn Cheng


Exception may be thrown when building source distribution without the .git 
folder(for the flink-runtime module):
{code:java}
[ERROR] Failed to execute goal 
pl.project13.maven:git-commit-id-plugin:4.0.0:revision (get-the-git-infos) on 
project flink-runtime_2.11: Could not complete Mojo execution... Error: Could 
not get HEAD Ref, are you sure you have set the dotGitDirectory property of 
this plugin to a valid path? -> [Help 1]
{code}
 

To solve the problem, we need to add the 
{{false}} 
configuration in addition to 
{{false}} in the pom of 
flink-runtime.

 

The reason is the plugin:git-commit-id-plugin would search up all the maven 
parent project hierarchy until a .git directory is found. For example, if we 
config dotGitDirectory as /a/b/c/.git and if /a/b/c/.git is invalid, the plugin 
would search /a/b/.git and /a/.git. However, once the plugin found a /a/.git 
folder, it would fail on extracting repo info which leads to the failure above. 
The search logic of the plugin can be found 
[here|https://github.com/git-commit-id/git-commit-id-maven-plugin/blob/v4.0.0/maven/src/main/java/pl/project13/maven/git/GitDirLocator.java#L74].


You can reproduce the exception with:
- download the 1.11.0 source distribution.
- put a .git folder under the path (or parent path) of 
${project.basedir}/../.git. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-130: Support for Python DataStream API (Stateless Part)

[Hequn Cheng]

erant and shouldn't be used in production
> >
> >  - KeyedStream.sum/min/max/min_by/max_by: Nowadays, the Table API should
> > be used for "declarative" use cases and I think these methods should not
> be
> > in the DataStream API
> >  - SingleOutputStreamOperator.can_be_parallel/forceNoParallel: these are
> > internal methods
> >
> >  - StreamExecutionEnvironment.from_parallel_collection(): I think the
> > usability is questionable
> >  - StreamExecutionEnvironment.from_collections -> should be called
> > from_collection
> >  - StreamExecutionEnvironment.generate_sequenece -> should be called
> > generate_sequence
> >
> > I think most of the predefined sources are questionable:
> >  - fromParallelCollection: I don't know if this is useful
> >  - readTextFile: most of the variants are not useful/fault-tolerant
> >  - readFile: same
> >  - socketTextStream: also not useful except for toy examples
> >  - createInput: also not useful, and it's legacy DataSet InputFormats
> >
> > I think we need to think hard whether we want to further expose Row in
> our
> > APIs. I think adding it to flink-core was more an accident than anything
> > else but I can see that it would be useful for Python/Java interop.
> >
> > Best,
> > Aljoscha
> >
> >
> > On Mon, Jul 13, 2020, at 04:38, jincheng sun wrote:
> > > Thanks for bring up this DISCUSS Shuiqiang!
> > >
> > > +1 for the proposal!
> > >
> > > Best,
> > > Jincheng
> > >
> > >
> > > Xingbo Huang  于2020年7月9日周四 上午10:41写道：
> > >
> > > > Hi Shuiqiang,
> > > >
> > > > Thanks a lot for driving this discussion.
> > > > Big +1 for supporting Python DataStream.
> > > > In many ML scenarios, operating Object will be more natural than
> > operating
> > > > Table.
> > > >
> > > > Best,
> > > > Xingbo
> > > >
> > > > Wei Zhong  于2020年7月9日周四 上午10:35写道：
> > > >
> > > > > Hi Shuiqiang,
> > > > >
> > > > > Thanks for driving this. Big +1 for supporting DataStream API in
> > PyFlink!
> > > > >
> > > > > Best,
> > > > > Wei
> > > > >
> > > > >
> > > > > > 在 2020年7月9日，10:29，Hequn Cheng  写道：
> > > > > >
> > > > > > +1 for adding the Python DataStream API and starting with the
> > stateless
> > > > > > part.
> > > > > > There are already some users that expressed their wish to have
> the
> > > > Python
> > > > > > DataStream APIs. Once we have the APIs in PyFlink, we can cover
> > more
> > > > use
> > > > > > cases for our users.
> > > > > >
> > > > > > Best, Hequn
> > > > > >
> > > > > > On Wed, Jul 8, 2020 at 11:45 AM Shuiqiang Chen <
> > acqua@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Sorry, the 3rd link is broken, please refer to this one: Support
> > > > Python
> > > > > >> DataStream API
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1H3hz8wuk22-8cDBhQmQKNw3m1q5gDAMkwTDEwnj3FBI/edit
> > > > > >>>
> > > > > >>
> > > > > >> Shuiqiang Chen  于2020年7月8日周三 上午11:13写道：
> > > > > >>
> > > > > >>> Hi everyone,
> > > > > >>>
> > > > > >>> As we all know, Flink provides three layered APIs: the
> > > > > ProcessFunctions,
> > > > > >>> the DataStream API and the SQL & Table API. Each API offers a
> > > > different
> > > > > >>> trade-off between conciseness and expressiveness and targets
> > > > different
> > > > > >> use
> > > > > >>> cases[1].
> > > > > >>>
> > > > > >>> Currently, the SQL & Table API has already been supported in
> > PyFlink.
> > > > > The
> > > > > >>> API provides relational operations as well as user-defined
> > functions
> > > > to
> > > > > >>> provide convenience for users who are familiar with python and
> > > > > relational
> > > > > >>> programming.
> > > > > >>>
> > > > > >>> Meanwhile, the DataStream API and ProcessFunctions provide more
> > > > generic
> > > > > >>> APIs to implement stream processing applications. The
> > > > ProcessFunctions
> > > > > >>> expose time and state which are the fundamental building blocks
> > for
> > > > any
> > > > > >>> kind of streaming application.
> > > > > >>> To cover more use cases, we are planning to cover all these
> APIs
> > in
> > > > > >>> PyFlink.
> > > > > >>>
> > > > > >>> In this discussion(FLIP-130), we propose to support the Python
> > > > > DataStream
> > > > > >>> API for the stateless part. For more detail, please refer to
> the
> > FLIP
> > > > > >> wiki
> > > > > >>> page here[2]. If interested in the stateful part, you can also
> > take a
> > > > > >>> look the design doc here[3] for which we are going to discuss
> in
> > a
> > > > > >> separate
> > > > > >>> FLIP.
> > > > > >>>
> > > > > >>> Any comments will be highly appreciated!
> > > > > >>>
> > > > > >>> [1]
> > https://flink.apache.org/flink-applications.html#layered-apis
> > > > > >>> [2]
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866298
> > > > > >>> [3]
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1H3hz8wuk228cDBhQmQKNw3m1q5gDAMkwTDEwnj3FBI/edit?usp=sharing
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Shuiqiang
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.11.1 soon?

[Hequn Cheng]

+1 for a quick bug fix release and Dian as the release manager.

Best,
Hequn


On Thu, Jul 9, 2020 at 9:22 PM Dian Fu  wrote:

> Hi Jark,
>
> Thanks for offering the help. It would definitely be helpful.
>
> Regards,
> Dian
>
> > 在 2020年7月9日，下午8:54，Benchao Li  写道：
> >
> > +1 for a quick bug fix release for 1.11
> >
> > Aljoscha Krettek  于2020年7月9日周四 下午8:11写道：
> >
> >> +1
> >>
> >> I'd also be in favour of releasing a 1.11.1 quickly
> >>
> >> Aljoscha
> >>
> >> On 09.07.20 13:57, Jark Wu wrote:
> >>> Hi Dian,
> >>>
> >>> Glad to hear that you want to be the release manager of Flink 1.11.1.
> >>> I am very willing to help you with the final steps of the release
> >> process.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>> On Thu, 9 Jul 2020 at 17:57, Jingsong Li 
> wrote:
> >>>
>  FLINK-18461 is really a blocker for the CDC feature.
> 
>  So +1 for releasing Flink 1.11.1 soon.
> 
>  Best,
>  Jingsong
> 
>  On Thu, Jul 9, 2020 at 5:34 PM jincheng sun  >
>  wrote:
> 
> > Thanks for bring up this discussion Jark.
> > +1, looking forward the first bugfix version of Flink 1.11.
> >
> > Best,
> > Jincheng
> >
> > Dian Fu  于2020年7月9日周四 下午5:28写道：
> >
> >> Thanks Jark for bringing up this discussion. I also noticed that
> there
> > are
> >> already users trying out the CDC feature and so it makes sense to
> >> have a
> >> quick 1.11.1 release.
> >>
> >> I would volunteer as the release manager of 1.11.1 if we finally
> >> decide
> > to
> >> have a quick release. Also +1 to create the first RC on next Monday.
> >>
> >> Thanks,
> >> Dian
> >>
> >>> 在 2020年7月9日，下午3:55，Dawid Wysakowicz  写道：
> >>>
> >>> I do agree it would be beneficial to have the 1.11.1 rather soon.
> >>>
> >>> Personally additionally to Jark's list I'd like to see:
> >>>
> >>> FLINK-18419  Can not create a catalog from user jar
> >>> (https://issues.apache.org/jira/browse/FLINK-18419)
> >>>
> >>> incluedd. It has a PR already.
> >>>
> >>> Best,
> >>>
> >>> Dawid
> >>>
> >>> On 09/07/2020 09:51, Leonard Xu wrote:
>  Hi, Jark
> 
>  Thanks fro bringing this discussion.
> 
>  We canceled the blocker priority of FLINK-18461 when release Flink
> >> 1.11.0 from time consideration, IIRC, the majority of us agreed to
> >> have
> > a
> >> timely 1.11.1 in the vote mail of Flink 1.11.0.
>  And I have noticed some user voice[1][2] that they’re using CDC
> > feature
> >> in mail list and offline communication with community user, I think
> >> the
> >> following notable issues should be included in 1.11.1:
> 
>  - FLINK-18461 Changelog source can't be insert into upsert sink
>  - FLINK-18426 Incompatible deprecated key type for registration
> > cluster
>  - FLINK-18434 Can not select fields with JdbcCatalog
>  - FLINK-17726 Scheduler should take care of tasks directly
> canceled
> > by
> >> TaskManager
>  - FLINK-17075 Add task status reconciliation between TM and JM
> 
>  So,  +1 to prepare a quick bug-fix version from my side.
> 
> 
>  Best,
>  Leonard Xu
>  [1] user:
> >>
> >
> >>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flip-105-can-the-debezium-canal-SQL-sink-to-database-directly-td36275.html#a36306
> >> <
> >>
> >
> >>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flip-105-can-the-debezium-canal-SQL-sink-to-database-directly-td36275.html#a36306
> >>>
>  [2] user-zh:
> >>
> >
> >>
> http://apache-flink.147419.n8.nabble.com/flink-1-11-cdc-flink-sql-kafka-topic-tt4567.html#a4590
> >> <
> >>
> >
> >>
> http://apache-flink.147419.n8.nabble.com/flink-1-11-cdc-flink-sql-kafka-topic-tt4567.html#a4590
> >>>
> 
> > 在 2020年7月9日，11:12，Jark Wu  写道：
> >
> > Besides, it would be great if we can figure out the performance
> >> regression
> > Thomas reported before.
> >
> > Do you know what's the status now? @zhijiang <
> >> wangzhijiang...@aliyun.com>
> > @Thomas
> >
> > Best,
> > Jark
> >
> > On Thu, 9 Jul 2020 at 11:10, Jark Wu  wrote:
> >
> >> Hi everyone,
> >>
> >> As discussed in the voting thread of 1.11.0-RC4 [1], we found a
> >> blocker
> >> issue about the CDC feature [1].
> >> Considering this is a new kind of connector, we don't want to
> >> block
> >> the
> >> ready-to-publish RC4 and prefer to have an immediately 1.11.1
> > release.
> >> Therefore, I would like to start the discussion about releasing
> > 1.11.1
> >> soon, to deliver a complete CDC feature.
> >> We can also release some notable 

Re: [DISCUSS] FLIP-130: Support for Python DataStream API (Stateless Part)

[Hequn Cheng]

+1 for adding the Python DataStream API and starting with the stateless
part.
There are already some users that expressed their wish to have the Python
DataStream APIs. Once we have the APIs in PyFlink, we can cover more use
cases for our users.

Best, Hequn

On Wed, Jul 8, 2020 at 11:45 AM Shuiqiang Chen  wrote:

> Sorry, the 3rd link is broken, please refer to this one: Support Python
> DataStream API
> <
> https://docs.google.com/document/d/1H3hz8wuk22-8cDBhQmQKNw3m1q5gDAMkwTDEwnj3FBI/edit
> >
>
> Shuiqiang Chen  于2020年7月8日周三 上午11:13写道：
>
> > Hi everyone,
> >
> > As we all know, Flink provides three layered APIs: the ProcessFunctions,
> > the DataStream API and the SQL & Table API. Each API offers a different
> > trade-off between conciseness and expressiveness and targets different
> use
> > cases[1].
> >
> > Currently, the SQL & Table API has already been supported in PyFlink. The
> > API provides relational operations as well as user-defined functions to
> > provide convenience for users who are familiar with python and relational
> > programming.
> >
> > Meanwhile, the DataStream API and ProcessFunctions provide more generic
> > APIs to implement stream processing applications. The ProcessFunctions
> > expose time and state which are the fundamental building blocks for any
> > kind of streaming application.
> > To cover more use cases, we are planning to cover all these APIs in
> > PyFlink.
> >
> > In this discussion(FLIP-130), we propose to support the Python DataStream
> > API for the stateless part. For more detail, please refer to the FLIP
> wiki
> > page here[2]. If interested in the stateful part, you can also take a
> > look the design doc here[3] for which we are going to discuss in a
> separate
> > FLIP.
> >
> > Any comments will be highly appreciated!
> >
> > [1] https://flink.apache.org/flink-applications.html#layered-apis
> > [2]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866298
> > [3]
> >
> https://docs.google.com/document/d/1H3hz8wuk228cDBhQmQKNw3m1q5gDAMkwTDEwnj3FBI/edit?usp=sharing
> >
> > Best,
> > Shuiqiang
> >
> >
> >
> >
>

[jira] [Created] (FLINK-18151) Resolve CWE22 problems in pyflink_gateway_server.py

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18151:
---

 Summary: Resolve CWE22 problems in pyflink_gateway_server.py 
 Key: FLINK-18151
 URL: https://issues.apache.org/jira/browse/FLINK-18151
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.10.1, 1.11.0, 1.12.0
Reporter: Hequn Cheng


For example, the code `if os.path.isfile(flink_conf_path):` contains CWE22 
problem that calling "os.path.isfile" with the tainted value in argument 1. 
This constructs a path or URI using the tainted value and may thus allow an 
attacker to access, modify, or test the existence of critical or sensitive 
files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song

[Hequn Cheng]

Congratulations Xintong!

Best, Hequn

On Fri, Jun 5, 2020 at 2:03 PM Marta Paes Moreira 
wrote:

> Congratulations, Tony! Great job!
>
> On Fri, Jun 5, 2020 at 8:00 AM Yang Wang  wrote:
>
> > Congratulations, Xintong.
> > Well deserved.
> >
> > Best,
> > Yang
> >
> > Weihua Hu  于2020年6月5日周五 下午1:57写道：
> >
> > > Congratulations!
> > >
> > > Best
> > > Weihua Hu
> > >
> > > > 2020年6月5日 13:50，Xingbo Huang  写道：
> > > >
> > > > Congratulations, Xintong.
> > > >
> > > > Best,
> > > > Xingbo
> > > >
> > > > Jiayi Liao  于2020年6月5日周五 下午1:49写道：
> > > >
> > > >> Congratulations!
> > > >>
> > > >> Best,
> > > >> Jiayi Liao
> > > >>
> > > >> On Fri, Jun 5, 2020 at 1:48 PM Biao Liu  wrote:
> > > >>
> > > >>> Congrats!
> > > >>>
> > > >>> Thanks,
> > > >>> Biao /'bɪ.aʊ/
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Fri, 5 Jun 2020 at 13:32, Thomas Weise  wrote:
> > > >>>
> > >  Congratulations!
> > > 
> > > 
> > >  On Thu, Jun 4, 2020, 10:17 PM Yuan Mei 
> > > wrote:
> > > 
> > > > Congrats, Xintong!
> > > >
> > > > On Fri, Jun 5, 2020 at 12:45 PM Becket Qin  >
> > > >>> wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> On behalf of the PMC, I’m very happy to announce Xintong Song
> as a
> > > >>> new
> > > >> Flink committer.
> > > >>
> > > >> Xintong started to contribute to Flink about two years ago and
> has
> > > >>> been
> > > >> active since. His major work is in Flink resource management,
> and
> > > >>> have
> > > > also
> > > >> participated in discussions, bug fixes and answering questions.
> > > >>
> > > >> Please join me in congratulating Xintong for becoming a Flink
> > >  committer!
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jiangjie (Becket) Qin
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > >
> > >
> >
>

[jira] [Created] (FLINK-18143) Fix Python meter metric not correct problem

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18143:
---

 Summary: Fix Python meter metric not correct problem
 Key: FLINK-18143
 URL: https://issues.apache.org/jira/browse/FLINK-18143
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.11.0
Reporter: Hequn Cheng
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 2.1.0, release candidate #1

[Hequn Cheng]

+1 (binding)

- Signatures and hash are correct.
- All artifacts to be released to Maven in the staging Nexus repository.
- Verify that the source archives do not contain any binaries.
- Go through all commits from the last release. No license problem spotted.
- Check end-to-end tests. All tests have been passed on Travis(both for JDK
1.8 and 1.11).

Best,
Hequn

On Thu, Jun 4, 2020 at 12:50 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi Hequn,
>
> Sorry, I mis-tagged the wrong commit.
> Just fixed this, the tag [1] [2] should now be pointing to the correct
> commit that contains the updated version.
>
> Gordon
>
> [1]
>
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=tag;h=c08c9850147d818fc8fed877a01ff87021f3cf21
> [2] https://github.com/apache/flink-statefun/tree/release-2.1.0-rc1
>
> On Thu, Jun 4, 2020 at 12:10 PM Hequn Cheng  wrote:
>
> > It seems the release tag is not correct? The version in the poms should
> > be 2.1.0 instead of 2.1-SNAPSHOT.
> >
> > Best,
> > Hequn
> >
> >
> > On Thu, Jun 4, 2020 at 10:33 AM Congxian Qiu 
> > wrote:
> >
> > > +1  (non-binding)
> > >
> > > maybe there is something that needs to be updated in
> README.md(currently
> > > the official docs link points to the master instead of 2.1)
> > >
> > > and have another question: do we need to add the command used to build
> > the
> > > base image locally(which was on the README.md in release-2.0.0)?
> > >
> > > checked
> > > - sha & gpg, ok
> > > - mvn clean install -Prun-e2e-test on 1.8.0_252, ok
> > > - source archives do not contains any binaries
> > > - maven clean install -Papache-release, ok (this step need a gpg secret
> > > key)
> > > - check all pom files, dockerfiles, examples point to the same version,
> > ok
> > > - check READM.md, nothing unexpected.
> > > - but the official docs link points to the master instead of 2.1
> > > - run greeter demo, ok
> > >
> > > Best,
> > > Congxian
> > >
> > >
> > > Tzu-Li (Gordon) Tai  于2020年6月1日周一 下午3:25写道：
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the *release candidate #1* for the version
> > > 2.1.0
> > > > of
> > > > Apache Flink Stateful Functions,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > ***Testing Guideline***
> > > >
> > > > You can find here [1] a page in the project wiki on instructions for
> > > > testing.
> > > > To cast a vote, it is not necessary to perform all listed checks,
> > > > but please mention which checks you have performed when voting.
> > > >
> > > > ***Release Overview***
> > > >
> > > > As an overview, the release consists of the following:
> > > > a) Stateful Functions canonical source distribution, to be deployed
> to
> > > the
> > > > release repository at dist.apache.org
> > > > b) Stateful Functions Python SDK distributions to be deployed to PyPI
> > > > c) Maven artifacts to be deployed to the Maven Central Repository
> > > >
> > > > ***Staging Areas to Review***
> > > >
> > > > The staging areas containing the above mentioned artifacts are as
> > > follows,
> > > > for your review:
> > > > * All artifacts for a) and b) can be found in the corresponding dev
> > > > repository at dist.apache.org [2]
> > > > * All artifacts for c) can be found at the Apache Nexus Repository
> [3]
> > > >
> > > > All artifacts are singed with the
> > > > key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
> > > >
> > > > Other links for your review:
> > > > * JIRA release notes [5]
> > > > * source code tag "release-2.1.0-rc1" [6] [7]
> > > >
> > > > ***Vote Duration***
> > > >
> > > > The vote will be open for at least 72 hours *(target end date is
> > > Wednesday,
> > > > Jun. 3rd).*
> > > > It is adopted by majority approval, with at least 3 PMC affirmative
> > > votes.
> > > >
> > > > Thanks,
> > > > Gordon
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> > > > [2]
> > > https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.1.0-rc1/
> > > > [3]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1373/
> > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [5]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12347861
> > > > [6]
> > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=tag;h=a372c69501b8816609f9e90872b1a1e10ab66e8e
> > > > [7] https://github.com/apache/flink-statefun/tree/release-2.1.0-rc1
> > > >
> > >
> >
>

Re: [VOTE] Apache Flink Stateful Functions 2.1.0, release candidate #1

[Hequn Cheng]

It seems the release tag is not correct? The version in the poms should
be 2.1.0 instead of 2.1-SNAPSHOT.

Best,
Hequn


On Thu, Jun 4, 2020 at 10:33 AM Congxian Qiu  wrote:

> +1  (non-binding)
>
> maybe there is something that needs to be updated in README.md(currently
> the official docs link points to the master instead of 2.1)
>
> and have another question: do we need to add the command used to build the
> base image locally(which was on the README.md in release-2.0.0)?
>
> checked
> - sha & gpg, ok
> - mvn clean install -Prun-e2e-test on 1.8.0_252, ok
> - source archives do not contains any binaries
> - maven clean install -Papache-release, ok (this step need a gpg secret
> key)
> - check all pom files, dockerfiles, examples point to the same version, ok
> - check READM.md, nothing unexpected.
> - but the official docs link points to the master instead of 2.1
> - run greeter demo, ok
>
> Best,
> Congxian
>
>
> Tzu-Li (Gordon) Tai  于2020年6月1日周一 下午3:25写道：
>
> > Hi everyone,
> >
> > Please review and vote on the *release candidate #1* for the version
> 2.1.0
> > of
> > Apache Flink Stateful Functions,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > ***Testing Guideline***
> >
> > You can find here [1] a page in the project wiki on instructions for
> > testing.
> > To cast a vote, it is not necessary to perform all listed checks,
> > but please mention which checks you have performed when voting.
> >
> > ***Release Overview***
> >
> > As an overview, the release consists of the following:
> > a) Stateful Functions canonical source distribution, to be deployed to
> the
> > release repository at dist.apache.org
> > b) Stateful Functions Python SDK distributions to be deployed to PyPI
> > c) Maven artifacts to be deployed to the Maven Central Repository
> >
> > ***Staging Areas to Review***
> >
> > The staging areas containing the above mentioned artifacts are as
> follows,
> > for your review:
> > * All artifacts for a) and b) can be found in the corresponding dev
> > repository at dist.apache.org [2]
> > * All artifacts for c) can be found at the Apache Nexus Repository [3]
> >
> > All artifacts are singed with the
> > key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
> >
> > Other links for your review:
> > * JIRA release notes [5]
> > * source code tag "release-2.1.0-rc1" [6] [7]
> >
> > ***Vote Duration***
> >
> > The vote will be open for at least 72 hours *(target end date is
> Wednesday,
> > Jun. 3rd).*
> > It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> >
> > Thanks,
> > Gordon
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> > [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.1.0-rc1/
> > [3]
> > https://repository.apache.org/content/repositories/orgapacheflink-1373/
> > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [5]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12347861
> > [6]
> >
> >
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=tag;h=a372c69501b8816609f9e90872b1a1e10ab66e8e
> > [7] https://github.com/apache/flink-statefun/tree/release-2.1.0-rc1
> >
>

Re: [VOTE] Release flink-shaded 11.0, release candidate #1

[Hequn Cheng]

+1 (binding)

1. Go through all the commits from 11.0.
2. website PR looks good
3. Built from source archive successfully.
4. Signatures and hash are correct.

Best,
Hequn

On Fri, May 29, 2020 at 4:08 PM Chesnay Schepler  wrote:

> +1 (binding)
>
> On 28/05/2020 18:49, Yu Li wrote:
> > +1 (non-binding)
> >
> > Checked issues listed in release notes: ok (*)
> > - Add shaded-11.0 as fixed version for FLINK-17513
> > - Minor: FLINK-16454 is listed in the release note but found no changes
> in
> > commit history
> > Checked sums and signatures: ok
> > Checked the maven central artifices: ok
> > Built from source: ok (8u101)
> > Built from source (with -Dshade-sources): ok (8u101)
> > Checked website pull request listing the new release: ok
> >
> > Best Regards,
> > Yu
> >
> >
> > On Fri, 29 May 2020 at 00:32, Till Rohrmann 
> wrote:
> >
> >> +1 (binding)
> >>
> >> - verified checksums and signature
> >> - mvn clean verify passes on source release
> >> - verified licenses
> >> - checked pom.xml changes
> >>
> >> Cheers,
> >> Till
> >>
> >> On Thu, May 28, 2020 at 1:05 PM Congxian Qiu 
> >> wrote:
> >>
> >>> +1 (non-binding)
> >>>
> >>> checked
> >>> - mvn clean verify, ok
> >>> - gpg & sha512, ok
> >>> - all pom files point to the same version, ok
> >>> - checked license, ok
> >>>
> >>> Best,
> >>> Congxian
> >>>
> >>>
> >>> Robert Metzger  于2020年5月27日周三 下午6:05写道：
> >>>
>  +1 (binding)
> 
>  Checks:
>  - diff to flink-shaded 1.10:
> 
> 
> >>
> https://github.com/apache/flink-shaded/compare/release-10.0...release-11.0-rc1
>  - mvn clean install passes on the source archive
>  - sha of source archive is correct
>  - source archive is signed by Chesnay
>  - mvn staging repo looks reasonable
>  - flink-shaded-zookeeper 3 jar license documentation seems correct
> 
> 
> 
>  On Mon, May 25, 2020 at 7:14 PM Chesnay Schepler 
>  wrote:
> 
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version
> >>> 11.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which
> >> includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to
> >> dist.apache.org
> > [2], which are signed with the key with fingerprint 11D464BA [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-11.0-rc1" [5],
> > * website pull request listing the new release [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by
> >> majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Chesnay
> >
> > [1]
> >
> >
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12347784
> > [2]
> >>> https://dist.apache.org/repos/dist/dev/flink/flink-shaded-11.0-rc1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> >
> >>>
> https://repository.apache.org/content/repositories/orgapacheflink-1372/
> > [5] https://github.com/apache/flink-shaded/tree/release-11.0-rc1
> > [6] https://github.com/apache/flink-web/pull/340
> >
> >
>
>

Re: [VOTE] Release flink-shaded 11.0, release candidate #1

[Hequn Cheng]

@Chesnay Schepler   Thanks a lot for your confirmation.
I think it is good if it wouldn't block the release. I have lower
the Priority to Critical :)

On Fri, May 29, 2020 at 3:20 PM Chesnay Schepler  wrote:

>  From my understanding the current year in the copyright header is not
> such an important thing, so I wouldn't block the release on it.
>
> On 29/05/2020 09:11, Hequn Cheng wrote:
> > Sorry about my late check.
> > I found one blocker just now that the year in the root NOTICE file should
> > be updated. I have created a jira(FLINK-18031) for it.
> >
> > Best,
> > Hequn
> >
> >
> > On Fri, May 29, 2020 at 12:47 AM Yu Li  wrote:
> >
> >> +1 (non-binding)
> >>
> >> Checked issues listed in release notes: ok (*)
> >> - Add shaded-11.0 as fixed version for FLINK-17513
> >> - Minor: FLINK-16454 is listed in the release note but found no changes
> in
> >> commit history
> >> Checked sums and signatures: ok
> >> Checked the maven central artifices: ok
> >> Built from source: ok (8u101)
> >> Built from source (with -Dshade-sources): ok (8u101)
> >> Checked website pull request listing the new release: ok
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Fri, 29 May 2020 at 00:32, Till Rohrmann 
> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> - verified checksums and signature
> >>> - mvn clean verify passes on source release
> >>> - verified licenses
> >>> - checked pom.xml changes
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Thu, May 28, 2020 at 1:05 PM Congxian Qiu 
> >>> wrote:
> >>>
> >>>> +1 (non-binding)
> >>>>
> >>>> checked
> >>>> - mvn clean verify, ok
> >>>> - gpg & sha512, ok
> >>>> - all pom files point to the same version, ok
> >>>> - checked license, ok
> >>>>
> >>>> Best,
> >>>> Congxian
> >>>>
> >>>>
> >>>> Robert Metzger  于2020年5月27日周三 下午6:05写道：
> >>>>
> >>>>> +1 (binding)
> >>>>>
> >>>>> Checks:
> >>>>> - diff to flink-shaded 1.10:
> >>>>>
> >>>>>
> >>
> https://github.com/apache/flink-shaded/compare/release-10.0...release-11.0-rc1
> >>>>> - mvn clean install passes on the source archive
> >>>>> - sha of source archive is correct
> >>>>> - source archive is signed by Chesnay
> >>>>> - mvn staging repo looks reasonable
> >>>>> - flink-shaded-zookeeper 3 jar license documentation seems correct
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, May 25, 2020 at 7:14 PM Chesnay Schepler  >>>>> wrote:
> >>>>>
> >>>>>> Hi everyone,
> >>>>>> Please review and vote on the release candidate #1 for the version
> >>>> 11.0,
> >>>>>> as follows:
> >>>>>> [ ] +1, Approve the release
> >>>>>> [ ] -1, Do not approve the release (please provide specific
> >> comments)
> >>>>>>
> >>>>>> The complete staging area is available for your review, which
> >>> includes:
> >>>>>> * JIRA release notes [1],
> >>>>>> * the official Apache source release to be deployed to
> >>> dist.apache.org
> >>>>>> [2], which are signed with the key with fingerprint 11D464BA [3],
> >>>>>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>>>>> * source code tag "release-11.0-rc1" [5],
> >>>>>> * website pull request listing the new release [6].
> >>>>>>
> >>>>>> The vote will be open for at least 72 hours. It is adopted by
> >>> majority
> >>>>>> approval, with at least 3 PMC affirmative votes.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Chesnay
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12347784
> >>>>>> [2]
> >>>> https://dist.apache.org/repos/dist/dev/flink/flink-shaded-11.0-rc1/
> >>>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>>>>> [4]
> >>>>>>
> >> https://repository.apache.org/content/repositories/orgapacheflink-1372/
> >>>>>> [5] https://github.com/apache/flink-shaded/tree/release-11.0-rc1
> >>>>>> [6] https://github.com/apache/flink-web/pull/340
> >>>>>>
> >>>>>>
>
>

Re: [VOTE] Release flink-shaded 11.0, release candidate #1

[Hequn Cheng]

Sorry about my late check.
I found one blocker just now that the year in the root NOTICE file should
be updated. I have created a jira(FLINK-18031) for it.

Best,
Hequn


On Fri, May 29, 2020 at 12:47 AM Yu Li  wrote:

> +1 (non-binding)
>
> Checked issues listed in release notes: ok (*)
> - Add shaded-11.0 as fixed version for FLINK-17513
> - Minor: FLINK-16454 is listed in the release note but found no changes in
> commit history
> Checked sums and signatures: ok
> Checked the maven central artifices: ok
> Built from source: ok (8u101)
> Built from source (with -Dshade-sources): ok (8u101)
> Checked website pull request listing the new release: ok
>
> Best Regards,
> Yu
>
>
> On Fri, 29 May 2020 at 00:32, Till Rohrmann  wrote:
>
> > +1 (binding)
> >
> > - verified checksums and signature
> > - mvn clean verify passes on source release
> > - verified licenses
> > - checked pom.xml changes
> >
> > Cheers,
> > Till
> >
> > On Thu, May 28, 2020 at 1:05 PM Congxian Qiu 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > checked
> > > - mvn clean verify, ok
> > > - gpg & sha512, ok
> > > - all pom files point to the same version, ok
> > > - checked license, ok
> > >
> > > Best,
> > > Congxian
> > >
> > >
> > > Robert Metzger  于2020年5月27日周三 下午6:05写道：
> > >
> > > > +1 (binding)
> > > >
> > > > Checks:
> > > > - diff to flink-shaded 1.10:
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink-shaded/compare/release-10.0...release-11.0-rc1
> > > >
> > > > - mvn clean install passes on the source archive
> > > > - sha of source archive is correct
> > > > - source archive is signed by Chesnay
> > > > - mvn staging repo looks reasonable
> > > > - flink-shaded-zookeeper 3 jar license documentation seems correct
> > > >
> > > >
> > > >
> > > > On Mon, May 25, 2020 at 7:14 PM Chesnay Schepler  >
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > > Please review and vote on the release candidate #1 for the version
> > > 11.0,
> > > > > as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > > * JIRA release notes [1],
> > > > > * the official Apache source release to be deployed to
> > dist.apache.org
> > > > > [2], which are signed with the key with fingerprint 11D464BA [3],
> > > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > > * source code tag "release-11.0-rc1" [5],
> > > > > * website pull request listing the new release [6].
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > Thanks,
> > > > > Chesnay
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12347784
> > > > > [2]
> > > https://dist.apache.org/repos/dist/dev/flink/flink-shaded-11.0-rc1/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1372/
> > > > > [5] https://github.com/apache/flink-shaded/tree/release-11.0-rc1
> > > > > [6] https://github.com/apache/flink-web/pull/340
> > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-18031) Update the copyright year in the NOTICE file in flink-shaded repo

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-18031:
---

 Summary: Update the copyright year in the NOTICE file in 
flink-shaded repo
 Key: FLINK-18031
 URL: https://issues.apache.org/jira/browse/FLINK-18031
 Project: Flink
  Issue Type: Bug
Affects Versions: shaded-10.0
Reporter: Hequn Cheng
 Fix For: shaded-11.0


The year in the root NOTICE file should be updated from `2014-2017` to 
`2014-2020`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17962) Add document for how to define Python UDF with DDL

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-17962:
---

 Summary: Add document for how to define Python UDF with DDL
 Key: FLINK-17962
 URL: https://issues.apache.org/jira/browse/FLINK-17962
 Project: Flink
  Issue Type: Improvement
  Components: API / Python, Documentation
Reporter: Hequn Cheng






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17960) Improve commands in the "Common Questions" document for PyFlink

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-17960:
---

 Summary: Improve commands in the "Common Questions" document for 
PyFlink
 Key: FLINK-17960
 URL: https://issues.apache.org/jira/browse/FLINK-17960
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.11.0
Reporter: Hequn Cheng


Currently, in the ["Common Questions" 
|https://ci.apache.org/projects/flink/flink-docs-master/dev/table/python/common_questions.html#preparing-python-virtual-environment]document,
 we have the command `$ setup-pyflink-virtual-env.sh` to run the script. 
However, the script is not executable. It would be better to replace the 
command with `$ sh setup-pyflink-virtual-env.sh` and add download command.
{code}
$ curl -O 
https://ci.apache.org/projects/flink/flink-docs-master/downloads/setup-pyflink-virtual-env.sh
$ sh setup-pyflink-virtual-env.sh
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17959) Exception: "CANCELLED: call already cancelled" is thrown when run python udf

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-17959:
---

 Summary: Exception: "CANCELLED: call already cancelled" is thrown 
when run python udf
 Key: FLINK-17959
 URL: https://issues.apache.org/jira/browse/FLINK-17959
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.10.1, 1.11.0
Reporter: Hequn Cheng


The exception is thrown when running Python UDF:
{code:java}
May 27, 2020 3:20:49 PM 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@3960b30e
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED: 
call already cancelled
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status.asRuntimeException(Status.java:524)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:366)
at 
org.apache.beam.runners.fnexecution.state.GrpcStateService$Inbound.onError(GrpcStateService.java:145)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.PartialForwardingServerCallListener.onCancel(PartialForwardingServerCallListener.java:40)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:23)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:40)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Contexts$ContextualizedServerCallListener.onCancel(Contexts.java:96)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:337)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:793)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at 
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}

The job can output the right results however it seems something goes wrong 
during the shutdown procedure.

You can reproduce the exception with the following code(note: the exception 
happens occasionally):
{code}
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import StreamTableEnvironment, DataTypes
from pyflink.table.descriptors import Schema, OldCsv, FileSystem
from pyflink.table.udf import udf

env = StreamExecutionEnvironment.get_execution_environment()
env.set_parallelism(1)
t_env = StreamTableEnvironment.create(env)

add = udf(lambda i, j: i + j, [DataTypes.BIGINT(), DataTypes.BIGINT()], 
DataTypes.BIGINT())

t_env.register_function("add", add)

t_env.connect(FileSystem().path('/tmp/input')) \
.with_format(OldCsv()
 .field('a', DataTypes.BIGINT())
 .field('b', DataTypes.BIGINT())) \
.with_schema(Schema()
 .field('a', DataTypes.BIGINT())
 .field('b', DataTypes.BIGINT())) \
.create_temporary_table('mySource')

t_env.connect(FileSystem().path('/tmp/output')) \
.with_format(OldCsv()
 .field('sum', DataTypes.BIGINT())) \
.with_schema(Schema()
 .field('sum', DataTypes.BIGINT())) \
.create_temporary_table('mySink')

t_env.from_path('mySource')\
.select("add(a, b)") \
.insert_into('mySink')

t_env.execute("tutorial_job")
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Release flink-shaded 11.0 (and include it in 1.11.0)

[Hequn Cheng]

+1
It would be nice to have these fixes!

Best,
Hequn

On Mon, May 25, 2020 at 5:00 PM Zhijiang 
wrote:

> Thanks for driving this, Chesnay!
> +1 on my side.
>
> Best,
> Zhijiang
> --
> From:Till Rohrmann 
> Send Time:2020年5月25日(星期一) 16:28
> To:dev 
> Subject:Re: [DISCUSS] Release flink-shaded 11.0 (and include it in 1.11.0)
>
> +1 for the new flink-shaded release.
>
> Cheers,
> Till
>
> On Mon, May 25, 2020 at 9:06 AM Chesnay Schepler 
> wrote:
>
> > Hello,
> >
> > I would like to do another flink-shaded release for 1.11.0, to include a
> > zookeeper 3.4 security fix and resolve a shading issue when working with
> > Gradle.
> >
> >
> >
>
>

[jira] [Created] (FLINK-17710) StreamSqlTests.test_execute_sql test is not stable

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-17710:
---

 Summary: StreamSqlTests.test_execute_sql test is not stable
 Key: FLINK-17710
 URL: https://issues.apache.org/jira/browse/FLINK-17710
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Reporter: Hequn Cheng


Failure log:
https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/1311/logs/144



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.10.1, release candidate #3

[Hequn Cheng]

+1 (binding)

- Go through all commits from 1.10.0 to 1.10.1 and spot no new license
problems.
- Built from source archive successfully.
- Travis e2e tests have passed.
- Signatures and hash are correct.
- Run SocketWindowWordCount on the local cluster and check web ui & logs.
- Install Python package and run Python WordCount example.
- Website PR looks good.

Best,
Hequn

On Mon, May 11, 2020 at 10:39 PM Ufuk Celebi  wrote:

> +1 (binding)
>
> - checked release notes
> - verified sums and hashes
> - reviewed website PR
> - successfully built an internal Flink distribution based on the 1.10.1-rc3
> commit
> - successfully built internal jobs against the staging repo and deployed
> those jobs to a 1.10.1 job cluster on Kubernetes and tested checkpointing
>
> –  Ufuk
>
> On Mon, May 11, 2020 at 11:47 AM Tzu-Li (Gordon) Tai 
> wrote:
> >
> > +1 (binding)
> >
> > Legal checks:
> > - checksum & gpg OK
> > - changes to Kinesis connector NOTICE files looks good
> > - built from source
> >
> > Downstream checks in flink-statefun:
> > - Built StateFun with Flink 1.10.1 + e2e tests enabled (mvn clean install
> > -Prun-e2e-tests), builds and passes
> > - Previous issue preventing successful failure recovery when using the
> new
> > scheduler, is now fixed with this RC
> >
> > Cheers,
> > Gordon
> >
> > On Mon, May 11, 2020 at 2:47 PM Congxian Qiu 
> wrote:
> >
> > > +1 (no-binding)
> > >
> > > - checksum & gpg ok
> > > - build from source OK
> > > - all pom files points to the same version OK
> > > - LICENSE OK
> > > - run demos OK
> > > Best,
> > > Congxian
> > >
> > >
> > > Dian Fu  于2020年5月10日周日 下午10:14写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > - checked the dependency changes since 1.10.0
> > > >   1) kafka was bumped from 0.10.2.1 to 0.10.2.2 for
> > > > flink-connector-kafka-0.10 and it has been reflected in the notice
> file
> > > >   2) amazon-kinesis-producer was bumped from 0.13.1 to 0.14.0 and it
> has
> > > > been reflected in the notice file
> > > >   3) the dependencies com.carrotsearch:hppc,
> > > > com.github.spullara.mustache.java,
> org.elasticsearch:elasticsearch-geo
> > > and
> > > > org.elasticsearch.plugin:lang-mustache-client was bundled in the jar
> of
> > > > flink-sql-connector-elasticsearch7 and it has been reflected in the
> > > notice
> > > > file
> > > >   4) influxdb-java was bumped from 2.16 to 2.17 and it has been
> reflected
> > > > in the notice file
> > > > - verified the checksum and signature
> > > > - checked that the PyFlink package could be pip installed
> > > > - have left a few minor comments on the website PR
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > On Sat, May 9, 2020 at 12:02 PM Thomas Weise  wrote:
> > > >
> > > > > Thanks for the RC!
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > - repeated benchmark runs
> > > > >
> > > > >
> > > > > On Fri, May 8, 2020 at 10:52 AM Robert Metzger <
> rmetz...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Thanks a lot for creating another RC!
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > - checked diff to last RC:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
> https://github.com/apache/flink/compare/release-1.10.1-rc2...release-1.10.1-rc3
> > > > > >   - kinesis dependency change is properly documented
> > > > > > - started Flink locally (on Java11 :) )
> > > > > >- seems to be build off the specified commit
> > > > > >- ran example
> > > > > >- checked logs
> > > > > > - staging repo looks ok
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, May 7, 2020 at 2:21 PM Yu Li  wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Please review and vote on the release candidate #3 for version
> > > > 1.10.1,
> > > > > as
> > > > > > > follows:
> > > > > > > [ ] +1, Approve the release
> > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > > >
> > > > > > >
> > > > > > > The complete staging area is available for your review, which
> > > > includes:
> > > > > > > * JIRA release notes [1],
> > > > > > > * the official Apache source release and binary convenience
> > > releases
> > > > to
> > > > > > be
> > > > > > > deployed to dist.apache.org [2], which are signed with the key
> > > with
> > > > > > > fingerprint D8D3D42E84C753CA5F170BDF93C07902771AB743 [3],
> > > > > > > * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > > > > * source code tag "release-1.10.1-rc3" [5],
> > > > > > > * website pull request listing the new release and adding
> > > > announcement
> > > > > > blog
> > > > > > > post [6].
> > > > > > >
> > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > majority
> > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yu
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
> 

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Hequn Cheng]

+1, thanks a lot for driving this.

Best, Hequn

On Tue, May 5, 2020 at 11:56 AM Dian Fu  wrote:

> +1
>
> Regards,
> Dian
>
> > 在 2020年5月5日，上午9:58，Yangze Guo  写道：
> >
> > +1
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, May 5, 2020 at 6:14 AM Thomas Weise  wrote:
> >>
> >> +1
> >>
> >>
> >> On Mon, May 4, 2020 at 10:02 AM Marta Paes Moreira  >
> >> wrote:
> >>
> >>> +1, this is quite annoying and distracting.
> >>>
> >>> Marta
> >>>
> >>> On Mon, May 4, 2020 at 6:27 PM Yu Li  wrote:
> >>>
>  +1
> 
>  Best Regards,
>  Yu
> 
> 
>  On Tue, 5 May 2020 at 00:21, Konstantin Knauf 
> wrote:
> 
> > Yes, please.
> >
> > On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz <
> >>> dwysakow...@apache.org>
> > wrote:
> >
> >> +1
> >>
> >> Yes, please. I've also observed a lot of noise in the past days.
> >>
> >> Best,
> >>
> >> Dawid
> >>
> >> On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> >>> +1
> >>>
> >>> All the recent new repos, flink-statefun / flink-statefun-docker /
> >>> flink-training etc. are also sending notifications to issues@.
> >>>
> >>> Gordon
> >>>
> >>>
> >>> On Mon, May 4, 2020, 11:44 PM Till Rohrmann 
> >> wrote:
> >>>
>  Hi everyone,
> 
>  due to some changes on the ASF side, we are now seeing issue and
>  pull
>  request notifications for the flink-web [1] and flink-shaded [2]
>  repo
> > on
>  dev@flink.apache.org. I think this is not ideal since the dev ML
> >>> is
> >> much
>  more noisy now.
> 
>  I would propose to send these notifications to
> > iss...@flink.apache.org
> >> as
>  we are currently doing it for the Flink main repo [3].
> 
>  What do you think?
> 
>  [1] https://github.com/apache/flink-web
>  [2] https://github.com/apache/flink-shaded
>  [3] https://gitbox.apache.org/schemes.cgi?flink
> 
>  Cheers,
>  Till
> 
> >>
> >>
> >
> > --
> >
> > Konstantin Knauf
> >
> > https://twitter.com/snntrable
> >
> > https://github.com/knaufk
> >
> 
> >>>
>
>

Re: [VOTE] Release 1.10.1, release candidate #2

[Hequn Cheng]

Thanks a lot for managing the release!

+1 (binding)

- Go through all new commits for 1.10.1 and spot no new license problems.
- Built from source archive successfully.
- Signatures and hash are correct.
- Run SocketWindowWordCount on the local cluster.
- Install Python package and run Python WordCount example.
- Reviewed website PR

Best,
Hequn

On Sun, May 3, 2020 at 9:10 PM Robert Metzger  wrote:

> Thanks a lot for addressing the issues from the last release candidate and
> creating this one!
>
> +1 (binding)
>
> - Started Flink on YARN on Google Cloud DataProc by setting
> HADOOP_CLASSPATH
> - checked staging repo
>
>
>
> On Sat, May 2, 2020 at 6:57 PM Thomas Weise  wrote:
>
> > +1 (binding)
> >
> > Checked signatures and hashes.
> >
> > Run internal benchmark applications.
> >
> > I found a regression that was actually introduced with 1.10.0, hence not
> a
> > blocker for this release:
> >
> > https://github.com/apache/flink/pull/11975
> >
> > Thanks,
> > Thomas
> >
> >
> > On Fri, May 1, 2020 at 5:37 AM Yu Li  wrote:
> >
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #2 for version 1.10.1,
> as
> > > follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release and binary convenience releases to
> > be
> > > deployed to dist.apache.org [2], which are signed with the key with
> > > fingerprint D8D3D42E84C753CA5F170BDF93C07902771AB743 [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag "release-1.10.1-rc2" [5],
> > > * website pull request listing the new release and adding announcement
> > blog
> > > post [6].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Yu
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346891
> > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.10.1-rc2/
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1363/
> > > [5]
> > >
> > >
> >
> https://github.com/apache/flink/commit/f92e8a9d60ef664acd66230da43c6f0a1cd87adc
> > > [6] https://github.com/apache/flink-web/pull/330
> > >
> >
>

Re: [ANNOUNCE] Apache Flink 1.9.3 released

[Hequn Cheng]

@Dian, thanks a lot for the release and for being the release manager.
Also thanks to everyone who made this release possible!

Best,
Hequn

On Sat, Apr 25, 2020 at 7:57 PM Dian Fu  wrote:

> Hi everyone,
>
> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.9.3, which is the third bugfix release for the Apache Flink 1.9
> series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
> https://flink.apache.org/news/2020/04/24/release-1.9.3.html
>
> The full release notes are available in Jira:
> https://issues.apache.org/jira/projects/FLINK/versions/12346867
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
> Also great thanks to @Jincheng for helping finalize this release.
>
> Regards,
> Dian
>

Re: New contributor

[Hequn Cheng]

Welcome, Etienne :)

Best,
Hequn

On Fri, Apr 24, 2020 at 10:04 PM Etienne Chauchot 
wrote:

> Hi Till,
>
> Looking forward too ...
>
> Thanks
>
> Etienne
>
> On 24/04/2020 15:09, Till Rohrmann wrote:
> > Hi Etienne,
> >
> > welcome to the Flink community. Looking forward to working with you on
> > Flink :-)
> >
> > Cheers,
> > Till
> >
> > On Fri, Apr 24, 2020 at 11:20 AM Etienne Chauchot 
> > wrote:
> >
> >> Hi everyone,
> >>
> >> Let me introduce myself, I'm Etienne Chauchot, I'm an Apache Beam
> >> committer and a PMC member and I would like to start working on Flink as
> >> well.
> >>
> >> For now I only did 3 simple Flink PRs but that will grow :)
> >>
> >> https://github.com/apache/flink/pull/11886
> >>
> >> https://github.com/apache/flink/pull/11740
> >>
> >> https://github.com/apache/flink/pull/11703
> >>
> >> Here is a link to my blog: https://echauchot.blogspot.com/
> >>
> >> Best.
> >>
> >> Etienne
> >>
> >>
>

[jira] [Created] (FLINK-17318) The comment is not right in `org.apache.flink.table.planner.delegation.PlannerBase`

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-17318:
---

 Summary: The comment is not right in 
`org.apache.flink.table.planner.delegation.PlannerBase`
 Key: FLINK-17318
 URL: https://issues.apache.org/jira/browse/FLINK-17318
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.10.0
Reporter: Hequn Cheng


`org.apache.flink.table.planner.delegation.PlannerBase` should be an 
implementation of Blink planner instead of legacy Flink planner.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

[Hequn Cheng]

Many thanks for your support. Thank you!

Best,
Hequn

On Sat, Apr 18, 2020 at 1:27 AM Jacky Bai  wrote:

> Congratulations！Hequn Chen.I hope to make so many contributions to Flink
> like you.
>
> Best
> Bai Xu
>
> Congxian Qiu  于2020年4月17日周五 下午10:47写道：
>
> > Congratulations, Hequn!
> >
> > Best,
> > Congxian
> >
> >
> > Yu Li  于2020年4月17日周五 下午9:36写道：
> >
> > > Congratulations, Hequn!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Fri, 17 Apr 2020 at 21:22, Kurt Young  wrote:
> > >
> > > > Congratulations Hequn!
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Fri, Apr 17, 2020 at 8:57 PM Till Rohrmann 
> > > > wrote:
> > > >
> > > > > Congratulations Hequn!
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Fri, Apr 17, 2020 at 2:49 PM Shuo Cheng 
> > wrote:
> > > > >
> > > > > > Congratulations, Hequn
> > > > > >
> > > > > > Best,
> > > > > > Shuo
> > > > > >
> > > > > > On 4/17/20, hufeih...@mails.ucas.ac.cn <
> hufeih...@mails.ucas.ac.cn
> > >
> > > > > wrote:
> > > > > > > Congratulations , Hequn
> > > > > > >
> > > > > > > Best wish
> > > > > > >
> > > > > > >
> > > > > > > hufeih...@mails.ucas.ac.cn
> > > > > > > Congratulations, Hequn!
> > > > > > >
> > > > > > > Paul Lam  于2020年4月17日周五 下午3:02写道：
> > > > > > >
> > > > > > >> Congrats Hequn! Thanks a lot for your contribution to the
> > > community!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Paul Lam
> > > > > > >>
> > > > > > >> Dian Fu  于2020年4月17日周五 下午2:58写道：
> > > > > > >>
> > > > > > >> > Congratulations, Hequn!
> > > > > > >> >
> > > > > > >> > > 在 2020年4月17日，下午2:36，Becket Qin  写道：
> > > > > > >> > >
> > > > > > >> > > Hi all,
> > > > > > >> > >
> > > > > > >> > > I am glad to announce that Hequn Chen has joined the Flink
> > > PMC.
> > > > > > >> > >
> > > > > > >> > > Hequn has contributed to Flink for years. He has worked on
> > > > several
> > > > > > >> > > components including Table / SQL，PyFlink and Flink ML
> > > Pipeline.
> > > > > > >> Besides,
> > > > > > >> > > Hequn is also very active in the community since the
> > > beginning.
> > > > > > >> > >
> > > > > > >> > > Congratulations, Hequn! Looking forward to your future
> > > > > > contributions.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > >
> > > > > > >> > > Jiangjie (Becket) Qin
> > > > > > >> > > (On behalf of the Apache Flink PMC)
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards
> > > > > > >
> > > > > > > Jeff Zhang
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-17215) Clean the build document for Pyflink

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-17215:
---

 Summary: Clean the build document for Pyflink
 Key: FLINK-17215
 URL: https://issues.apache.org/jira/browse/FLINK-17215
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.9.4


Previously the build document of Pyflink has been adjusted in 
https://github.com/apache/flink/pull/11013, however, we can clean it now as we 
have pick the fix in FLINK-15638 into the 1.9. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Releasing Flink 1.10.1

[Hequn Cheng]

gt; >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>  - For 1.10.1 I am not completely sure,
> >> because
> >>>>>> users
> >>>>>>>>>> expect
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> upgrade
> >>>>>>>>>>>>>>>>>> that without config adjustments. That might not
> >>> be
> >>>>>>>>>> possible
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> change.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Ok, makes sense, I will revert it for 1.10 and
> >> only
> >>>>>> try to
> >>>>>>>>>>>> improve
> >>>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>>>> message and docs.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 12 Mar 2020, at 13:15, Stephan Ewen <
> >>>>>>>>>> se...@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> @Andrey about the increase in metaspace size
> >>>>>>>>>>>>>>>>>>  - I have no concerns for 1.11.0.
> >>>>>>>>>>>>>>>>>>  - For 1.10.1 I am not completely sure,
> >> because
> >>>>>> users
> >>>>>>>>>> expect
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> upgrade
> >>>>>>>>>>>>>>>>>> that without config adjustments. That might not
> >>> be
> >>>>>>>>>> possible
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> change.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 12:55 PM Andrey
> >> Zagrebin
> >>> <
> >>>>>>>>>>>>>>>>> azagrebin.apa...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> About "FLINK-16142 Memory Leak causes
> >> Metaspace
> >>>> OOM
> >>>>>>>>>> error
> >>>>>>>>>>> on
> >>>>>>>>>>>>>>> repeated
> >>>>>>>>>>>>>>>>>>> job”
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> My understanding that the issue is basically
> >>>> covered
> >>>>>>>>>> by:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> - [FLINK-16225] Metaspace Out Of Memory should
> >>> be
> >>>>>>>>>> handled as
> >>>>>>>>>>>>> Fatal
> >>>>>>>>>>>>>>>> Error
> >>>>>>>>>>>>>>>>>>> in TaskManager
> >>>>>>>>>>>>>>>>>>>  no full consensus there but improving error
> >>>>>> message
> >>>>>>>>>> for
> >>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>> task
> >>>>>>>>&g

Re: [DISCUSS] Releasing Flink 1.9.3

[Hequn Cheng]

+1 for the release and for Dian being the RM.
Thanks Jincheng for your continuous efforts on helping the releasing.

Best,
Hequn

On Wed, Apr 15, 2020 at 3:45 PM Till Rohrmann  wrote:

> Hi Dian,
>
> creating a new 1.9 bug fix release is a very good idea. +1 for creating it
> soon. Also thanks for volunteering as our release manager.
>
> Cheers,
> Till
>
> On Fri, Apr 10, 2020 at 7:27 AM Dian Fu  wrote:
>
> > Hi Jincheng,
> >
> > Thanks a lot for offering help. It would be very helpful. Thanks again!
> >
> > Regards,
> > Dian
> >
> > > 在 2020年4月10日，下午12:46，jincheng sun  写道：
> > >
> > > Hi Dian,
> > >
> > > Thanks for bring up the discussion. I would like to give you a hand at
> > the
> > > last stage when the RC is finished.  :)
> > >
> > > Best,
> > > Jincheng
> > >
> > >
> > >
> > > Dian Fu  于2020年4月10日周五 上午11:08写道：
> > >
> > >> Hi all,
> > >>
> > >> It has been more than two months since we released Flink 1.9.2. There
> > are
> > >> already 36 improvements/bugfixes in the release-1.9 branch.
> Therefore, I
> > >> propose to create the next bugfix release 1.9.3 for Flink 1.9.
> > >>
> > >> Most notable fixes are:
> > >>
> > >> - [FLINK-15085] HistoryServer dashboard config json out of sync
> > >> - [FLINK-15575] Azure Filesystem Shades Wrong Package "httpcomponents"
> > >> - [FLINK-15638] releasing/create_release_branch.sh does not set
> version
> > in
> > >> flink-python/pyflink/version.py
> > >> - [FLINK-16242] BinaryGeneric serialization error cause checkpoint
> > failure
> > >> - [FLINK-16573] Kinesis consumer does not properly shutdown
> > RecordFetcher
> > >> threads
> > >> - [FLINK-16047] Blink planner produces wrong aggregate results with
> > state
> > >> clean up
> > >> - [FLINK-16860] TableException: Failed to push filter into table
> source!
> > >> when upgrading flink to 1.9.2
> > >> - [FLINK-16916] The logic of NullableSerializer#copy is wrong
> > >> - [FLINK-16389] Bump Kafka 0.10 to 0.10.2.2
> > >> - [FLINK-15812] HistoryServer archiving is done in Dispatcher main
> > thread
> > >> - [FLINK-17062] Fix the conversion from Java row type to Python row
> type
> > >>
> > >> Furthermore, there is one blocker issue which should be merged before
> > >> 1.9.3 release:
> > >>
> > >> - [FLINK-16576] State inconsistency on restore with memory state
> > backends
> > >> (reviewing)
> > >>
> > >> I would volunteer as the release manager and kick off the release
> > process.
> > >> What do you think?
> > >>
> > >> Please let me know if there are any concerns or any other blocker
> issues
> > >> need to be fixed in 1.9.3. Thanks.
> > >>
> > >> Appreciated if there is any PMC could help with the final steps of the
> > >> release process.
> > >>
> > >> Regards,
> > >> Dian
> >
> >
>

Re: [VOTE] FLIP-121: Support Cython Optimizing Python User Defined Function

[Hequn Cheng]

+1 (binding)

Best,
Hequn

On Mon, Apr 13, 2020 at 1:50 PM jincheng sun 
wrote:

> +1(binding)
>
> Best,
> Jincheng
>
>
>
> Dian Fu  于2020年4月8日周三 上午9:59写道：
>
> > +1 (binding)
> >
> > > 在 2020年4月8日，上午9:53，Xingbo Huang  写道：
> > >
> > > Hi all,
> > > I would like to start the vote for FLIP-121[1], which is discussed and
> > > reached a consensus in the discussion thread[2].
> > >
> > > The vote will be open for at least 72h, unless there is an objection or
> > not
> > > enough votes.
> > >
> > > Best,
> > > Xingbo
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-121%3A+Support+Cython+Optimizing+Python+User+Defined+Function
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-121-Support-Cython-Optimizing-Python-User-Defined-Function-tt39577.html
> >
> >
>

Re: [VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[Hequn Cheng]

+1 (binding)

Best,
Hequn

On Mon, Apr 13, 2020 at 1:28 PM Jeff Zhang  wrote:

> +1
>
>
> jincheng sun  于2020年4月13日周一 下午1:24写道：
>
> > +1(binding)
> >
> > Best,
> > Jincheng
> >
> >
> >
> > Xingbo Huang  于2020年4月9日周四 下午8:27写道：
> >
> > > Hi Dian,
> > >
> > > +1 (non-binding)
> > > Thanks a lot for driving this.
> > >
> > > Best,
> > > Xingbo
> > >
> > > Dian Fu  于2020年4月8日周三 上午10:03写道：
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start the vote for FLIP-120[1] which is discussed and
> > reached
> > > > consensus in the discussion thread[2].
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > objection
> > > > or we have not received sufficient votes.
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120:+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > > > >
> > > > [2]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > > > <
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > > > >
> > >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Hequn Cheng]

It’s much more convenient now. Thanks you!

> On Apr 9, 2020, at 8:01 PM, Aljoscha Krettek  wrote:
> 
> That is very nice! Thanks for taking care of this ~3q
> 
> On 09.04.20 11:08, Dian Fu wrote:
>> Cool! Thanks Yun for this effort. Very useful feature.
>> Regards,
>> Dian
>>> 在 2020年4月9日，下午4:32，Yu Li  写道：
>>> 
>>> Great! Thanks for the efforts Yun.
>>> 
>>> Best Regards,
>>> Yu
>>> 
>>> 
>>> On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:
>>> 
 Thanks Yun,
 
 This's a great feature! I was surprised by the autolink feature yesterday
 (didn't know your work at that time).
 
 Best,
 Jark
 
 On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:
 
> Hi community
> 
> The autolink to Flink JIRA ticket has taken effect. You could refer to
 the
> commit details page[1] to see all Flink JIRA titles within commits has
 the
> hyper link underline. Moreover, you don't need to use markdown language
 to
> create hyper link to Flink JIRA ticket when discussing in the pull
> requests. e.g FLINK-16850 could point to the link instead of
 [FLINK-16850](
> https://issues.apache.org/jira/browse/FLINK-16850)
> 
> 
> [1] https://github.com/apache/flink/commits/master
> 
> Best
> Yun Tang
> 
> 
> From: Till Rohrmann 
> Sent: Thursday, April 2, 2020 23:11
> To: dev 
> Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos
> 
> Nice, this is a cool feature. Thanks for asking INFRA for it.
> 
> Cheers,
> Till
> 
> On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
> 
>> Hi community.
>> 
>> I noticed that Github supports autolink reference recently [1]. This is
>> helpful to allow developers could open Jira ticket link from pull
> requests
>> title directly when accessing github repo.
>> 
>> I have already created INFRA-20055 [2] to ask for configuration for
 seven
>> Flink related github repos. Hope it could be resolved soon 
>> 
>> 
>> [1]
>> 
> 
 https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
>> [2] https://issues.apache.org/jira/browse/INFRA-20055
>> 
>> Best
>> Yun Tang
>> 
> 
 
> 

Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released

[Hequn Cheng]

Thanks a lot for the release and your great job, Gordon!
Also thanks to everyone who made this release possible!

Best,
Hequn

On Tue, Apr 7, 2020 at 8:58 PM Tzu-Li (Gordon) Tai 
wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink Stateful Functions 2.0.0.
>
> Stateful Functions is an API that simplifies building distributed stateful
> applications.
> It's based on functions with persistent state that can interact
> dynamically with strong consistency guarantees.
>
> Please check out the release blog post for an overview of the release:
> https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Stateful Functions can be found at:
> https://search.maven.org/search?q=g:org.apache.flink%20statefun
>
> Python SDK for Stateful Functions published to the PyPI index can be found
> at:
> https://pypi.org/project/apache-flink-statefun/
>
> Official Docker image for building Stateful Functions applications is
> currently being published to Docker Hub.
> Dockerfiles for this release can be found at:
> https://github.com/apache/flink-statefun-docker/tree/master/2.0.0
> Progress for creating the Docker Hub repository can be tracked at:
> https://github.com/docker-library/official-images/pull/7749
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346878
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Cheers,
> Gordon
>

Re: [DISCUSS] FLIP-121: Support Cython Optimizing Python User Defined Function

[Hequn Cheng]

Hi,

+1 on integrating with Azure, it is consistent with the long term goal and
we are also going to switch from Travis to Azure.
The performance improvement is very impressive. Looking forward to the vote.

Best, Hequn

On Tue, Apr 7, 2020 at 9:10 PM Xingbo Huang  wrote:

> Hi everyone,
>
> Thanks all of you for the discussion.
> If there are no objections, I would like to start a vote thread tomorrow.
>
> Best,
> Xingbo
>
> jincheng sun  于2020年4月7日周二 下午6:22写道：
>
> > Hi Xingbo,
> >
> > Thanks for bring up this discussion!
> >
> > I agree with Robert, +1 for integration with Azure.
> >
> > Best,
> > Jincheng
> >
> > Dian Fu  于2020年4月7日周二 下午2:21写道：
> >
> > > Hi Xingbo,
> > >
> > > Thanks a lot for the great work. Big +1 to this feature. The
> performance
> > > improvement is impressive.
> > >
> > > Regards,
> > > Dian
> > >
> > > > 在 2020年4月7日，下午12:38，Robert Metzger  写道：
> > > >
> > > > Thank you for posting the FLIP.
> > > >
> > > > The proposed integration with Azure Pipelines looks good to me.
> > > >
> > > > On Tue, Mar 31, 2020 at 1:23 PM Xingbo Huang 
> > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> I would like to start a discussion thread on "Support Cython
> > Optimizing
> > > >> Python User Defined Function"
> > > >>
> > > >> Scalar Python UDF FLIP-58[1] has already been supported in release
> > 1.10
> > > and
> > > >> Python UDTF will be supported in the coming release of 1.11. In
> > release
> > > >> 1.10, we focused on supporting UDF features and did not make many
> > > >> optimizations in terms of performance. Although we have made a lot
> of
> > > >> optimizations in master[2], Cython can further greatly improve the
> > > >> performance of Python UDF.
> > > >>
> > > >> Robert Metzger, Jincheng Sun and I have discussed offline and have
> > > drafted
> > > >> the FLIP-121[3]. It includes the following items:
> > > >>
> > > >> - Introduces Cython implementation of coder and operations
> > > >>
> > > >> - Doc changes for building sdist and wheel packages from source code
> > > >>
> > > >> - Solutions for packages building
> > > >>
> > > >>
> > > >> Looking forward to your feedback!
> > > >>
> > > >> Best,
> > > >>
> > > >> Xingbo
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > > >>
> > > >> [2] https://issues.apache.org/jira/browse/FLINK-16747
> > > >>
> > > >> [3]
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-121%3A+Support+Cython+Optimizing+Python+User+Defined+Function
> > > >>
> > >
> > >
> >
>

Re: [ANNOUNCE] New Flink committer: Seth Wiesman

[Hequn Cheng]

Congratulations Seth!

Best, Hequn

On Tue, Apr 7, 2020 at 4:11 PM Fabian Hueske  wrote:

> Congrats Seth! Well deserved :-)
>
> Cheers, Fabian
>
> Am Di., 7. Apr. 2020 um 10:09 Uhr schrieb Yangze Guo :
>
> > Congratulations Seth!
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Apr 7, 2020 at 4:07 PM Jiayi Liao 
> wrote:
> > >
> > > >
> > > > Congratulations Seth :)
> > > >
> > > >
> >
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #6

[Hequn Cheng]

Thanks a lot for the new RC!

+1 (non-binding)

- Signatures and hash are correct.
- The source distribution contains no binaries.
- The source distribution is building properly with `-Prun-e2e-tests`
(JDK8).
- All POM files / README / Python SDK setup.py point to the same version.
- Verify license and notice.
  - Source distribution. Everything looks good and the jquery has been
added.
  - Jar artifacts. No missing dependencies, no version errors.
  - Python source distribution (source and wheel). It contains the license
and notice file.
- Flink Harness works in IDE.

Best,
Hequn

On Mon, Apr 6, 2020 at 10:05 PM Seth Wiesman  wrote:

> +1 (non-binding)
>
> legal / source
> - checked sources for binary files
> - checked license headers
>
> functional
> - built from source (mvn clean verify -Prun-e2e-tests)
> - built python sdk and ran tests
> - ran examples
> - deployed mixed python / java application on k8s with checkpointing.
> Failed TM's and watched it recover.
> - deployed application on Flink session cluster
> - created a savepoint using the bootstrap api and successfully used it to
> start an application.
>
> Seth
>
> On Mon, Apr 6, 2020 at 5:49 AM Igal Shilman  wrote:
>
> > +1 (non binding)
> >
> > legal / source:
> > - downloaded and verified the signature
> > - verified that pom and versions in the docs match
> > - no binary files in the distribution
> > - built and run e2e test with Java 8 and Java 11
> > - created a project from a maven archetype.
> >
> > functional:
> > - run all the examples
> > - deployed to Python greeter example to k8s
> > - enabled checkpointing, created an application with two Python
> functions,
> > that send both local and remote messages, restarted TMs randomly and
> > verified
> > the sequential output in the output kafka topic (exactly once test)
> > -  run the harness tests
> > -  run the ridesharing example in paraliisim 10 overnight
> > -  created a savepoint with the state bootstrapping tool and
> > successfully started a job from that.
> >
> > Kind regards,
> > Igal
> >
> > On Mon, Apr 6, 2020 at 10:23 AM Robert Metzger 
> > wrote:
> >
> > > Thanks a lot for preparing another RC!
> > >
> > > +1 (binding)
> > >
> > > - source archive looks fine (no binaries, copied sources are properly
> > > reported)
> > > - staging repository looks fine (bundled binaries seem documented,
> > versions
> > > are correct)
> > > - *mvn clean install *(mvn clean verify fails, "install" is required)
> w/
> > > e2e passes locally from source dir
> > >
> > >
> > >
> > >
> > > On Mon, Apr 6, 2020 at 9:22 AM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org>
> > > wrote:
> > >
> > > > FYI -
> > > > There are these open PRs to add blog posts and update the Flink
> website
> > > for
> > > > the Stateful Functions 2.0 release:
> > > > * https://github.com/apache/flink-web/pull/322
> > > > * https://github.com/apache/flink-web/pull/321
> > > >
> > > > On Mon, Apr 6, 2020 at 2:53 PM Konstantin Knauf <
> > > konstan...@ververica.com>
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > ** Functional **
> > > > > - Building from source dist with end-to-end tests enabled (mvn
> clean
> > > > verify
> > > > > -Prun-e2e-tests) passes (JDK 8)
> > > > > - Flink Harness works in IDE
> > > > > - Building Python SDK dist from source
> > > > >
> > > > > On Mon, Apr 6, 2020 at 5:12 AM Tzu-Li (Gordon) Tai <
> > > tzuli...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > ** Legal **
> > > > > > - checksums and GPG files match corresponding release files
> > > > > > - Source distribution does not contain binaries, contents are
> sane
> > > (no
> > > > > > .git* / .travis* / generated html content files)
> > > > > > - Bundled source LICENSEs and NOTICE looks good. Mentions bundled
> > > > > > font-awesome, jquery dependency in docs and copied sources from
> > > > fastutil
> > > > > (
> > > > > > http://fastutil.di.unimi.it/)
> > > > > > - Bundled LICENSEs and NOTICE files for Maven artifacts looks
> good.
> > > > > > Artifacts that do bundle dependencies are:
> > > statefun-flink-distribution,
> > > > > > statefun-ridesharing-example-simulator, statefun-flink-core
> (copied
> > > > > > sources). All non-ASLv2 deps have license files explicitly
> bundled.
> > > > > > - Python SDK distributions (source and wheel) contain ASLv2
> LICENSE
> > > and
> > > > > > NOTICE files (no bundled dependencies)
> > > > > > - All POMs / README / Python SDK setup.py / Dockerfiles / doc
> > configs
> > > > > point
> > > > > > to same version “2.0.0”
> > > > > > - README looks good
> > > > > >
> > > > > > ** Functional **
> > > > > > - Building from source dist with end-to-end tests enabled (mvn
> > clean
> > > > > verify
> > > > > > -Prun-e2e-tests) passes (JDK 8)
> > > > > > - Generated quickstart from archetype looks good (correct POM /
> > > > > Dockerfile
> > > > > > / service file)
> > > > > > - Examples run: Java Greeter / Java Ridesharing / Python Greeter
> /
> > > > 

Re: [VOTE] FLIP-114: Support Python UDF in SQL Client

[Hequn Cheng]

+1  (binding)

Best,
Hequn

On Fri, Apr 3, 2020 at 10:20 AM jincheng sun 
wrote:

> +1(binding)
>
> Best,
> Jincheng
>
>
>
> Xingbo Huang  于2020年4月1日周三 下午5:36写道：
>
> > Hi Wei,
> >
> > +1 (non-binding)
> > Thanks a lot for driving this.
> >
> > Best,
> > Xingbo
> >
> > Wei Zhong  于2020年3月31日周二 上午10:34写道：
> >
> > > Hi all,
> > >
> > > I would like to start the vote for FLIP-114[1] which is discussed and
> > > reached consensus in the discussion thread[2].
> > >
> > > The vote will be open for at least 72 hours. I'll try to close it by
> > > 2020-04-03 03:00 UTC, unless there is an objection or not enough votes.
> > >
> > > Best,
> > > Wei
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-114-Support-Python-UDF-in-SQL-Client-td38655.html
> > >
> > >
> >
>

Re: [ANNOUNCE] New Committers and PMC member

[Hequn Cheng]

Congratulations to all of you!

Best, Hequn

On Wed, Apr 1, 2020 at 5:07 PM Jiayi Liao  wrote:

> Congratulations to you all!
>
> On Wed, Apr 1, 2020 at 5:05 PM Arvid Heise  wrote:
>
> > Congratulations!
> >
> > On Wed, Apr 1, 2020 at 11:03 AM Dian Fu  wrote:
> >
> > > Congratulations to you all.
> > >
> > >
> > > > 在 2020年4月1日，下午5:00，Robert Metzger  写道：
> > > >
> > > > Welcome & congratulations to all of you!
> > > >
> > > >
> > > > On Wed, Apr 1, 2020 at 10:58 AM Jingsong Li 
> > > wrote:
> > > >
> > > >> Congratulations! Konstantin, Dawid, Zhijiang. Well deserved.
> > > >>
> > > >> Best,
> > > >> Jingsong Lee
> > > >>
> > > >> On Wed, Apr 1, 2020 at 4:52 PM Stephan Ewen 
> wrote:
> > > >>
> > > >>> Hi all!
> > > >>>
> > > >>> Happy to announce that over the last few weeks, several people in
> the
> > > >>> community joined in new roles:
> > > >>>
> > > >>>  - Konstantin Knauf joined as a committer. You may know him, for
> > > >> example,
> > > >>> from the weekly community updates.
> > > >>>
> > > >>>  - Dawid Wysakowicz joined the PMC. Dawid is one of the main
> > developers
> > > >> on
> > > >>> the Table API.
> > > >>>
> > > >>>  - Zhijiang Wang joined the PMC. Zhijiang is a veteran of Flink's
> > > >> network
> > > >>> / data shuffle system.
> > > >>>
> > > >>> A warm welcome to your new roles in the Flink project!
> > > >>>
> > > >>> Best,
> > > >>> Stephan
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Best, Jingsong Lee
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] FLIP-114: Support Python UDF in SQL Client

[Hequn Cheng]

Hi Wei,

Thanks a lot for the proposal! +1 for the VOTE.

Best,
Hequn



On Mon, Mar 30, 2020 at 3:31 PM Dian Fu  wrote:

> Thanks Wei for this work! +1 to bring up the VOTE thread.
>
> > 在 2020年3月30日，下午2:43，jincheng sun  写道：
> >
> > Hi Wei,
> >
> > +1, Thanks for this discussion which is crucial for SQL users to use
> > PyFlink. Would be great to bring up the VOTE thread.
> >
> > Best,
> > Jincheng
> >
> >
> > Wei Zhong  于2020年3月30日周一 下午2:38写道：
> >
> >> Hi everyone,
> >>
> >> Are there more comments about this FLIP? If not, I would like to bring
> up
> >> the VOTE.
> >>
> >> Best,
> >> Wei
> >>
> >>> 在 2020年3月9日，23:18，Xingbo Huang  写道：
> >>>
> >>> Hi Godfrey,
> >>> thanks for your suggestion.
> >>> I have added two examples how to use python UDF
> >>> in SQL and how to start sql-client.sh with full python dependencies In
> >> FLIP.
> >>>
> >>> Best,
> >>> Xingo
> >>>
> >>> godfrey he  于2020年3月9日周一 下午10:24写道：
> >>>
>  Hi Wei, thanks for the proposal.
> 
>  I think it's better to give two more examples, one is how to use
> python
> >> UDF
>  in SQL, another is how to start sql-client.sh with full python
>  dependencies.
> 
>  Best,
>  Godfrey
> 
>  Wei Zhong  于2020年3月9日周一 下午10:09写道：
> 
> > Hi everyone,
> >
> > I would like to start discussion about how to support Python UDF in
> SQL
> > Client.
> >
> > Flink Python UDF(FLIP-58[1]) has already been introduced in the
> release
>  of
> > 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
> >
> > SQL Client defines UDF via the environment file and has its own CLI
> > implementation to manage dependencies, but neither of which supports
>  Python
> > UDF. We want to introduce the support of Python UDF for SQL Client,
> > including the registration and the dependency management of Python
> UDF.
> >
> > Here is the design doc:
> >
> >
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
> >
> > Looking forward to your feedback!
> >
> > Best,
> > Wei
> >
> > [1]
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > [2]
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> >
> >
> 
> >>
> >>
>
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

[Hequn Cheng]

Hi,

@Gordon I have created the corresponding JIRAs and PRs for the license
problems:
- https://issues.apache.org/jira/browse/FLINK-16853
- https://issues.apache.org/jira/browse/FLINK-16854

@Igal @Gordon, Thanks a lot for the confirmation of the package name, that
addresses my concerns now.

Best,
Hequn

On Mon, Mar 30, 2020 at 4:10 AM Konstantin Knauf 
wrote:

> Hi Gordon,
>
> +1 (non-binding)
>
> * built from sources...check
> * build python SDK from sources...check
> * went through walkthrough based on local builds...check
>
> Cheers,
>
> Konstantin
>
>
> On Sun, Mar 29, 2020 at 6:30 PM Igal Shilman  wrote:
>
>> Hi @Hequn Cheng , and @Tzu-Li (Gordon) Tai
>> 
>> Indeed the names are generated automatically and are following a
>> convention.
>>
>> Cheers,
>> Igal.
>>
>> On Sun, Mar 29, 2020 at 6:28 PM Tzu-Li (Gordon) Tai 
>> wrote:
>>
>> > @Hequn Cheng 
>> > On second thought I think it would not hurt to create a RC4 to fix the
>> > version strings. Will fix those and create that now.
>> >
>> > On Mon, Mar 30, 2020 at 12:06 AM Tzu-Li (Gordon) Tai <
>> tzuli...@apache.org>
>> > wrote:
>> >
>> >>
>> >>
>> >> On Mon, Mar 30, 2020 at 12:00 AM Tzu-Li (Gordon) Tai <
>> tzuli...@apache.org>
>> >> wrote:
>> >>
>> >>> @Hequn Cheng 
>> >>> Good catches again!
>> >>>
>> >>> Regarding the incorrect versions:
>> >>> I think technically those would not be hard blockers, since what
>> matters
>> >>> is their inclusion and licenses being acknowledged.
>> >>> It would still be good to fix those though - could you open a ticket
>> for
>> >>> those?
>> >>>
>> >>
>> >> Here, I mean to fix those in future bugfix releases (if this RC does
>> >> indeed pass the vote as the official release).
>> >>
>> >>
>> >>>
>> >>> Regarding the names of the Python dists -
>> >>> the name of those distributions are generated from the setup.py file,
>> >>> and seems to be a convention used by PyPI.
>> >>> @i...@ververica.com   can you confirm here?
>> >>>
>> >>> On Sun, Mar 29, 2020 at 11:36 PM Hequn Cheng 
>> wrote:
>> >>>
>> >>>> Hi Gordon,
>> >>>>
>> >>>> Thanks a lot for the new RC. I found some new blockers about
>> licenses:
>> >>>>
>> >>>> - Module statefun-flink-distribution
>> >>>> com.google.protobuf:protobuf-java:3.8.0 (The version should be 3.7.1)
>> >>>>
>> >>>> - Module statefun-ridesharing-example-simulator
>> >>>> com.google.code.findbugs:jsr305:3.0.2:compile (Remove compile)
>> >>>> org.hibernate.validator:hibernate-validator:6.0.17 (Version should be
>> >>>> 6.0.17.Final)
>> >>>> org.jboss.logging:jboss-logging:3.3.2  (Version should be
>> 3.3.2.Final)
>> >>>>
>> >>>> Non-blocker feedback:
>> >>>> - py3 is added in the name of "whl dist" but it is missing in the
>> >>>> "source
>> >>>> dist"[1]. Should we make them consistent?
>> >>>>
>> >>>> Best,
>> >>>> Hequn
>> >>>>
>> >>>> [1]
>> >>>>
>> https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.0.0-rc3/
>> >>>>
>> >>>>
>> >>>> On Sun, Mar 29, 2020 at 3:31 PM Tzu-Li (Gordon) Tai <
>> >>>> tzuli...@apache.org>
>> >>>> wrote:
>> >>>>
>> >>>> > Hi everyone,
>> >>>> >
>> >>>> > Please review and vote on the *release candidate #3* for the
>> version
>> >>>> 2.0.0
>> >>>> > of Apache Flink Stateful Functions,
>> >>>> > as follows:
>> >>>> > [ ] +1, Approve the release
>> >>>> > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> >>>> >
>> >>>> > **Testing Guideline**
>> >>>> >
>> >>>> > You can find here [1] a doc that we can use for collaborating
>> testing
>> >>>> > efforts.
>> >>>> > The listed testing tasks

[jira] [Created] (FLINK-16854) Correct dependency versions in the NOTICE file of module statefun-ridesharing-example-simulator

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16854:
---

 Summary: Correct dependency versions in the NOTICE file of module 
statefun-ridesharing-example-simulator
 Key: FLINK-16854
 URL: https://issues.apache.org/jira/browse/FLINK-16854
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Reporter: Hequn Cheng
 Fix For: statefun-2.0


There are some dependencies with a wrong version in the NOTICE file:
{code}
com.google.code.findbugs:jsr305:3.0.2:compile (Remove compile)
org.hibernate.validator:hibernate-validator:6.0.17 (Version should be 
6.0.17.Final)
org.jboss.logging:jboss-logging:3.3.2  (Version should be 3.3.2.Final)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16853) Update the version of protobuf-java from 3.8.0 to 3.7.1 in the NOTICE file of module statefun-flink-distribution

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16853:
---

 Summary: Update the version of protobuf-java from 3.8.0 to 3.7.1 
in the NOTICE file of module statefun-flink-distribution
 Key: FLINK-16853
 URL: https://issues.apache.org/jira/browse/FLINK-16853
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Reporter: Hequn Cheng
 Fix For: statefun-2.0


In the NOTICE file, the version of protobuf-java should be 3.7.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

[Hequn Cheng]

Hi Gordon,

Thanks a lot for the new RC. I found some new blockers about licenses:

- Module statefun-flink-distribution
com.google.protobuf:protobuf-java:3.8.0 (The version should be 3.7.1)

- Module statefun-ridesharing-example-simulator
com.google.code.findbugs:jsr305:3.0.2:compile (Remove compile)
org.hibernate.validator:hibernate-validator:6.0.17 (Version should be
6.0.17.Final)
org.jboss.logging:jboss-logging:3.3.2  (Version should be 3.3.2.Final)

Non-blocker feedback:
- py3 is added in the name of "whl dist" but it is missing in the "source
dist"[1]. Should we make them consistent?

Best,
Hequn

[1] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.0.0-rc3/


On Sun, Mar 29, 2020 at 3:31 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on the *release candidate #3* for the version 2.0.0
> of Apache Flink Stateful Functions,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Testing Guideline**
>
> You can find here [1] a doc that we can use for collaborating testing
> efforts.
> The listed testing tasks in the doc also serve as a guideline in what to
> test for this release.
> If you wish to take ownership of a testing task, simply put your name down
> in the "Checked by" field of the task.
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are singed with the
> key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-2.0.0-rc3" [6] [7]
>
> **Extra Remarks**
>
> * Part of the release is also official Docker images for Stateful
> Functions. This can be a separate process, since the creation of those
> relies on the fact that we have distribution jars already deployed to
> Maven. I will follow-up with this after these artifacts are officially
> released.
> In the meantime, there is this discussion [8] ongoing about where to host
> the StateFun Dockerfiles.
> * The Flink Website and blog post is also being worked on (by Marta) as
> part of the release, to incorporate the new Stateful Functions project. We
> can follow up with a link to those changes afterwards in this vote thread,
> but that would not block you to test and cast your votes already.
> * Since the Flink website changes are still being worked on, you will not
> yet be able to find the Stateful Functions docs from there. Here are the
> links [9] [10].
>
> **Vote Duration**
>
> The vote will be open for at least 72 hours starting Monday
> *(target end date is Wednesday, April 1st).*
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1]
>
> https://docs.google.com/document/d/1P9yjwSbPQtul0z2AXMnVolWQbzhxs68suJvzR6xMjcs/edit?usp=sharing
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.0.0-rc3/
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1342/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346878
> [6]
>
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=commit;h=752e07fd9987ee430eb9d1c1d3fadff632ef9213
> [7] https://github.com/apache/flink-statefun/tree/release-2.0.0-rc3
> [8]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Creating-a-new-repo-to-host-Stateful-Functions-Dockerfiles-td39342.html
> [9] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> [10] https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/
>
> TIP: You can create a `settings.xml` file with these contents:
>
> """
> 
>   
> flink-statefun-2.0.0
>   
>   
> 
>   flink-statefun-2.0.0
>   
> 
>   flink-statefun-2.0.0
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1342/
> 
> 
> 
>   archetype
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1342/
> 
> 
>   
> 
>   
> 
> """
>
> And reference that in you maven commands via `--settings
> path/to/settings.xml`.
> This is useful for creating a quickstart based on the staged release and
> for building against the staged jars.
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #2

[Hequn Cheng]

Thanks a lot for such a quick update! @Gordon

Best, Hequn

On Sun, Mar 29, 2020 at 3:31 PM Tzu-Li (Gordon) Tai 
wrote:

> All blockers are resolved.
>
> This vote thread is cancelled.
> There is a new vote thread for RC3:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-Stateful-Functions-Release-2-0-0-release-candidate-3-td39424.html
>
> On Sat, Mar 28, 2020 at 7:00 PM Tzu-Li (Gordon) Tai 
> wrote:
>
> > After a check-through on the licenses, there are in total the following 3
> > blockers due to licensing issues:
> >
> > - https://issues.apache.org/jira/browse/FLINK-16841
> > - https://issues.apache.org/jira/browse/FLINK-16842
> > - https://issues.apache.org/jira/browse/FLINK-16843
> >
> > Will open a new RC as soon as those are addressed.
> >
> > On Sat, Mar 28, 2020 at 1:34 AM Tzu-Li (Gordon) Tai  >
> > wrote:
> >
> >> Hi Hequn,
> >>
> >> That's a good catch.
> >>
> >> Unfortunately, the spring boot dependency there, while itself being
> ASLv2
> >> licensed, pulls in other dependencies that are not ASLv2.
> >> That would indeed make this problem a blocker.
> >>
> >> I'll do a thorough check again on the Maven artifacts that do bundle
> >> dependencies, before creating a new RC. AFAIK, there should be no more
> >> other than:
> >> - statefun-flink-distribution
> >> - statefun-ridesharing-example-simulator
> >>
> >> BR,
> >> Gordon
> >>
> >> On Fri, Mar 27, 2020 at 10:41 PM Hequn Cheng  wrote:
> >>
> >>> Thanks Gordon for the release and the nice release checking guide!
> >>>
> >>> It seems the NOTICE file is missing in the
> >>> `statefun-ridesharing-example-simulator` module while it bundles
> >>> dependencies like
> >>> `org.springframework.boot:spring-boot-loader:2.1.6.RELEASE`.
> >>>
> >>> Best,
> >>> Hequn
> >>>
> >>> On Fri, Mar 27, 2020 at 3:35 PM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
> >>> >
> >>> wrote:
> >>>
> >>> > Hi everyone,
> >>> >
> >>> > Please review and vote on the release candidate #2 for the version
> >>> 2.0.0 of
> >>> > Apache Flink Stateful Functions,
> >>> > as follows:
> >>> > [ ] +1, Approve the release
> >>> > [ ] -1, Do not approve the release (please provide specific comments)
> >>> >
> >>> > **Testing Guideline**
> >>> >
> >>> > You can find here [1] a doc that we can use for collaborating testing
> >>> > efforts.
> >>> > The listed testing tasks in the doc also serve as a guideline in what
> >>> to
> >>> > test for this release.
> >>> > If you wish to take ownership of a testing task, simply put your name
> >>> down
> >>> > in the "Checked by" field of the task.
> >>> >
> >>> > **Release Overview**
> >>> >
> >>> > As an overview, the release consists of the following:
> >>> > a) Stateful Functions canonical source distribution, to be deployed
> to
> >>> the
> >>> > release repository at dist.apache.org
> >>> > b) Stateful Functions Python SDK distributions to be deployed to PyPI
> >>> > c) Maven artifacts to be deployed to the Maven Central Repository
> >>> >
> >>> > **Staging Areas to Review**
> >>> >
> >>> > The staging areas containing the above mentioned artifacts are as
> >>> follows,
> >>> > for your review:
> >>> > * All artifacts for a) and b) can be found in the corresponding dev
> >>> > repository at dist.apache.org [2]
> >>> > * All artifacts for c) can be found at the Apache Nexus Repository
> [3]
> >>> >
> >>> > All artifacts are singed with the
> >>> > key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
> >>> >
> >>> > Other links for your review:
> >>> > * JIRA release notes [5]
> >>> > * source code tag "release-2.0.0-rc2" [6] [7]
> >>> >
> >>> > **Extra Remarks**
> >>> >
> >>> > * Part of the release is also official Docker images for Stateful
> >>> > Functions. This can be a separate process, since the creat

Re: contribute to Apache Flink

[Hequn Cheng]

Hi Leping,

Welcome to the community!

You no longer need contributor permissions. You can simply create a JIRA
ticket and ask to be assigned to work on it. For some reasons[1], only
committers can assign a
Jira ticket now. You can also take a look at the Flink's contribution
guidelines [2] for more
information.

Best,
Hequn

[1]
https://flink.apache.org/contributing/contribute-code.html#create-jira-ticket-and-reach-consensus
[2] https://flink.apache.org/contributing/how-to-contribute.html

On Sat, Mar 28, 2020 at 7:09 AM Leping Huang 
wrote:

> Hi Guys,
>
> I want to contribute to Apache Flink.
> Would you please give me the permission as a contributor?
> My JIRA ID is soundhearer.
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #2

[Hequn Cheng]

Thanks Gordon for the release and the nice release checking guide!

It seems the NOTICE file is missing in the
`statefun-ridesharing-example-simulator` module while it bundles
dependencies like
`org.springframework.boot:spring-boot-loader:2.1.6.RELEASE`.

Best,
Hequn

On Fri, Mar 27, 2020 at 3:35 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version 2.0.0 of
> Apache Flink Stateful Functions,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Testing Guideline**
>
> You can find here [1] a doc that we can use for collaborating testing
> efforts.
> The listed testing tasks in the doc also serve as a guideline in what to
> test for this release.
> If you wish to take ownership of a testing task, simply put your name down
> in the "Checked by" field of the task.
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are singed with the
> key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-2.0.0-rc2" [6] [7]
>
> **Extra Remarks**
>
> * Part of the release is also official Docker images for Stateful
> Functions. This can be a separate process, since the creation of those
> relies on the fact that we have distribution jars already deployed to
> Maven. I will follow-up with this after these artifacts are officially
> released.
> In the meantime, there is this discussion [8] ongoing about where to host
> the StateFun Dockerfiles.
> * The Flink Website and blog post is also being worked on (by Marta) as
> part of the release, to incorporate the new Stateful Functions project. We
> can follow up with a link to those changes afterwards in this vote thread,
> but that would not block you to test and cast your votes already.
> * Since the Flink website changes are still being worked on, you will not
> yet be able to find the Stateful Functions docs from there. Here are the
> links [9] [10].
>
> **Vote Duration**
>
> The vote will be open for at least 72 hours *(target end date is next
> Tuesday, April 31).*
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1]
>
> https://docs.google.com/document/d/1P9yjwSbPQtul0z2AXMnVolWQbzhxs68suJvzR6xMjcs/edit?usp=sharing
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.0.0-rc2/
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1340/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346878
> [6]
>
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=commit;h=14ce58048a3dda792f2329cf14d30aa952f6cb24
> [7] https://github.com/apache/flink-statefun/tree/release-2.0.0-rc2
> [8]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Creating-a-new-repo-to-host-Stateful-Functions-Dockerfiles-td39342.html
> [9] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> [10] https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/
>
> TIP: You can create a `settings.xml` file with these contents:
>
> """
> 
>   
> flink-statefun-2.0.0
>   
>   
> 
>   flink-statefun-2.0.0
>   
> 
>   flink-statefun-2.0.0
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1340/
> 
> 
> 
>   archetype
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1340/
> 
> 
>   
> 
>   
> 
> """
>
> And reference that in you maven commands via `--settings
> path/to/settings.xml`.
> This is useful for creating a quickstart based on the staged release and
> for building against the staged jars.
>

Re: [DISCUSS] Creating a new repo to host Stateful Functions Dockerfiles

[Hequn Cheng]

+1 for a separate repository.
The dedicated `flink-docker` repo works fine now. We can do it similarly.

Best,
Hequn

On Fri, Mar 27, 2020 at 1:16 AM Till Rohrmann  wrote:

> +1 for a separate repository.
>
> Cheers,
> Till
>
> On Thu, Mar 26, 2020 at 5:13 PM Ufuk Celebi  wrote:
>
> > +1.
> >
> > The repo creation process is a light-weight, automated process on the ASF
> > side. When Patrick Lucas contributed docker-flink back to the Flink
> > community (as flink-docker), there was virtually no overhead in creating
> > the repository. Reusing build scripts should still be possible at the
> cost
> > of some duplication which is fine imo.
> >
> > – Ufuk
> >
> > On Thu, Mar 26, 2020 at 4:18 PM Stephan Ewen  wrote:
> > >
> > > +1 to a separate repository.
> > >
> > > It seems to be best practice in the docker community.
> > > And since it does not add overhead, why not go with the best practice?
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > > On Thu, Mar 26, 2020 at 4:15 PM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
> > >
> > wrote:
> > >>
> > >> Hi Flink devs,
> > >>
> > >> As part of a Stateful Functions release, we would like to publish
> > Stateful
> > >> Functions Docker images to Dockerhub as an official image.
> > >>
> > >> Some background context on Stateful Function images, for those who are
> > not
> > >> familiar with the project yet:
> > >>
> > >>- Stateful Function images are built on top of the Flink official
> > >>images, with additional StateFun dependencies being added.
> > >>You can take a look at the scripts we currently use to build the
> > images
> > >>locally for development purposes [1].
> > >>- They are quite important for user experience, since building a
> > Docker
> > >>image is the recommended go-to deployment mode for StateFun user
> > >>applications [2].
> > >>
> > >>
> > >> A prerequisite for all of this is to first decide where we host the
> > >> Stateful Functions Dockerfiles,
> > >> before we can proceed with the process of requesting a new official
> > image
> > >> repository at Dockerhub.
> > >>
> > >> We’re proposing to create a new dedicated repo for this purpose,
> > >> with the name `apache/flink-statefun-docker`.
> > >>
> > >> While we did initially consider integrating the StateFun Dockerfiles
> to
> > be
> > >> hosted together with the Flink ones in the existing
> > `apache/flink-docker`
> > >> repo, we had the following concerns:
> > >>
> > >>- In general, it is a convention that each official Dockerhub image
> > is
> > >>backed by a dedicated source repo hosting the Dockerfiles.
> > >>- The `apache/flink-docker` repo already has quite a few dedicated
> > >>tooling and CI smoke tests specific for the Flink images.
> > >>- Flink and StateFun have separate versioning schemes and
> independent
> > >>release cycles. A new Flink release does not necessarily require a
> > >>“lock-step” to release new StateFun images as well.
> > >>- Considering the above all-together, and the fact that creating a
> > new
> > >>repo is rather low-effort, having a separate repo would probably
> make
> > more
> > >>sense here.
> > >>
> > >>
> > >> What do you think?
> > >>
> > >> Cheers,
> > >> Gordon
> > >>
> > >> [1]
> > >>
> >
> >
> https://github.com/apache/flink-statefun/blob/master/tools/docker/build-stateful-functions.sh
> > >> [2]
> > >>
> >
> >
> https://ci.apache.org/projects/flink/flink-statefun-docs-master/deployment-and-operations/packaging.html
> >
>

[jira] [Created] (FLINK-16786) Fix pyarrow version incompatible problem

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16786:
---

 Summary: Fix pyarrow version incompatible problem
 Key: FLINK-16786
 URL: https://issues.apache.org/jira/browse/FLINK-16786
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.11.0


As reported in FLINK-16483, we should make the version of pyarrow consistent 
between pyflink and beam. Other dependencies should also be checked. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16766) Support create StreamTableEnvironment without passing StreamExecutionEnvironment

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16766:
---

 Summary: Support create StreamTableEnvironment without passing 
StreamExecutionEnvironment
 Key: FLINK-16766
 URL: https://issues.apache.org/jira/browse/FLINK-16766
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.9.3, 1.10.1, 1.11.0


Currently, when we create a BatchTableEnvironment, the ExecutionEnvironment is 
an optional parameter, while for the StreamTableEnvironment, the 
ExecutionEnvironment is not optional. We should make them consistent



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16765) Replace all BatchTableEnvironment to StreamTableEnvironment in the document of PyFlink

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16765:
---

 Summary: Replace all BatchTableEnvironment to 
StreamTableEnvironment in the document of PyFlink
 Key: FLINK-16765
 URL: https://issues.apache.org/jira/browse/FLINK-16765
 Project: Flink
  Issue Type: Improvement
  Components: API / Python, Documentation
Reporter: Hequn Cheng
 Fix For: 1.10.1, 1.11.0


For example, in the 
[tutorial|https://ci.apache.org/projects/flink/flink-docs-master/getting-started/walkthroughs/python_table_api.html],
 replace the BatchTableEnvironment to StreamTableEnvironment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16763) Should not use BatchTableEnvironment for Python UDF in the document of flink-1.10

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16763:
---

 Summary: Should not use BatchTableEnvironment for Python UDF in 
the document of flink-1.10
 Key: FLINK-16763
 URL: https://issues.apache.org/jira/browse/FLINK-16763
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Reporter: Hequn Cheng
 Fix For: 1.10.1


Currently, for flink-1.10, Python UDF is not supported in old planner under 
batch mode, so we should not use BatchTableEnvironment as an example for Python 
UDF in the document[1] of flink-1.10.  The problem may also exist in other 
places in the document and should be checked.

[1] 
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/python/python_udfs.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16761) Return JobExecutionResult for Python ExecutionEnvironment and TableEnvironment

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16761:
---

 Summary: Return JobExecutionResult for Python ExecutionEnvironment 
and TableEnvironment
 Key: FLINK-16761
 URL: https://issues.apache.org/jira/browse/FLINK-16761
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.10.0, 1.9.2
Reporter: Hequn Cheng
 Fix For: 1.9.3, 1.10.1, 1.11.0


For ExecutionEnvironment and TableEnvironment, the JobExecutionResult should be 
returned in order to make jobID or other information accessible for users. This 
is also the behavior of Java and we should make python consistent with it. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16674) Add documentation about how to use user-defined metrics for Python UDF

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16674:
---

 Summary: Add documentation about how to use user-defined metrics 
for Python UDF
 Key: FLINK-16674
 URL: https://issues.apache.org/jira/browse/FLINK-16674
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python, Documentation
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16673) Support Counter, Gauge, Meter, Distribution metric type for Python UDTF

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16673:
---

 Summary: Support Counter, Gauge, Meter, Distribution metric type 
for Python UDTF
 Key: FLINK-16673
 URL: https://issues.apache.org/jira/browse/FLINK-16673
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.11.0


Support Counter, Gauge, Meter, Distribution metric type for Python UDTF



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16672) Support Counter, Gauge, Meter, Distribution metric type for Python UDF

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16672:
---

 Summary: Support Counter, Gauge, Meter, Distribution metric type 
for Python UDF
 Key: FLINK-16672
 URL: https://issues.apache.org/jira/browse/FLINK-16672
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.11.0


Support Counter, Gauge, Meter, Distribution metric type for Python UDF



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16671) Support define scopes and variables on Python metric group

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16671:
---

 Summary: Support define scopes and variables on Python metric group
 Key: FLINK-16671
 URL: https://issues.apache.org/jira/browse/FLINK-16671
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.11.0


In this ticket, we are going to support define scopes and variables on Python 
metric group, i.e., adding `get_metric_group()` on `FunctionContext`, adding 
`add_group`, `get_scope_components`, `get_all_variables` and 
`get_metric_identifier` in `MetricGroup`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16670) Support User-Defined Metrics in Python UDF

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16670:
---

 Summary: Support User-Defined Metrics in  Python UDF
 Key: FLINK-16670
 URL: https://issues.apache.org/jira/browse/FLINK-16670
 Project: Flink
  Issue Type: New Feature
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.11.0


This is the umbrella Jira for FLIP112, which intends to support User-Defined 
Metrics in  Python UDF.

FLIP wiki page: 
[https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF]

FLIP-58 adds the support for Python UDFs, but user-defined metrics have not 
been supported yet. With metrics, users can report and monitor the UDF status 
to get a deeper understanding of the execution. In this FLIP, we want to 
support metrics for Python UDFs. In this FLIP we propose to:
 * Support user-defined metrics including Counters, Gauges, Meters, 
Distributions in Python UDFs. (Note: Histogram is not supported in this FLIP, 
instead, Distributions is supported to report statistics about the distribution 
of value. See more in the Distribution section.)
 * Support defining user scopes.
 * Support defining user variables.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

Thanks all for the votes.

So far, we have
   - 3 binding +1 votes (Jincheng, Dian, and Hequn)
   - 1 non-binding +1 votes (Wei)
   - 0 -1 votes

I will move the FLIP to accepted!

Best, Hequn

On Thu, Mar 19, 2020 at 2:07 PM Hequn Cheng  wrote:

> +1 (binding)
>
> Best, Hequn
>
> On Thu, Mar 19, 2020 at 10:39 AM Wei Zhong  wrote:
>
>> +1 (non-binding)
>>
>> Best,
>> Wei
>>
>> > 在 2020年3月17日，19:15，Dian Fu  写道：
>> >
>> > +1 (binding)
>> >
>> > On Tue, Mar 17, 2020 at 10:35 AM jincheng sun > >
>> > wrote:
>> >
>> >> +1
>> >>
>> >> Best,
>> >> Jincheng
>> >>
>> >>
>> >>
>> >> Hequn Cheng  于2020年3月16日周一 上午10:01写道：
>> >>
>> >>> Hi everyone,
>> >>>
>> >>> I'd like to start the vote of FLIP-112[1] which is discussed and
>> reached
>> >>> consensus in the discussion thread[2].
>> >>> The vote will be open for at least 72 hours. Unless there is an
>> >> objection,
>> >>> I will try to close it by March 19, 2020 03:00 UTC if we have received
>> >>> sufficient votes.
>> >>>
>> >>> Thanks,
>> >>> Hequn
>> >>>
>> >>> [1]
>> >>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
>> >>> [2]
>> >>>
>> >>>
>> >>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-112-Support-User-Defined-Metrics-for-Python-UDF-td38609.html
>> >>>
>> >>
>>
>>

Re: [VOTE] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

+1 (binding)

Best, Hequn

On Thu, Mar 19, 2020 at 10:39 AM Wei Zhong  wrote:

> +1 (non-binding)
>
> Best,
> Wei
>
> > 在 2020年3月17日，19:15，Dian Fu  写道：
> >
> > +1 (binding)
> >
> > On Tue, Mar 17, 2020 at 10:35 AM jincheng sun 
> > wrote:
> >
> >> +1
> >>
> >> Best,
> >> Jincheng
> >>
> >>
> >>
> >> Hequn Cheng  于2020年3月16日周一 上午10:01写道：
> >>
> >>> Hi everyone,
> >>>
> >>> I'd like to start the vote of FLIP-112[1] which is discussed and
> reached
> >>> consensus in the discussion thread[2].
> >>> The vote will be open for at least 72 hours. Unless there is an
> >> objection,
> >>> I will try to close it by March 19, 2020 03:00 UTC if we have received
> >>> sufficient votes.
> >>>
> >>> Thanks,
> >>> Hequn
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
> >>> [2]
> >>>
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-112-Support-User-Defined-Metrics-for-Python-UDF-td38609.html
> >>>
> >>
>
>

Re: [VOTE] FLIP-106: Support Python UDF in SQL Function DDL

[Hequn Cheng]

+1 (binding)

Best,
Hequn

> On Mar 17, 2020, at 5:03 PM, Benchao Li  wrote:
> 
> +1 (non-binding)
> 
> BTW it's in the same thread in my gmail too.
> 
> 
> 
> Kurt Young  于2020年3月17日周二 上午11:47写道：
> 
>> Looks like I hit the gmail's bug again...
>> 
>> Best,
>> Kurt
>> 
>> 
>> On Tue, Mar 17, 2020 at 11:11 AM Wei Zhong  wrote:
>> 
>>> Hi Kurt,
>>> 
>>> This vote thread is independent from my side[1]. If this thread is
>>> combined with another thread from your side, you can try to change the
>> mail
>>> client.
>>> 
>>> Best,
>>> Wei
>>> 
>>> [1]
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-106-Support-Python-UDF-in-SQL-Function-DDL-td38895.html
>>> <
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-106-Support-Python-UDF-in-SQL-Function-DDL-td38895.html
 
>>> 
 在 2020年3月17日，10:57，Kurt Young  写道：
 
 Hi, please use a dedicated vote thread.
 
 Best,
 Kurt
 
 
 On Tue, Mar 17, 2020 at 10:36 AM jincheng sun <
>> sunjincheng...@gmail.com>
 wrote:
 
> +1
> 
> Best,
> Jincheng
> 
> 
> 
> Wei Zhong  于2020年3月13日周五 下午9:04写道：
> 
>> Hi all,
>> 
>> I would like to start the vote for FLIP-106[1] which is discussed and
>> reached consensus in the discussion thread[2].
>> 
>> The vote will be open for at least 72 hours. I'll try to close it by
>> 2020-03-18 14:00 UTC, unless there is an objection or not enough
>> votes.
>> 
>> Best,
>> Wei
>> 
>> [1]
>> 
> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
>> [2]
>> 
> 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-106-Support-Python-UDF-in-SQL-Function-DDL-td38107.html
>> 
>> 
> 
>>> 
>>> 
>> 
> 
> 
> -- 
> 
> Benchao Li
> School of Electronics Engineering and Computer Science, Peking University
> Tel:+86-15650713730
> Email: libenc...@gmail.com; libenc...@pku.edu.cn

[VOTE] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

Hi everyone,

I'd like to start the vote of FLIP-112[1] which is discussed and reached
consensus in the discussion thread[2].
The vote will be open for at least 72 hours. Unless there is an objection,
I will try to close it by March 19, 2020 03:00 UTC if we have received
sufficient votes.

Thanks,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-112-Support-User-Defined-Metrics-for-Python-UDF-td38609.html

Re: [DISCUSS] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

Hi everyone,

If there are no more concerns, I will raise the vote next week on Monday.
Thank you all for the feedback.

Best,
Hequn


On Fri, Mar 13, 2020 at 9:00 AM jincheng sun 
wrote:

> Hi Hequn,
>
> +1, thank you for this discussion, and metrics are very important for
> monitoring the running state of Python UDF.
>
> Best,
> Jincheng
>
>
> Hequn Cheng  于2020年3月12日周四 下午11:39写道：
>
> > @Dian Fu   Thanks a lot for the advice. The FLIP
> > page has been updated.
> >
> > Best,
> > Hequn
> >
> > On Thu, Mar 12, 2020 at 4:19 PM Dian Fu  wrote:
> >
> > > Hi Hequn,
> > >
> > > Thanks for driving this. +1 to this feature.
> > >
> > > Just one minor comment: It seems that we will add an API
> get_metric_group
> > > for the Python class FunctionContext, could you update the FLIP
> > reflecting
> > > this?
> > >
> > > Thanks,
> > > Dian
> > >
> > > > 在 2020年3月10日，下午3:38，Wei Zhong  写道：
> > > >
> > > > Hi Hequn,
> > > >
> > > > Thanks for driving this. +1 for the metrics support for Python UDF,
> > > which makes it much easier for users to monitor the execution of Python
> > > UDFs.
> > > >
> > > > Best,
> > > > Wei
> > > >
> > > >
> > > >> 在 2020年3月10日，15:32，Xingbo Huang  写道：
> > > >>
> > > >> Hi Hequn,
> > > >> thanks for drafting the FLIP and kicking off the discussion.
> > > >>
> > > >> +1 for this feature.
> > > >> I think this feature will be extremely convenient for PyFlink users.
> > > >>
> > > >> Best,
> > > >> Xingbo
> > > >>
> > > >> Hequn Cheng  于2020年3月9日周一 上午11:32写道：
> > > >>
> > > >>> Hi everyone,
> > > >>>
> > > >>> FLIP-58 adds the support for Python UDFs, but user-defined metrics
> > > >>> have not been supported yet. With metrics, users can report and
> > monitor
> > > >>> the UDF status to get a deeper understanding of the execution,
> > > >>> so in this FLIP, we want to support metrics for Python UDFs.
> > > >>>
> > > >>> Previously, Jincheng and I discussed offline about the support of
> > > >>> metrics for Python UDFs. We'd like to achieve three goals for
> > > >>> supporting metrics for Python UDFs:
> > > >>> - Support user-defined metrics including Counters, Gauges, Meters,
> > > >>> Distributions in Python UDFs.
> > > >>> - Support defining user scopes.
> > > >>> - Support defining user variables.
> > > >>>
> > > >>> More details can be found in the FLIP wiki page[1] and we are
> looking
> > > >>> forward
> > > >>> to your feedback.
> > > >>>
> > > >>> Best,
> > > >>> Hequn
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
> > > >>>
> > > >
> > >
> > >
> >
>

Re: [DISCUSS] FLIP-106: Support Python UDF in SQL Function DDL

[Hequn Cheng]

Big +1 on this feature! It would be great to extend the usage of Python UDF
in SQL scenarios.
The design doc looks good from my side now. Thank you for the update.

Best,
Hequn

On Tue, Mar 10, 2020 at 3:50 PM Wei Zhong  wrote:

> Hi Timo,
>
> Thanks for your reply.
>
> If we aim for the option 1, it makes sense for me to include the change in
> this FLIP as the option 1 does not change any public API. I'll update the
> FLIP page to illustrate this.
>
> Best,
> Wei
>
> > 在 2020年3月9日，17:58，Timo Walther  写道：
> >
> > Hi Wei,
> >
> > I agree with Dawid that we should defer the instantiation of temporary
> functions to compile time. In the long-term, we would like to integrate
> FunctionCatalog as a component of CatalogManager and unify the handling of
> catalog objects as much as possible.
> >
> > We should aim for your proposed option 1. For fluent definition of
> functions in Table API, we would still like to offer passing instances like
> `t.select(call(new ScalarFunction() { ... }))` that would be registered as
> temporary system functions.
> >
> > Regrds,
> > Timo
> >
> >
> > On 09.03.20 09:24, Wei Zhong wrote:
> >> Hi Dawid,
> >> I think defering the instantiation of temporary functions to compile
> time is quite a good idea but needs further discussion. As it is orthogonal
> with this FLIP, we could continue the discussion in a new thread later.
> What do you think?
> >> Best,
> >> Wei
> >>> 在 2020年3月5日，21:11，Wei Zhong  写道：
> >>>
> >>> Hi Dawid,
> >>>
> >>> Thanks for your suggestion.
> >>>
> >>> After some investigation, there are two designs in my mind about how
> to defer the instantiation of temporary system function and temporary
> catalog function to compile time.
> >>>
> >>> 1. FunctionCatalog accepts both FunctionDefinitions and uninstantiated
> temporary functions. The uninstantiated temporary functions will be
> instantiated when compiling. There is no public API change in this design,
> but the FunctionCatalog needs to store and process both FunctionDefinitions
> and uninstantiated temporary functions.
> >>>
> >>> 2. FunctionCatalog accepts only uninstantiated temporary functions. In
> this design we need to remove those APIs that accepts FunctionDefinitions
> from TableEnvironment, i.e. `void createTemporaryFunction(String path,
> UserDefinedFunction functionInstance)` and `void
> createTemporarySystemFunction(String name, UserDefinedFunction
> functionInstance)`. But the FunctionCatalog only needs to store and process
> uninstantiated temporary functions.
> >>>
> >>> As I don't know the details about the plan to store temporary
> functions as catalog functions instead of FunctionDefinitions, I'm not sure
> which solution fits more. It would be great if you could share more details
> or share some thoughts on these two solutions?
> >>>
> >>> Best,
> >>> Wei
> >>>
>  在 2020年3月4日，16:17，Dawid Wysakowicz  写道：
> 
>  Hi all,
>  I had a really quick look and from my perspective the proposal looks
> fine.
>  I share Jarks opinion that the instantiation could be done at a later
>  stage. I agree with Wei it requires some changes in the internal
>  implementation of the FunctionCatalog, to store temporary functions as
>  catalog functions instead of FunctionDefinitions, but we have that on
> our
>  agenda anyway. I would suggest investigating if we could do that as
> part of
>  this flip already. Nevertheless this in theory can be also done later.
> 
>  Best,
>  Dawid
> 
>  On Mon, 2 Mar 2020, 14:58 Jark Wu,  wrote:
> 
> > Thanks for the explanation, Wei!
> >
> > On Mon, 2 Mar 2020 at 20:59, Wei Zhong 
> wrote:
> >
> >> Hi Jark,
> >>
> >> Thanks for your suggestion.
> >>
> >> Actually, the timing of starting a Python process depends on the UDF
> > type,
> >> because the Python process is used to provide the necessary
> information
> > to
> >> instantiate the FunctionDefinition object of the Python UDF. For
> catalog
> >> function, the FunctionDefinition will be instantiated when
> compiling the
> >> job, which means the Python process is required during the
> compilation
> >> instead of the registeration. For temporary system function and
> temporary
> >> catalog function, the FunctionDefinition will be instantiated
> during the
> >> UDF registeration, so the Python process need to be started at that
> time.
> >>
> >> But this FLIP will only support registering the temporary system
> function
> >> and temporary catalog function in SQL DDL because registering
> Python UDF
> > to
> >> catalog is not supported yet. We plan to support the registeration
> of
> >> Python catalog function (via Table API and SQL DDL) in a separate
> FLIP.
> >> I'll add a non-goal section to the FLIP page to illustrate this.
> >>
> >> Best,
> >> Wei
> >>
> >>
> >>> 在 2020年3月2日，15:11，Jark Wu  写道：
> >>>
> >>> Hi Weizhong,
> >>>
> 

Re: [DISCUSS] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

@Dian Fu   Thanks a lot for the advice. The FLIP
page has been updated.

Best,
Hequn

On Thu, Mar 12, 2020 at 4:19 PM Dian Fu  wrote:

> Hi Hequn,
>
> Thanks for driving this. +1 to this feature.
>
> Just one minor comment: It seems that we will add an API get_metric_group
> for the Python class FunctionContext, could you update the FLIP reflecting
> this?
>
> Thanks,
> Dian
>
> > 在 2020年3月10日，下午3:38，Wei Zhong  写道：
> >
> > Hi Hequn,
> >
> > Thanks for driving this. +1 for the metrics support for Python UDF,
> which makes it much easier for users to monitor the execution of Python
> UDFs.
> >
> > Best,
> > Wei
> >
> >
> >> 在 2020年3月10日，15:32，Xingbo Huang  写道：
> >>
> >> Hi Hequn,
> >> thanks for drafting the FLIP and kicking off the discussion.
> >>
> >> +1 for this feature.
> >> I think this feature will be extremely convenient for PyFlink users.
> >>
> >> Best,
> >> Xingbo
> >>
> >> Hequn Cheng  于2020年3月9日周一 上午11:32写道：
> >>
> >>> Hi everyone,
> >>>
> >>> FLIP-58 adds the support for Python UDFs, but user-defined metrics
> >>> have not been supported yet. With metrics, users can report and monitor
> >>> the UDF status to get a deeper understanding of the execution,
> >>> so in this FLIP, we want to support metrics for Python UDFs.
> >>>
> >>> Previously, Jincheng and I discussed offline about the support of
> >>> metrics for Python UDFs. We'd like to achieve three goals for
> >>> supporting metrics for Python UDFs:
> >>> - Support user-defined metrics including Counters, Gauges, Meters,
> >>> Distributions in Python UDFs.
> >>> - Support defining user scopes.
> >>> - Support defining user variables.
> >>>
> >>> More details can be found in the FLIP wiki page[1] and we are looking
> >>> forward
> >>> to your feedback.
> >>>
> >>> Best,
> >>> Hequn
> >>>
> >>> [1]
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
> >>>
> >
>
>

Re: [DISCUSS] Drop Bucketing Sink

[Hequn Cheng]

Good idea! +1 for dropping the BucketingSink.

Best,
Hequn

> On Mar 12, 2020, at 10:40 PM, Robert Metzger  wrote:
> 
> Hi all,
> 
> I'm currently investigating a failing end to end test for the bucketing
> sink [1].
> The bucketing sink has been deprecated in the 1.9 release [2], because we
> have the new StreamingFileSink [3] for quite a while.
> Before putting any effort into fixing the end to end test for the sink, I
> wanted to propose dropping the bucketing sink from master for the upcoming
> 1.11 release.
> 
> What do you think?
> 
> 
> 
> [1] https://issues.apache.org/jira/browse/FLINK-16227
> [2] https://issues.apache.org/jira/browse/FLINK-13396
> [3] https://issues.apache.org/jira/browse/FLINK-9749

[jira] [Created] (FLINK-16565) Make Pipeline Json compitable between Java and Python if all Pipelinestage are Java ones

[Hequn Cheng (Jira)]

Hequn Cheng created FLINK-16565:
---

 Summary: Make Pipeline Json compitable between Java and Python if 
all Pipelinestage are Java ones
 Key: FLINK-16565
 URL: https://issues.apache.org/jira/browse/FLINK-16565
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python, Library / Machine Learning
Reporter: Hequn Cheng
Assignee: Hequn Cheng
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Features of Apache Flink 1.11

[Hequn Cheng]

Thanks Zhijiang and Piotr for kicking off the discussion and providing the
detailed list.
This would be very helpful for tracking the features.

BTW, as for PyFlink, it would be great if the feature list can also include
the following features:
- FLIP-112: Support User-Defined Metrics in Python UDF
- FLIP-114: Support Python UDF in SQL Client

Looking forward to the release!

Best,
Hequn



On Wed, Mar 11, 2020 at 1:02 PM Yu Li  wrote:

> Thanks for compiling the list of 1.11 efforts Zhijiang and Piotr! This
> helps a lot to better understand what the community is currently working
> on. Looking forward to another successful release.
>
> Best Regards,
> Yu
>
>
> On Wed, 11 Mar 2020 at 11:17, Zhijiang  .invalid>
> wrote:
>
> > Hi community,
> >
> >
> > Not more than one month ago we have released Flink 1.10. We are now
> > heading for the Flink 1.11 release and we, as release managers, would
> like
> > to share with you what are the features that the community is currently
> > working on and we are hoping that will be part of the Flink 1.11 release.
> > Currently we are aiming with the feature freeze to happen in late April.
> >
> > As for now, some of the features are in the very early stages of the
> > development or even brainstorming. Because of that, some of them do not
> > have associated JIRA tickets or FLIP documents. For the next progress
> > announcement we are hoping that this will be no longer the case.
> >
> > Please also note that because we are still relatively at the beginning of
> > the release cycle, some of the FLIPs haven’t yet been voted.
> >
> > - SQL / Table
> > - FLIP-42: Restructure documentation [1]
> > - FLIP-65: New type inference for Table API UDFs [2]
> > - FLIP-84: Improve TableEnv’s interface [3]
> > - FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
> > - FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
> > - FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
> > - FLIP-107: Reading table columns from different parts of source records
> > [7]
> > - [FLINK-14807] Add Table#collect API for fetching data [8]
> > - Support query and table hints
> > - ML / Connectors
> > - FLIP-27: New source API [9]
> > - [FLINK-15670] Wrap a source/sink pair to persist intermediate result
> for
> > subgraph failure recovery [10]
> > - Pulsar source / sink / catalog
> > - Update ML Pipeline API interface to better support Flink ML lib
> > algorithms
> > - PyFlink
> > - FLIP-58: Debugging and monitoring of Python UDF [11]
> > - FLIP-106: Expand the usage scope of Python UDF [12]
> > - Integration with most popular Python libraries (Pandas)
> > - Performance improvements of Python UDF
> > - Support running python UDF in docker workers
> > - Add Python ML API
> > - Fully support all kinds of Python UDF
> > - Web UI
> > - FLIP-98: Better back pressure detection [13]
> > - FLIP-99: Make max exception configurable [14]
> > - FLIP-100: Add attempt information [15]
> > - FLIP-102: Add more metrics to TaskManager [16]
> > - FLIP-103: Better TM/JM log display [17]
> > - [FLINK-14816] Add thread dump feature for TaskManager [18]
> > - Runtime
> > - FLIP-56: Support for dynamic slots on the TaskExecutor [19]
> > - FLIP-67: Support for cluster partitions [20]
> > - FLIP-76: Unaligned checkpoints [21]
> > - FLIP-83: Flink e2e performance testing framework [22]
> > - FLIP-85: Support cluster deploy mode [23]
> > - FLIP-92: Add N-Ary input stream operator in Flink [24]
> > - FLIP-108: Add GPU to the resource management (specifically for UDTF &
> > UDF) [25]
> > - FLIP-111: Consolidate docker images [26]
> > - Unified memory configuration for JobManager
> > - Specify upper bound for number of allocated TaskManagers
> > - [FLINK-9407] ORC format for StreamingFileSink [27]
> > - [FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
> > - [FLINK-10934] Support per-job mode for Kubernetes integration [29]
> > - [FLINK-11395] Avro writer for StreamingFileSink [30]
> > - [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
> > - [FLINK-11499] Extend StreamingFileSink BulkFormats to support arbitrary
> > roll policies [32]
> > - [FLINK-14106] Make SlotManager pluggable [33]
> > - [FLINK-15672] Switch to Log4j2 by default [34]
> > - [FLINK-15674] Consolidate Java and Scala type extraction stack [35]
> > - [FLINK-15679] Improve Flink’s ID system [36]
> > - [FLINK-15786] Use the separated classloader to load connectors’ jar
> [37]
> > - [FLINK-15788] Various Kubernetes improvements [38]
> > - [FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
> > - [FLINK-16408] Bind user code class loader to lifetime of a slot [40]
> > - [FLINK-16428] Network memory management for backpressure [41]
> > - [FLINK-16430] Pipelined region scheduling [42]
> > - Calculate required shuffle memory before allocating slots
> > - State Backend:
> > - [FLINK-5763] Make savepoint self-contained / relocatable [43]
> > - [FLINK-8871] Complete checkpoint cancellation messages 

Re: [DISCUSS] Link Stateful Functions from the Flink Website

[Hequn Cheng]

Hi,

Thanks a lot for raising the discussion @Stephan.
+1 to increase the visibilities of the Stateful Functions.

Another option I'm think is adding a section(named Stateful Functions or
Flink Projects?)
under the "Latest Blog Posts". The advantage is we can add a picture and
some descriptions here.
A picture may attract more attention from the users when he/she visit the
website.
The picture can be the same one in [1].

In the future, if we have multiple Flink individual projects, we can also
turn the section into a Table list
to expose all of them.

What do you think?

Best,
Hequn

[1] https://ci.apache.org/projects/flink/flink-statefun-docs-master/

On Tue, Mar 10, 2020 at 11:13 PM Tzu-Li (Gordon) Tai 
wrote:

> +1 on the suggestion to add "What is Stateful Functions" to the left
> navigation bar.
> That might also mean it would be nice to have a slight rework to the main
> image on the website, illustrating the use cases of Flink (this one [1]).
> On the image it does mention "Event-Driven Applications", but there's
> somewhat missing a more direct connection from that term to the Stateful
> Functions project.
>
> As for what the "What is Stateful Functions?" button directs to, maybe that
> should point to a general concepts page. Initially, we can begin with the
> README contents on the project repo [2].
> As for the actual Statefun documentation link [3], I think we should link
> that from an item in the "Documentation" pull-down list.
>
> One last thing to increase visibility of the Statefun project just a bit
> more:
> There's a "Flink on Github" button on the very bottom of the navigation
> bar.
> What do you think about adding a "Flink Stateful Functions on Github"
> button there as well?
>
> Cheers,
> Gordon
>
> [1]
>
> https://github.com/apache/flink-web/blob/asf-site/img/flink-home-graphic.png
> [2] https://github.com/apache/flink-statefun/blob/master/README.md
> [3] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
>
>
> On Tue, Mar 10, 2020 at 8:29 PM Yu Li  wrote:
>
> > +1 on adding a "What is Stateful Functions" link below the "What is
> Apache
> > Flink" entry and integrating into the Flink docs gradually (instead of
> > hiding it behind until fully integrated).
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 10 Mar 2020 at 19:33, Stephan Ewen  wrote:
> >
> > > Hi all!
> > >
> > > I think it would be nice to mention Stateful Function on the Flink
> > website.
> > > At the moment, Stateful Functions is very hard to discover, and with
> the
> > > first release of it under Apache Flink, it would be a good time to
> change
> > > that.
> > >
> > > My proposal would be to add a "What is Stateful Functions?" below the
> > "What
> > > is Apache Flink" entry in the sidenav, and point it to
> > > https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> > > It is not ideal, yet, but it may serve as an intermediate solution
> until
> > we
> > > can make more involved attempt to rethink the website (for example to
> > also
> > > make SQL more prominent than it currently is).
> > >
> > > An alternative idea was to link it only from the docs, but this would
> be
> > a
> > > bit hidden, in my opinion.
> > >
> > > As a bit of background:
> > >   - The Stateful Functions docs a are a separate doc tree at the
> moment,
> > > because the code (with the docs) is for now in a separate repository
> and
> > > separately versioned/releases.
> > >   - The layout of the Stateful Functions docs is still in the
> > old/original
> > > format, from when it was an "outside Flink" project.
> > >   - There are plans to migrate this to the same stack as the Flink docs
> > and
> > > make it look consistent, but it would be nice to have the docs
> available
> > > in the meantime already.
> > >
> > > Best,
> > > Stephan
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.10.1

[Hequn Cheng]

Hi Yu,

Thanks a lot for raising the discussion and volunteer as the release
manager!

I found there are some other issues[1] which are marked as a blocker:
- FLINK-16454 Update the copyright year in NOTICE files
- FLINK-16262 Class loader problem with
FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
- FLINK-16170 SearchTemplateRequest ClassNotFoundException when use
flink-sql-connector-elasticsearch7
- FLINK-16018 Improve error reporting when submitting batch job (instead of
AskTimeoutException)

These may also need to be resolved in 1.10.1.

Best,
Hequn

[1] https://issues.apache.org/jira/projects/FLINK/versions/12346891


On Tue, Mar 10, 2020 at 6:48 PM Yu Li  wrote:

> Hi Jincheng,
>
> Yes, your help would be very helpful. Thanks a lot!
>
> Best Regards,
> Yu
>
>
> On Tue, 10 Mar 2020 at 18:24, jincheng sun 
> wrote:
>
> > Thanks for bring up the discussion Yu. I would like to give you a hand at
> > the last stage when the RC is finished.(If you need)  :)
> >
> > Best,
> > Jincheng
> >
> >
> >
> > Yu Li  于2020年3月10日周二 下午5:49写道：
> >
> > > Hi All,
> > >
> > > It has been almost one month since we released Flink 1.10.0. We already
> > > have more than 40 resolved improvements/bugs in the release-1.10
> branch,
> > > and I propose to start the 1.10.1 release cycle.
> > >
> > > Most noticeable fixes are:
> > >
> > > - FLINK-16241 [legal] Remove the license and notice file in
> flink-ml-lib
> > > module
> > > - FLINK-16313 Fix RocksDB resource leak in flink-state-processor-api
> > > - FLINK-16161 Statistics zero should be known in HiveCatalog
> > > - FLINK-2336 ArrayIndexOufOBoundsException in TypeExtractor when
> mapping
> > > - FLINK-16108 StreamSQLExample is failed if running in blink planner
> > > - FLINK-16139 Co-location constraints are not reset on task recovery in
> > > DefaultScheduler
> > > - FLINK-16414 Create udaf/udtf function using sql casuing
> > > ValidationException: SQL validation failed
> > >
> > > Furthermore, I think the following issues should be merged before
> 1.10.1
> > > release (especially the Metaspace OOM issue):
> > >
> > > - FLINK-16142 Memory Leak causes Metaspace OOM error on repeated job
> > > submission
> > > - FLINK-16406 Increase default value for JVM Metaspace to minimise its
> > > OutOfMemoryError
> > > - FLINK-16047 Blink planner produces wrong aggregate results with state
> > > clean up
> > > - FLINK-16070 Blink planner can not extract correct unique key for
> > > UpsertStreamTableSink
> > >
> > > I would volunteer as the release manager and kick off the release
> process
> > > once blocker issues are merged. What do you think?
> > >
> > > If there are any concerns or missing blocker issues need to be fixed in
> > > 1.10.1, please let me know. Thanks.
> > >
> > > Best Regards,
> > > Yu
> > >
> >
>

[DISCUSS] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

Hi everyone,

FLIP-58 adds the support for Python UDFs, but user-defined metrics
have not been supported yet. With metrics, users can report and monitor
the UDF status to get a deeper understanding of the execution,
so in this FLIP, we want to support metrics for Python UDFs.

Previously, Jincheng and I discussed offline about the support of
metrics for Python UDFs. We'd like to achieve three goals for
supporting metrics for Python UDFs:
- Support user-defined metrics including Counters, Gauges, Meters,
  Distributions in Python UDFs.
- Support defining user scopes.
- Support defining user variables.

More details can be found in the FLIP wiki page[1] and we are looking
forward
to your feedback.

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF

Re: [DISCUSS] Disable "Squash and merge" button for Flink repository on GitHub

[Hequn Cheng]

Hi,

Thank you all for the discussion!

On one hand, due to the network problem, the "Squash and merge" button is
very helpful. I’m also getting more and more rely on it as it is also very
convenient.

On the other hand, I think the concerns raised by Stephan are valid and we
should pay attention to it, i.e., add PR id and don’t squash everything,
etc. Such changes can never be changed once been checked in. Considering
this, I have updated the committer guide wiki page[1] with some
descriptions about the GitHub web UI and some notices about merging code.
Hope it helps and feel free to add more if you find something has still
been missed.

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/General+Information+for+Committers


On Fri, Mar 6, 2020 at 6:55 PM Stephan Ewen  wrote:

> All right sounds fair.
> Especially that the button helps in case of unstable networks makes sense.
>
>
> On Fri, Mar 6, 2020 at 11:04 AM Aljoscha Krettek 
> wrote:
>
> > If there is a noreply email address that could be on purpose. This
> > happens when you configure github to not show your real e-mail address.
> > This also happens when contributors open a PR and don't want to show
> > their real e-mail address. I talked to at least 1 person that had it set
> > up like this on purpose.
> >
> > Best,
> > Aljoscha
> >
> > On 05.03.20 17:37, Stephan Ewen wrote:
> > > It looks like this feature still messes up email addresses, for example
> > if
> > > you do a "git log | grep noreply" in the repo.
> > >
> > > Don't most PRs consist anyways of multiple commits where we want to
> > > preserve "refactor" and "feature" differentiation in the history,
> rather
> > > than squash everything?
> > >
> > > On Thu, Mar 5, 2020 at 4:54 PM Piotr Nowojski 
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> If it’s really not preserving ownership (I didn’t notice the problem
> > >> before), +1 for removing “squash and merge”.
> > >>
> > >> However -1 for removing “rebase and merge”. I didn’t see any issues
> with
> > >> it and I’m using it constantly.
> > >>
> > >> Piotrek
> > >>
> > >>> On 5 Mar 2020, at 16:40, Jark Wu  wrote:
> > >>>
> > >>> Hi all,
> > >>>
> > >>> Thanks for the feedbacks. But I want to clarify the motivation to
> > disable
> > >>> "Squash and merge" is just because of the regression/bug of the
> missing
> > >>> author information.
> > >>> If GitHub fixes this later, I think it makes sense to bring this
> button
> > >>> back.
> > >>>
> > >>> Hi Stephan & Zhijiang,
> > >>>
> > >>> To be honest, I love the "Squash and merge" button and often use it.
> It
> > >>> saves me a lot of time to merge PRs, because pulling and pushing
> > commits
> > >> in
> > >>> China is very unstable.
> > >>>
> > >>> I don't think the potential problems you mentioned is a "problem".
> > >>> For "Squash and merge",
> > >>> - "Merge commits": there is no "merge" commits, because GitHub will
> > >> squash
> > >>> commits and rebase the commit and then add to the master branch.
> > >>> - "This closes #" line to track back: when you click "Squash and
> > >>> merge", it allows you to edit the title and description, so you can
> > >>> add "This closes #" message to the description the same with in
> the
> > >>> local git. Besides, GitHub automatically append "(#)" after the
> > >> title,
> > >>> which is also helpful to track.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> On Thu, 5 Mar 2020 at 23:36, Robert Metzger 
> > wrote:
> > >>>
> >  +1 for disabling this feature for now.
> > 
> >  Thanks a lot for spotting this!
> > 
> >  On Thu, Mar 5, 2020 at 3:54 PM Zhijiang  >  .invalid>
> >  wrote:
> > 
> > > +1 for disabling "Squash and merge" if feasible to do that.
> > >
> > > The possible benefit to use this button is for saving some efforts
> to
> > > squash some intermediate "[fixup]" commits during PR review.
> > > But it would bring more potential problems as mentioned below,
> > missing
> > > author information and message of "This closes #", etc.
> > > Even it might cause unexpected format of long commit content
> > >> description
> > > if not handled carefully in the text box.
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > --
> > > From:tison 
> > > Send Time:2020 Mar. 5 (Thu.) 21:34
> > > To:dev 
> > > Subject:Re: [DISCUSS] Disable "Squash and merge" button for Flink
> > > repository on GitHub
> > >
> > > Hi Yadong,
> > >
> > > Maybe we firstly reach out INFRA team and see the reply from their
> > >> side.
> > >
> > > Since the actual operator is INFRA team, in the dev mailing list we
> > can
> > > focus on motivation and
> > > wait for the reply.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Yadong Xie  于2020年3月5日周四 下午9:29写道：
> > >
> > >> Hi Jark
> > >>
> > >> I think GitHub UI can not disable 

Re: Request for more Jira rights

[Hequn Cheng]

Hi Niels,

Thanks a lot for your contribution!

The rule has been changed now. As the contribution guide said, only
committers have the permission to assign somebody.
Feel free to ask committers to assign the jira to you with a comment under
the jira once all requirements for the ticket are met.

Thanks,
Hequn

On Sat, Mar 7, 2020 at 3:07 AM Niels Basjes  wrote:

> Hi,
>
> I've created some jira tickets and for some of them I've put up a merge
> request.
>
> I noticed that the Flinkbot warns:
>
> This pull request references an unassigned Jira ticket
> . According to the code
> contribution guide
> , tickets need
> to be assigned before starting with the implementation work.
>
> Sounds good to me.
> However at this point I cannot assign these tickets to myself.
> The strange thing is that I used to have that option in the past.
>
> I kindly request the privilege to assign tickets to myself if I choose to
> pick them up.
>
> Jira userid:  nielsbasjes
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

Re: [PROPOSAL] Reverse the dependency from flink-streaming-java to flink-client

[Hequn Cheng]

Hi,

+1 to make flink-streaming-java an API only module and solve it sooner
rather than later.
It would be more clear to only expose an SDK module for writing jobs.

Another benefit I can see is: the flink-streaming-java would be scala-free
if we reverse the dependencies and this would be really nice for the Java
API module.

As for the issue of dependencies setup of users, I agree with Stephan that
it's ok to do so
if we add corresponding document and runtime error messages about the
changes.

Best,
Hequn


On Fri, Mar 6, 2020 at 3:03 AM Kostas Kloudas  wrote:

> Big +1 also from my side.
>
> This will eliminate some work-arounds used so far to bypass the module
> structure (like code using reflection to extract a JobGraph from a
> Pipeline).
>
> I agree with Stephan that with proper documentation, release notes and
> tooling update, it will hopefully not be a big hassle for users to
> migrate.
> Also I think it should be done as early in the release as possible, so
> that we can give it enough exposure and testing. In the past, such
> deep changes late in the release have led to longer release-testing
> periods and, eventually, longer release cycles.
>
> Cheers,
> Kostas
>
> On Thu, Mar 5, 2020 at 3:35 PM Stephan Ewen  wrote:
> >
> > +1 to this fix, in general.
> >
> > If the main issue is that users have to now add "flink-clients"
> explicitly,
> > then I think this is okay, if we spell it out prominently in the release
> > notes, and make sure quickstarts / etc are updated, and have a good error
> > message when client/runtime classes are not found.
> >
> > On Thu, Mar 5, 2020 at 2:56 PM Aljoscha Krettek 
> wrote:
> >
> > > Hi,
> > >
> > > thanks for starting the discussion, Tison!
> > >
> > > I'd like to fix this dependency mess rather sooner than later, but we
> do
> > > have to consider the fact that we are breaking the dependency setup of
> > > users. If they they only had a dependency on flink-streaming-java
> before
> > > but used classes from flink-clients they would have to explicitly add
> > > this dependency now.
> > >
> > > Let's see what others think.
> > >
> > > Best,
> > > Aljoscha
> > >
> > > On 05.03.20 02:53, tison wrote:
> > > > Hi devs,
> > > >
> > > > Here is a proposal to reverse the dependency from
> flink-streaming-java to
> > > > flink-client, for a proper
> > > > module dependency graph. Since it changes current structure, it
> should be
> > > > discussed publicly.
> > > >
> > > > The original idea comes from that flink-streaming-java acts as an API
> > > only
> > > > module just as what
> > > > we do in its batch companion flink-java. If a Flink user want to
> write a
> > > > minimum DataStream
> > > > program, the only dependency should be flink-streaming java.
> > > >
> > > > However, currently as it is implemented, flink-client and even
> > > > flink-runtime are transitively polluted
> > > > in when user depends on flink-streaming-java. These dependencies
> polluted
> > > > in as
> > > >
> > > > flink-client:
> > > >- previously, ClusterClient, which is removed by FLIP-73 Executors
> > > >- accidentally, ProgramInvocationException, we just throw in
> place as
> > > it
> > > > is accessible.
> > > >- transitively, flink-optimizer, for one utility.
> > > >- transitively, flink-java, for several utilities.
> > > > flink-runtime:
> > > >- mainly for JobGraph generating.
> > > >
> > > > With a previous discussion with @Aljoscha Krettek <
> aljos...@apache.org>
> > > our
> > > > goal is briefly making flink-streaming-java
> > > > an API only module. As a first step we can break the dependency from
> > > > flink-streaming-java to
> > > > flink-client[1][2].
> > > >
> > > > With this first step, continuously we factor out common utilities in
> > > > flink-java to
> > > > flink-core and eventually eliminate dependencies from streaming to
> batch;
> > > > while
> > > > orthogonally, we factor out job compilation logic into
> > > > flink-streaming-compiler module and
> > > > break the dependency to flink-runtime. The final dependency graph
> will
> > > be:
> > > >
> > > >
> > > > flink-client -> flink-streaming-compiler -> flink-runtime
> > > >   \->
> > > > flink-streaming-java
> > > >
> > > > Looking forward to your feedback. Basically whether or not it is in a
> > > right
> > > > direction, and if so,
> > > > how the community integrates this proposal.
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-15090
> > > > [2] https://issues.apache.org/jira/browse/FLINK-16427
> > > >
> > >
>