Re: [Discussion] FLIP-79 Flink Function DDL Support

2019-10-29 Thread Peter Huang
Hi Bowen, I can't agree more about we first have an agreement on the DDL syntax and focus on the MVP in the current phase. 1) what's the syntax to distinguish function language Currently, there are two opinions: - USING 'python .' - [LANGUAGE JVM|PYTHON] USING JAR '...' As we need to

Re: [Discussion] FLIP-79 Flink Function DDL Support

2019-10-29 Thread Peter Huang
Hi Jingsong, Thanks for the input. The FLINK function DDL definitely needs to align with HQL, I updated the doc accordingly. CREATE FUNCTION [db_name.]function_name AS class_name [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; For you other questions below: 1) how to

[jira] [Created] (FLINK-14565) Shutdown SystemResourcesCounter on (JM|TM)MetricGroup closed

2019-10-29 Thread Zili Chen (Jira)
Zili Chen created FLINK-14565: - Summary: Shutdown SystemResourcesCounter on (JM|TM)MetricGroup closed Key: FLINK-14565 URL: https://issues.apache.org/jira/browse/FLINK-14565 Project: Flink

[VOTE] Move flink-orc to flink-formats from flink-connectors

2019-10-29 Thread Jingsong Li
Hi all: We already have the parent model of formats. we have put other formats(flink-avro, flink-json, flink-parquet, flink-json, flink-csv, flink-sequence-file) to flink-formats. flink-orc is a format too. So we can move it to flink-formats. In theory, there should be no compatibility problem,

Builds Notification Update

2019-10-29 Thread Jark Wu
Hi, This is an update and discussion thread to the notification bot and builds mailing list. We created the bui...@flink.apache.org mailing list to monitor the builds status. And a notification bot is running behind it. You can subscribe the mailing list via sending an email to

Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors

2019-10-29 Thread Jingsong Li
Thanks jark for reminding me. I will start a vote thread. On Wed, Oct 30, 2019 at 11:40 AM Jark Wu wrote: > Hi Jingsong, > > Do we need a formal vote? Or do we need to wait for "Lazy > Majority" approval [1] which requires 3 binding votes? > > Best, > Jark > > [1]: >

Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors

2019-10-29 Thread Jark Wu
Hi Jingsong, Do we need a formal vote? Or do we need to wait for "Lazy Majority" approval [1] which requires 3 binding votes? Best, Jark [1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026 On Wed, 30 Oct 2019 at 11:12, Jingsong Li wrote: > Hi bowen, > I think the

Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors

2019-10-29 Thread Jingsong Li
Hi bowen, I think the reason is that flink-orc was developed earlier than flink-formats. Thanks the responses of everyone, I will create JIRA to do it. On Wed, Oct 30, 2019 at 10:27 AM Danny Chan wrote: > +1 to move to flink-format. > > Best, > Danny Chan > 在 2019年10月29日 +0800

Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors

2019-10-29 Thread Danny Chan
+1 to move to flink-format. Best, Danny Chan 在 2019年10月29日 +0800 AM11:10,dev@flink.apache.org,写道: > > +1 to move to flink-format.

Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors

2019-10-29 Thread Jark Wu
+1 to move to flink-format. Best, Jark On Wed, 30 Oct 2019 at 02:27, Bowen Li wrote: > Hi Jingsong, > > Thanks for bringing it up. I found flink-orc lives under flink-connectors > instead of flink-formats to be confusing too. > > I wonder why it's that way in the beginning? If there's no good

Re: [DISCUSS] How to prepare the Python environment of PyFlink release in the current Flink release process

2019-10-29 Thread Hequn Cheng
Hi Dian, Thanks a lot for bringing the discussion. It would be a headache to address these environmental problems, so +1 for option #1 to use the virtual environment. Best, Hequn On Tue, Oct 29, 2019 at 9:31 PM Till Rohrmann wrote: > Thanks for bringing this topic up Dian. I'd be in favour of

[jira] [Created] (FLINK-14564) all-local slave config doesn't respect $FLINK_SSH_OPTS

2019-10-29 Thread Robert Lugg (Jira)
Robert Lugg created FLINK-14564: --- Summary: all-local slave config doesn't respect $FLINK_SSH_OPTS Key: FLINK-14564 URL: https://issues.apache.org/jira/browse/FLINK-14564 Project: Flink Issue

[jira] [Created] (FLINK-14563) reuse HiveShim instance in Hive functions code path rather than creating new ones repeatedly

2019-10-29 Thread Bowen Li (Jira)
Bowen Li created FLINK-14563: Summary: reuse HiveShim instance in Hive functions code path rather than creating new ones repeatedly Key: FLINK-14563 URL: https://issues.apache.org/jira/browse/FLINK-14563

Re: [Discussion] FLIP-79 Flink Function DDL Support

2019-10-29 Thread Bowen Li
Hi all, Besides all the good questions raised above, we seem all agree to have a MVP for Flink 1.10, "to support users to create and persist a java class-based udf that's already in classpath (no extra resource loading), and use it later in queries". IIUIC, to achieve that in 1.10, the following

Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors

2019-10-29 Thread Bowen Li
Hi Jingsong, Thanks for bringing it up. I found flink-orc lives under flink-connectors instead of flink-formats to be confusing too. I wonder why it's that way in the beginning? If there's no good reason to remain that way, +1 to move flink-orc to flink-formats Bowen On Mon, Oct 28, 2019 at

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-29 Thread Seth Wiesman
+1 (non-binding) Seth > On Oct 23, 2019, at 9:31 PM, Jingsong Li wrote: > > +1 (non-binding) > > Best, > Jingsong Lee > >> On Wed, Oct 23, 2019 at 9:02 PM Yu Li wrote: >> >> +1 (non-binding) >> >> Best Regards, >> Yu >> >> >>> On Wed, 23 Oct 2019 at 16:56, Haibo Sun wrote: >>> >>> +1

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-29 Thread Xuannan Su
Congratulations Becket! :-) Xuannan > On Oct 28, 2019, at 6:07 AM, Fabian Hueske wrote: > > Hi everyone, > > I'm happy to announce that Becket Qin has joined the Flink PMC. > Let's congratulate and welcome Becket as a new member of the Flink PMC! > > Cheers, > Fabian

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-29 Thread Bowen Li
Congrats Becket! On Tue, Oct 29, 2019 at 06:32 Till Rohrmann wrote: > Congrats Becket :-) > > On Tue, Oct 29, 2019 at 10:27 AM Yang Wang wrote: > > > Congratulations Becket :) > > > > Best, > > Yang > > > > Vijay Bhaskar 于2019年10月29日周二 下午4:31写道: > > > > > Congratulations Becket > > > > > >

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-29 Thread Andrey Zagrebin
+1 (binding) On Tue, Oct 29, 2019 at 10:09 AM Igal Shilman wrote: > +1 (non-binding) > > Thanks, > Igal Shilman > > On Sat, Oct 26, 2019 at 12:25 AM Ufuk Celebi wrote: > > > +1 (binding) > > > > On Fri, Oct 25, 2019 at 6:39 PM Maximilian Michels > wrote: > > > > > +1 (binding) > > > > > > On

[jira] [Created] (FLINK-14562) RMQSource leaves idle consumer after closing

2019-10-29 Thread Nicolas Deslandes (Jira)
Nicolas Deslandes created FLINK-14562: - Summary: RMQSource leaves idle consumer after closing Key: FLINK-14562 URL: https://issues.apache.org/jira/browse/FLINK-14562 Project: Flink Issue

Re: compile failed

2019-10-29 Thread Xu Jianhai
> > Sorry for my terrible mac skill, I found I have many java version, And I > write the JAVA_HOME with absolute path, so even I update to latest jdk > version, I remain failed. > >

[jira] [Created] (FLINK-14561) Don't write FLINK_PLUGINS_DIR ENV variable to Flink configuration

2019-10-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14561: - Summary: Don't write FLINK_PLUGINS_DIR ENV variable to Flink configuration Key: FLINK-14561 URL: https://issues.apache.org/jira/browse/FLINK-14561 Project: Flink

Re: Build error: package does not exist

2019-10-29 Thread Hynek Noll
And I've tried just `mvn clean install -DskipTests -Drat.skip=true -DskipITs` as well. It takes around half an hour, so I'm not too keen to try all the possibilities. I guess there might be some other SDKs/libraries that I'm missing and Maven won't tell me? Or just some random incompatibility?

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-29 Thread Till Rohrmann
Congrats Becket :-) On Tue, Oct 29, 2019 at 10:27 AM Yang Wang wrote: > Congratulations Becket :) > > Best, > Yang > > Vijay Bhaskar 于2019年10月29日周二 下午4:31写道: > > > Congratulations Becket > > > > Regards > > Bhaskar > > > > On Tue, Oct 29, 2019 at 1:53 PM Danny Chan wrote: > > > > >

Re: How to specify a test to run in Flink?

2019-10-29 Thread Till Rohrmann
Add `-DfailIfNoTests=false` to your maven command and then it should work. Cheers, Till On Tue, Oct 29, 2019 at 6:51 AM Jark Wu wrote: > Usually, I use the following commands to execute single test and it works > well. > > ``` > $ mvn clean install -DskipTests > # go to the moudle where the

Re: [DISCUSS] How to prepare the Python environment of PyFlink release in the current Flink release process

2019-10-29 Thread Till Rohrmann
Thanks for bringing this topic up Dian. I'd be in favour of option #1 because this would also allow to create reproducible builds. Cheers, Till On Tue, Oct 29, 2019 at 5:28 AM jincheng sun wrote: > Hi, > Thanks for bringing up the discussion Dian. > +1 for the #1. > > Hi Jeff, this changes is

[jira] [Created] (FLINK-14560) The value of taskmanager.memory.size in flink-conf.yaml is set to zero will cause taskmanager not to work

2019-10-29 Thread fa zheng (Jira)
fa zheng created FLINK-14560: Summary: The value of taskmanager.memory.size in flink-conf.yaml is set to zero will cause taskmanager not to work Key: FLINK-14560 URL:

Re: [VOTE] FLIP-70: Flink SQL Computed Column Design

2019-10-29 Thread Dawid Wysakowicz
+1 Best, Dawid On 29/10/2019 04:03, Zhenghua Gao wrote: > +1 (non-binding) > > *Best Regards,* > *Zhenghua Gao* > > > On Mon, Oct 28, 2019 at 2:26 PM Danny Chan wrote: > >> Hi all, >> >> I would like to start the vote for FLIP-70[1] which is discussed and >> reached consensus in the discussion

[jira] [Created] (FLINK-14559) flink sql connector for stream table stdout

2019-10-29 Thread xiaodao (Jira)
xiaodao created FLINK-14559: --- Summary: flink sql connector for stream table stdout Key: FLINK-14559 URL: https://issues.apache.org/jira/browse/FLINK-14559 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-14558) Fix the ClassNotFoundException issue for run python job in standalone mode

2019-10-29 Thread sunjincheng (Jira)
sunjincheng created FLINK-14558: --- Summary: Fix the ClassNotFoundException issue for run python job in standalone mode Key: FLINK-14558 URL: https://issues.apache.org/jira/browse/FLINK-14558 Project:

[jira] [Created] (FLINK-14557) Clean up the package of py4j

2019-10-29 Thread sunjincheng (Jira)
sunjincheng created FLINK-14557: --- Summary: Clean up the package of py4j Key: FLINK-14557 URL: https://issues.apache.org/jira/browse/FLINK-14557 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-14556) Correct the package of cloud pickle

2019-10-29 Thread sunjincheng (Jira)
sunjincheng created FLINK-14556: --- Summary: Correct the package of cloud pickle Key: FLINK-14556 URL: https://issues.apache.org/jira/browse/FLINK-14556 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-14555) Streaming File Sink s3 end-to-end test stalls

2019-10-29 Thread Gary Yao (Jira)
Gary Yao created FLINK-14555: Summary: Streaming File Sink s3 end-to-end test stalls Key: FLINK-14555 URL: https://issues.apache.org/jira/browse/FLINK-14555 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-29 Thread Yang Wang
Congratulations Becket :) Best, Yang Vijay Bhaskar 于2019年10月29日周二 下午4:31写道: > Congratulations Becket > > Regards > Bhaskar > > On Tue, Oct 29, 2019 at 1:53 PM Danny Chan wrote: > > > Congratulations :) > > > > Best, > > Danny Chan > > 在 2019年10月29日 +0800 PM4:14,dev@flink.apache.org,写道: > > >

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-29 Thread Igal Shilman
+1 (non-binding) Thanks, Igal Shilman On Sat, Oct 26, 2019 at 12:25 AM Ufuk Celebi wrote: > +1 (binding) > > On Fri, Oct 25, 2019 at 6:39 PM Maximilian Michels wrote: > > > +1 (binding) > > > > On 25.10.19 14:31, Congxian Qiu wrote: > > > +1 (non-biding) > > > Best, > > > Congxian > > > > > >

Re: How to use two continuously window with EventTime in sql

2019-10-29 Thread Jark Wu
Hi, You can use TUMBLE_ROWTIME(...) to get the rowtime attribute of the first window result, and use this field to apply a following window aggregate. See more https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql.html#group-windows Best, Jark On Tue, 29 Oct 2019 at 15:39, 刘建刚

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-29 Thread Vijay Bhaskar
Congratulations Becket Regards Bhaskar On Tue, Oct 29, 2019 at 1:53 PM Danny Chan wrote: > Congratulations :) > > Best, > Danny Chan > 在 2019年10月29日 +0800 PM4:14,dev@flink.apache.org,写道: > > > > Congratulations :) >

[jira] [Created] (FLINK-14554) Correct the comment of ExistingSavepoint#readKeyedState to generate java doc

2019-10-29 Thread vinoyang (Jira)
vinoyang created FLINK-14554: Summary: Correct the comment of ExistingSavepoint#readKeyedState to generate java doc Key: FLINK-14554 URL: https://issues.apache.org/jira/browse/FLINK-14554 Project: Flink

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-29 Thread Danny Chan
Congratulations :) Best, Danny Chan 在 2019年10月29日 +0800 PM4:14,dev@flink.apache.org,写道: > > Congratulations :)

[jira] [Created] (FLINK-14553) Respect non-blocking output in StreamTask#processInput

2019-10-29 Thread zhijiang (Jira)
zhijiang created FLINK-14553: Summary: Respect non-blocking output in StreamTask#processInput Key: FLINK-14553 URL: https://issues.apache.org/jira/browse/FLINK-14553 Project: Flink Issue Type:

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-29 Thread Piotr Nowojski
Congratulations :) Piotrek > On 29 Oct 2019, at 08:59, Dawid Wysakowicz wrote: > > Congrats, > > Best > > Dawid > > On 28/10/2019 17:37, Yun Tang wrote: >> Congratulations, Becket >> >> Best >> Yun Tang >> >> From: Rong Rong >> Sent: Tuesday, October 29,

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-29 Thread Dawid Wysakowicz
Congrats, Best Dawid On 28/10/2019 17:37, Yun Tang wrote: > Congratulations, Becket > > Best > Yun Tang > > From: Rong Rong > Sent: Tuesday, October 29, 2019 0:19 > To: dev > Cc: Becket Qin > Subject: Re: [ANNOUNCE] Becket Qin joins the Flink PMC > >

[jira] [Created] (FLINK-14552) Enable partition statistics in blink planner

2019-10-29 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-14552: Summary: Enable partition statistics in blink planner Key: FLINK-14552 URL: https://issues.apache.org/jira/browse/FLINK-14552 Project: Flink Issue Type:

[jira] [Created] (FLINK-14551) Unaligned checkpoints

2019-10-29 Thread zhijiang (Jira)
zhijiang created FLINK-14551: Summary: Unaligned checkpoints Key: FLINK-14551 URL: https://issues.apache.org/jira/browse/FLINK-14551 Project: Flink Issue Type: New Feature Components:

How to use two continuously window with EventTime in sql

2019-10-29 Thread 刘建刚
For one sql window, I can register table with event time and use time field in the tumble window. But if I want to use the result for the first window and use another window to process it, how can I do it? Thank you.

Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors

2019-10-29 Thread Zhenghua Gao
+1 to move flink-orc to flink-formats. Since we have put other file-based formats(flink-avro, flink-json, flink-parquet, flink-json, flink-csv, flink-sequence-file) in flink-formats, flink-orc should be the same. *Best Regards,* *Zhenghua Gao* On Tue, Oct 29, 2019 at 11:10 AM Jingsong Lee