[ANNOUNCE] Apache Flink Table Store 0.1.0 released

2022-05-10 Thread Jingsong Lee
The Apache Flink community is very happy to announce the release of Apache Flink Table Store 0.1.0. Apache Flink Table Store provides storage for building dynamic tables for both stream and batch processing in Flink, supporting high speed data ingestion and timely data query. Please check out

Pulsar/Flink Error: PulsarAdminException$NotFoundException: Topic not exist

2022-05-10 Thread Jason Kania
Hi, I am attempting to upgrade from 1.12.7 to 1.15.0. One of the issues I am encountering is the following exception when attempting to submit a job from the command line: switched from INITIALIZING to FAILED with failure cause: 

Re: [Discuss] Creating an Apache Flink slack workspace

2022-05-10 Thread Xintong Song
I agree with Robert on reworking the "Community" and "Getting Help" pages to emphasize how we position the mailing lists and Slack, and on revisiting in 6-12 months. Concerning dedicated Apache Flink Slack vs. ASF Slack, I'm with Konstantin. I'd expect it to be easier for having more channels and

Flink job restarted from empty state when execution.shutdown-on-application-finish is false

2022-05-10 Thread Gyula Fóra
Hi Devs! I ran into a concerning situation and would like to hear your thoughts on this. I am running Flink 1.15 on Kubernetes native mode (using the operator but that is besides the point here) with Flink Kubernetes HA enabled. We have set *execution.shutdown-on-application-finish = false* I

Re: Flink job restarted from empty state when execution.shutdown-on-application-finish is enabled

2022-05-10 Thread Gyula Fóra
Sorry I messed up the email, I meant false . So when we set it to not shut down … :) Gyula On Wed, 11 May 2022 at 05:06, Yun Tang wrote: > Hi Gyula, > > Why are you sure that the configuration of > execution.shutdown-on-application-finish leading to this error? I noticed > that the default

Re: Re: [DISCUSS] FLIP-168: Speculative execution for Batch Job

2022-05-10 Thread Zhu Zhu
Hi everyone, According to the discussion and updates of the blocklist mechanism[1] (FLIP-224), I have updated FLIP-168 to make decision on itself to block identified slow nodes. A new configuration is also added to control how long a slow node should be blocked. [1]

Re: [DISCUSS] Planning Flink 1.16 with the community

2022-05-10 Thread Xintong Song
I'd like to kindly remind that, if someone adds / modifies release notes of a JIRA ticket close to or after the release finalization, it would be important to sync with the release managers and make sure that change appears in the final release notes. We have recently discovered a breaking change

[jira] [Created] (FLINK-27568) Cannot submit JobGraph

2022-05-10 Thread Jason Kania (Jira)
Jason Kania created FLINK-27568: --- Summary: Cannot submit JobGraph Key: FLINK-27568 URL: https://issues.apache.org/jira/browse/FLINK-27568 Project: Flink Issue Type: Bug Affects Versions:

Re: [DISCUSS] DockerHub repository maintainers

2022-05-10 Thread Xintong Song
Thanks for volunteering, Marton. Since the discussion has been open for quite long and no one else is volunteering, I'll reply to the INFRA with the list of 4 maintainers (Xintong, Jinsong, Till, Marton). Thank you~ Xintong Song On Sat, May 7, 2022 at 3:15 PM Márton Balassi wrote: > Hi

Re: Flink job restarted from empty state when execution.shutdown-on-application-finish is enabled

2022-05-10 Thread Yun Tang
Hi Gyula, Why are you sure that the configuration of execution.shutdown-on-application-finish leading to this error? I noticed that the default value of this configuration is just "true". >From my understanding, the completed checkpoint store should only clear its >persisted checkpoint

Flink job restarted from empty state when execution.shutdown-on-application-finish is enabled

2022-05-10 Thread Gyula Fóra
Hi Devs! I ran into a concerning situation and would like to hear your thoughts on this. I am running Flink 1.15 on Kubernetes native mode (using the operator but that is besides the point here) with Flink Kubernetes HA enabled. We have enabled *execution.shutdown-on-application-finish = true*

Re: [DISCUSS ] HybridSouce Table & Sql api timeline

2022-05-10 Thread Thomas Weise
fyi there is a related ticket here: https://issues.apache.org/jira/browse/FLINK-22793 On Mon, May 9, 2022 at 11:34 PM Becket Qin wrote: > > Cool, I've granted you the permission. > > Cheers, > > Jiangjie (Becket) Qin > > On Tue, May 10, 2022 at 1:14 PM Ran Tao wrote: > > > Hi, Becket. Thanks

Re: JobManager High Availability

2022-05-10 Thread Thomas Weise
+1 to what Konstantin said. There is no real benefit running multiple JMs on k8s, unless you need to optimize the JM startup time. Often the time to get a replacement pod is negligible compared to the job restart time. Thomas On Tue, May 10, 2022 at 2:27 AM Őrhidi Mátyás wrote: > > Ah, ok.

Re: flink Job is throwing depdnecy issue when submitted to clusrer

2022-05-10 Thread Konstantin Knauf
Hi there, are you using any of Flink S3 Filesystems? If so, where do you load it from a) lib/ b) plugins/ c) bundled with your Job in a fat JAR b) would be the right way to do it in Flink 1.13. I don't know if this fixes the issue, but IIRC because we introduced the plugin mechansim we don't

Re: Re: [ANNOUNCE] New Flink PMC member: Yang Wang

2022-05-10 Thread Thomas Weise
Congratulations, Yang! On Tue, May 10, 2022 at 3:15 AM Márton Balassi wrote: > > Congrats, Yang. Well deserved :-) > > On Tue, May 10, 2022 at 9:16 AM Terry Wang wrote: > > > Congrats Yang! > > > > On Mon, May 9, 2022 at 11:19 AM LuNing Wang wrote: > > > > > Congrats Yang! > > > > > > Best, >

[jira] [Created] (FLINK-27567) [JUnit5 Migration] Module: flink-connector-aws-kinesis-streams

2022-05-10 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-27567: --- Summary: [JUnit5 Migration] Module: flink-connector-aws-kinesis-streams Key: FLINK-27567 URL: https://issues.apache.org/jira/browse/FLINK-27567 Project: Flink

[jira] [Created] (FLINK-27566) [JUnit5 Migration] Module: flink-connector-aws-kinesis-firehose

2022-05-10 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-27566: --- Summary: [JUnit5 Migration] Module: flink-connector-aws-kinesis-firehose Key: FLINK-27566 URL: https://issues.apache.org/jira/browse/FLINK-27566 Project: Flink

[jira] [Created] (FLINK-27565) Bump minimist from 1.2.5 to 1.2.6 in /flink-runtime-web/web-dashboard

2022-05-10 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-27565: -- Summary: Bump minimist from 1.2.5 to 1.2.6 in /flink-runtime-web/web-dashboard Key: FLINK-27565 URL: https://issues.apache.org/jira/browse/FLINK-27565 Project:

[jira] [Created] (FLINK-27564) Split PyFlink DataStream connectors into separate files

2022-05-10 Thread Dian Fu (Jira)
Dian Fu created FLINK-27564: --- Summary: Split PyFlink DataStream connectors into separate files Key: FLINK-27564 URL: https://issues.apache.org/jira/browse/FLINK-27564 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

2022-05-10 Thread Timo Walther
Hi Shengkai, sorry for jumping in so late into the discussion. There was a lot going on in the last 3 weeks. This FLIP is very important and it is great that you tackle this topic which is planned since we started with the SQL Client. The ML discussion is already quite long. It would be

Re: Re: [ANNOUNCE] New Flink PMC member: Yang Wang

2022-05-10 Thread Márton Balassi
Congrats, Yang. Well deserved :-) On Tue, May 10, 2022 at 9:16 AM Terry Wang wrote: > Congrats Yang! > > On Mon, May 9, 2022 at 11:19 AM LuNing Wang wrote: > > > Congrats Yang! > > > > Best, > > LuNing Wang > > > > Dian Fu 于2022年5月7日周六 17:21写道: > > > > > Congrats Yang! > > > > > > Regards, >

[jira] [Created] (FLINK-27563) Resource Providers - Yarn doc page has minor display error

2022-05-10 Thread Wei Zhong (Jira)
Wei Zhong created FLINK-27563: - Summary: Resource Providers - Yarn doc page has minor display error Key: FLINK-27563 URL: https://issues.apache.org/jira/browse/FLINK-27563 Project: Flink Issue

Re: Re: 【Could we support distribute by For FlinkSql】

2022-05-10 Thread godfrey he
Hi, Ipengdream. I will drive this work. We will support this functionality via hints, because "distribute by" is not in the sql standard. But it will be supported in hive dialect. I will post the FLIP doc recently. Best, Godfrey Jark Wu 于2022年5月9日周一 16:03写道: > > We will start a FLIP

Re: JobManager High Availability

2022-05-10 Thread Őrhidi Mátyás
Ah, ok. Thanks, Konstantin for the clarification, I appreciate the quick response. Best, Matyas On Tue, May 10, 2022 at 10:59 AM Konstantin Knauf wrote: > Hi Matyas, > > yes, that's expected. The feature should have never been called "high > availability", but something like "Flink Jobmanager

[jira] [Created] (FLINK-27562) Creating HDFS table in SQL client throws UnsupportedFileSystemSchemeException even if hadoop jar is added through --jar argument

2022-05-10 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-27562: --- Summary: Creating HDFS table in SQL client throws UnsupportedFileSystemSchemeException even if hadoop jar is added through --jar argument Key: FLINK-27562 URL:

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

2022-05-10 Thread Lijie Wang
Hi Becket Qin, Thanks for your suggestions. I have moved the description of configurations, metrics and REST API into "Public Interface" section, and made a few updates according to your suggestion. And in this FLIP, there no public java Interfaces or pluggables that users need to implement by

Re: JobManager High Availability

2022-05-10 Thread Konstantin Knauf
Hi Matyas, yes, that's expected. The feature should have never been called "high availability", but something like "Flink Jobmanager failover", because that's what it is. With standby Jobmanagers what you gain is a faster failover, because a new Jobmanager does not need to be started before

JobManager High Availability

2022-05-10 Thread Őrhidi Mátyás
Hi Folks! I've been goofing around with the JobManager HA configs using multiple JM replicas (in the Flink Kubernetes Operator). It's working seemingly fine, however the job itself is being restarted when you kill the leader JM pod. Is this expected? Thanks, Matyas

Re: [Discuss] Creating an Apache Flink slack workspace

2022-05-10 Thread Konstantin Knauf
Thanks for starting this discussion again. I am pretty much with Timo here. Slack or Discourse as an alternative for the user community, and mailing list for the contributing, design discussion, etc. I definitely see the friction of joining a mailing list and understand if users are intimidated.

[jira] [Created] (FLINK-27561) [JUnit5 Migration] Module: flink-connector-aws-base

2022-05-10 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-27561: --- Summary: [JUnit5 Migration] Module: flink-connector-aws-base Key: FLINK-27561 URL: https://issues.apache.org/jira/browse/FLINK-27561 Project: Flink

Re: [Discuss] Creating an Apache Flink slack workspace

2022-05-10 Thread Timo Walther
I also think that a real-time channel is long overdue. The Flink community in China has shown that such a platform can be useful for improving the collaboration within the community. The DingTalk channel of 10k+ users collectively helping each other is great to see. It could also reduce the

Re: [Discuss] Creating an Apache Flink slack workspace

2022-05-10 Thread David Anderson
Thank you @Xintong Song for sharing the experience of the Flink China community. I'm become convinced we should give Slack a try, both for discussions among the core developers, and as a place where the community can reach out for help. I am in favor of using the ASF slack, as we will need a

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

2022-05-10 Thread Jark Wu
Thank Shengkai for driving this work! +1 to start a VOTE. Best, Jark On Tue, 10 May 2022 at 12:13, Shengkai Fang wrote: > Hi, everyone. > > Thanks for all the inputs. Hope my feedback can address most of questions. > > After the long discussion, I think we have reached the consensus about the

Re: Re: [ANNOUNCE] New Flink PMC member: Yang Wang

2022-05-10 Thread Terry Wang
Congrats Yang! On Mon, May 9, 2022 at 11:19 AM LuNing Wang wrote: > Congrats Yang! > > Best, > LuNing Wang > > Dian Fu 于2022年5月7日周六 17:21写道: > > > Congrats Yang! > > > > Regards, > > Dian > > > > On Sat, May 7, 2022 at 12:51 PM Jacky Lau wrote: > > > > > Congrats Yang and well Deserved! > > >

Re: [VOTE] FLIP-225: Implement standalone mode support in the kubernetes operator

2022-05-10 Thread Gyula Fóra
Sounds good Jassat! In the next 1-2 weeks the community will be pretty busy hardening and finalizing the 1.0.0 release but we will try to cut the release branch next week so we can make some progress on your PRs as well. Thank you Gyula On Mon, May 9, 2022 at 1:40 PM Jassat, Usamah wrote: >

Re: [DISCUSS ] HybridSouce Table & Sql api timeline

2022-05-10 Thread Becket Qin
Cool, I've granted you the permission. Cheers, Jiangjie (Becket) Qin On Tue, May 10, 2022 at 1:14 PM Ran Tao wrote: > Hi, Becket. Thanks for your suggestions. My id is: Ran Tao > And i will draft this flip in a few days. thanks~ > > Becket Qin 于2022年5月10日周二 12:40写道: > > > Hi Ran, > > > > The

Re: [Discuss] Creating an Apache Flink slack workspace

2022-05-10 Thread Robert Metzger
It seems that we'd have to use invite links on the Flink website for people to join our Slack (1) These links can be configured to have no time-expiration, but they will expire after 100 guests have joined. I guess we'd have to use a URL shortener (https://s.apache.org) that we update once the