Re: [DISCUSS] Update Scala 2.12.7 to 2.12.15

2022-06-07 Thread Ran Tao
Hi, Martijn the scala community wait the publish of some corresponding components with 2.12.16 now. A few days or a week later will be final released. here is the scala 2.12.16 timeline[1] Hope it helps. [1] https://contributors.scala-lang.org/t/scala-2-12-16-release-planning/5692/8 Martijn

Re: [DISCUSS] Update Scala 2.12.7 to 2.12.15

2022-06-07 Thread Ran Tao
Hi, Martijn. I have asked the scala community, i will reply to you if there is any respond. however, i think you can vote or discuss with 2.12.15. The 2.12.16 just backport some functionalies from higher scala version, such as jvm11, jvm-17 target, the flink may need it. But as Chesnay says, we

[jira] [Created] (FLINK-27950) flink streaming写入hive表,设置的文件后缀参数失效

2022-06-07 Thread yantao (Jira)
yantao created FLINK-27950: -- Summary: flink streaming写入hive表,设置的文件后缀参数失效 Key: FLINK-27950 URL: https://issues.apache.org/jira/browse/FLINK-27950 Project: Flink Issue Type: Bug Components:

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

2022-06-07 Thread Xingbo Huang
Thanks everyone for the great effort driving this. +1 for building a dedicated channel for CI builds. Best, Xingbo Martijn Visser 于2022年6月8日周三 01:32写道: > Hi Matthias, > > I've only thought about it, but I absolutely would +1 this. Making this > information available in dedicated channels will

[jira] [Created] (FLINK-27949) kafka create topic error

2022-06-07 Thread sandy du (Jira)
sandy du created FLINK-27949: Summary: kafka create topic error Key: FLINK-27949 URL: https://issues.apache.org/jira/browse/FLINK-27949 Project: Flink Issue Type: Bug Components:

[jira] [Created] (FLINK-27948) Unifying how result partition should be released

2022-06-07 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-27948: -- Summary: Unifying how result partition should be released Key: FLINK-27948 URL: https://issues.apache.org/jira/browse/FLINK-27948 Project: Flink Issue Type:

[jira] [Created] (FLINK-27947) Introduce Spark Reader for table store

2022-06-07 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-27947: Summary: Introduce Spark Reader for table store Key: FLINK-27947 URL: https://issues.apache.org/jira/browse/FLINK-27947 Project: Flink Issue Type: New

Re: Let’s meet in Texas for amazing Apache Beam workshops and talks!

2022-06-07 Thread Orozco Colin
El lun., 6 de junio de 2022 10:37 p. m., Mara Ruvalcaba escribió: > * > * > > ** > > * > > Tailoring pipelines, Powering real-time data, > > Implementing ML workflows, > > ** > > and much more at Beam Summit. > > Take the opportunity to explore in-depth Apache Beam by participating at > the

[jira] [Created] (FLINK-27946) Extending SQL syntax in a way similar to 'spark.sql.extensions'

2022-06-07 Thread liwei li (Jira)
liwei li created FLINK-27946: Summary: Extending SQL syntax in a way similar to 'spark.sql.extensions' Key: FLINK-27946 URL: https://issues.apache.org/jira/browse/FLINK-27946 Project: Flink

Re: [DISCUSS] Deprecate SourceFunction APIs

2022-06-07 Thread Alexander Fedulov
Hey Austin, Since we are getting deeper into the implementation details of the DataGeneratorSource and it is not the main topic of this thread, I propose to move our discussion to where it belongs: [DISCUSS] FLIP-238 [1]. Could you please briefly formulate your requirements to make it easier for

Re: [DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

2022-06-07 Thread Alexander Fedulov
I looked a bit further and it seems it should actually be easier than I initially thought: SourceReader extends CheckpointListener interface and with its custom implementation it should be possible to achieve similar results. A prototype that I have for the generator uses an IteratorSourceReader

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

2022-06-07 Thread Jing Ge
Hi Paul, Sorry that I am a little bit too late to join this thread. Thanks for driving this and starting this informative discussion. The FLIP looks really interesting. It will help us a lot to manage Flink SQL jobs. Have you considered the ETL scenario with Flink SQL, where multiple SQLs build

Re: [DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

2022-06-07 Thread Alexander Fedulov
Hi Steven, This is going to be tricky since in the new Source API the checkpointing aspects that you based your logic on are pushed further away from the low-level interfaces responsible for handling data and splits [1]. At the same time, the SourceCoordinatorProvider is hardwired into the

[jira] [Created] (FLINK-27945) Support HA for flink operator

2022-06-07 Thread Jeesmon Jacob (Jira)
Jeesmon Jacob created FLINK-27945: - Summary: Support HA for flink operator Key: FLINK-27945 URL: https://issues.apache.org/jira/browse/FLINK-27945 Project: Flink Issue Type: Improvement

Access to Apache Flink wiki (FLIP submission)

2022-06-07 Thread drreta
Hi Folks, Following the FLIP submission process, I would like to ask for write permissions to Apache Flink wiki space in order to create a FLIP proposal (username is reta). Thank you in advance. Best Regards, Andriy Redko

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

2022-06-07 Thread Martijn Visser
Hi Matthias, I've only thought about it, but I absolutely would +1 this. Making this information available in dedicated channels will only improve things. Best regards, Martijn Op di 7 jun. 2022 om 18:12 schreef Matthias Pohl : > Thanks for driving the Slack channel efforts and setting

[jira] [Created] (FLINK-27944) IO metric collision happens when a task has union inputs

2022-06-07 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-27944: --- Summary: IO metric collision happens when a task has union inputs Key: FLINK-27944 URL: https://issues.apache.org/jira/browse/FLINK-27944 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-237: Thrift Format Support in Flink

2022-06-07 Thread Chen Qin
Thanks for the pointers, I moved the proposal to the wiki and updated the FLIP status page. https://cwiki.apache.org/confluence/display/FLINK/FLIP-237%3A+Thrift+Format+Support Looking forward to getting community feedback. Chen On Tue, Jun 7, 2022 at 12:12 AM Jark Wu wrote: > Yes. The

Re: [DISCUSS] Deprecate SourceFunction APIs

2022-06-07 Thread Austin Cawley-Edwards
> @Austin, in the FLIP I mentioned above [1], the user is expected to pass a MapFunction to the generator. I wonder if you could have your external client and polling logic wrapped in a custom MapFunction implementation class? Would that answer your needs or do you have some more sophisticated

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

2022-06-07 Thread Matthias Pohl
Thanks for driving the Slack channel efforts and setting everything up. :-) I'm wondering whether we should extend the Slack space to also have a channel dedicated for CI builds to enable the release managers to monitor CI builds on master and the release branches through the Apache Flink Slack

Re: [DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

2022-06-07 Thread Steven Wu
In Iceberg source, we have a data generator source that can control the records per checkpoint cycle. Can we support sth like this in the DataGeneratorSource? https://github.com/apache/iceberg/blob/master/flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java

[DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

2022-06-07 Thread Alexander Fedulov
Hi everyone, I would like to open a discussion on FLIP-238: Introduce FLIP-27-based Data Generator Source [1]. During the discussion about deprecating the SourceFunction API [2] it became evident that an easy-to-use FLIP-27-compatible data generator source is needed so that the current

[jira] [Created] (FLINK-27943) Link Kubernetes Operator RoadMap Page To Flink RoadMap Page Web Site

2022-06-07 Thread ConradJam (Jira)
ConradJam created FLINK-27943: - Summary: Link Kubernetes Operator RoadMap Page To Flink RoadMap Page Web Site Key: FLINK-27943 URL: https://issues.apache.org/jira/browse/FLINK-27943 Project: Flink

[jira] [Created] (FLINK-27941) [JUnit5 Migration] Module: flink-connector-kinesis

2022-06-07 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27941: Summary: [JUnit5 Migration] Module: flink-connector-kinesis Key: FLINK-27941 URL: https://issues.apache.org/jira/browse/FLINK-27941 Project: Flink

[jira] [Created] (FLINK-27942) [JUnit5 Migration] Module: flink-connector-rabbitmq

2022-06-07 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27942: Summary: [JUnit5 Migration] Module: flink-connector-rabbitmq Key: FLINK-27942 URL: https://issues.apache.org/jira/browse/FLINK-27942 Project: Flink

[jira] [Created] (FLINK-27940) [JUnit5 Migration] Module: flink-connector-jdbc

2022-06-07 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27940: Summary: [JUnit5 Migration] Module: flink-connector-jdbc Key: FLINK-27940 URL: https://issues.apache.org/jira/browse/FLINK-27940 Project: Flink

[jira] [Created] (FLINK-27939) [JUnit5 Migration] Module: flink-connector-hive

2022-06-07 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27939: Summary: [JUnit5 Migration] Module: flink-connector-hive Key: FLINK-27939 URL: https://issues.apache.org/jira/browse/FLINK-27939 Project: Flink

[jira] [Created] (FLINK-27938) [JUnit5 Migration] Module: flink-connector-hbase

2022-06-07 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27938: Summary: [JUnit5 Migration] Module: flink-connector-hbase Key: FLINK-27938 URL: https://issues.apache.org/jira/browse/FLINK-27938 Project: Flink

[jira] [Created] (FLINK-27937) [JUnit5 Migration] Module: flink-connector-gcp-pubsub

2022-06-07 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27937: Summary: [JUnit5 Migration] Module: flink-connector-gcp-pubsub Key: FLINK-27937 URL: https://issues.apache.org/jira/browse/FLINK-27937 Project: Flink

[jira] [Created] (FLINK-27936) [JUnit5 Migration] Module: flink-connector-cassandra

2022-06-07 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-27936: Summary: [JUnit5 Migration] Module: flink-connector-cassandra Key: FLINK-27936 URL: https://issues.apache.org/jira/browse/FLINK-27936 Project: Flink

Re: [DISCUSS] Deprecate SourceFunction APIs

2022-06-07 Thread Alexander Fedulov
Hi everyone, Thanks for all the input and a lively discussion. It seems that there is a consensus that due to the inherent complexity of FLIP-27 sources we should provide more user-facing utilities to bridge the gap between the existing SourceFunction-based functionality and the new APIs. To

Re: [DISCUSS] FLIP-228: Support Within between events in CEP Pattern

2022-06-07 Thread Martijn Visser
Hi Nicholas, It is disappointing that we can't support this in SQL. I am under the impression that currently all CEP capabilities are supported in both DataStream/Table API as well as SQL. If that's indeed the case, then I would rather have this also fixed for SQL to avoid introducing feature

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

2022-06-07 Thread Martijn Visser
Hi Paul, I'm still doubting the keyword for the SQL applications. SHOW QUERIES could imply that this will actually show the query, but we're returning IDs of the running application. At first I was also not very much in favour of SHOW JOBS since I prefer calling it 'Flink applications' and not

[GitHub] [flink-connector-redis] sazzad16 commented on pull request #2: [FLINK-15571][connector][WIP] Redis Stream connector for Flink

2022-06-07 Thread GitBox
sazzad16 commented on PR #2: URL: https://github.com/apache/flink-connector-redis/pull/2#issuecomment-1148599822 @MartijnVisser It is already rebased. Just rechecked. Is there anything missing? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [flink-connector-redis] MartijnVisser commented on pull request #2: [FLINK-15571][connector][WIP] Redis Stream connector for Flink

2022-06-07 Thread GitBox
MartijnVisser commented on PR #2: URL: https://github.com/apache/flink-connector-redis/pull/2#issuecomment-1148589486 @sazzad16 One request from my end: can you rebase on the current `main` branch to get the correct ASF repository configuration in. It will avoid creating a lot of comments

[GitHub] [flink-connector-redis] MartijnVisser commented on pull request #2: [FLINK-15571][connector][WIP] Redis Stream connector for Flink

2022-06-07 Thread GitBox
MartijnVisser commented on PR #2: URL: https://github.com/apache/flink-connector-redis/pull/2#issuecomment-1148571757 @chayim Not yet, I'm bringing this PR up in our release meet-up which is scheduled for next week on Tuesday! -- This is an automated message from the Apache Git Service.

Re: Needed help with skipping savepoint state (unsure how to set --allowNonRestoredState in Docker)

2022-06-07 Thread Chesnay Schepler
You are on the right path with using the --allowNonRestoredState flag; we'll just have to find the right place to put it w.r.t. your setup. Which docker images are you using (flink/statefun/something custom), and how do you submit the job? On 03/06/2022 01:17, Bhavani Balasubramanyam wrote:

[jira] [Created] (FLINK-27935) Add Python doc of create temporary view API

2022-06-07 Thread pengmd (Jira)
pengmd created FLINK-27935: -- Summary: Add Python doc of create temporary view API Key: FLINK-27935 URL: https://issues.apache.org/jira/browse/FLINK-27935 Project: Flink Issue Type: Improvement

[GitHub] [flink-connector-redis] chayim commented on pull request #2: [FLINK-15571][connector][WIP] Redis Stream connector for Flink

2022-06-07 Thread GitBox
chayim commented on PR #2: URL: https://github.com/apache/flink-connector-redis/pull/2#issuecomment-1148455033 @MartijnVisser any luck finding someone to help? Hope you're enjoying your time off! -- This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Created] (FLINK-27934) Python API- Inefficient deserialization/serialization of state variables within a batch

2022-06-07 Thread Frans King (Jira)
Frans King created FLINK-27934: -- Summary: Python API- Inefficient deserialization/serialization of state variables within a batch Key: FLINK-27934 URL: https://issues.apache.org/jira/browse/FLINK-27934

Re: [VOTE] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-06-07 Thread Jark Wu
+1 (binding) Best, Jark On Tue, 7 Jun 2022 at 12:17, Lincoln Lee wrote: > Dear Flink developers, > > Thanks for all your feedback for FLIP-234: Support Retryable Lookup Join To > Solve Delayed Updates Issue In External Systems[1] on the discussion > thread[2]. > > I'd like to start a vote for

Re: [VOTE] FLIP-223: Support HiveServer2 Endpoint

2022-06-07 Thread Jark Wu
+1 (binding) Best, Jark On Tue, 7 Jun 2022 at 13:32, Shengkai Fang wrote: > Hi, everyone. > > Thanks for all feedback for FLIP-223: Support HiveServer2 Endpoint[1] on > the discussion thread[2]. I'd like to start a vote for it. The vote will be > open for at least 72 hours unless there is an

Re: [VOTE] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-06-07 Thread Jark Wu
+1 (binding) Best, Jark On Tue, 7 Jun 2022 at 13:44, Jing Ge wrote: > Hi Godfrey, > > +1 (non-binding) > > Best regards, > Jing > > > On Tue, Jun 7, 2022 at 4:42 AM godfrey he wrote: > > > Hi everyone, > > > > Thanks for all the feedback so far. Based on the discussion[1] we seem > > to have

Re: [DISCUSS] Planning Flink 1.16 with the community

2022-06-07 Thread Martijn Visser
We indeed set the July 25 deadline as a fixed deadline. We wanted to get back in the quicker cadence of releasing. Op di 31 mei 2022 om 09:03 schreef Robert Metzger : > > > > Is the feature freeze deadline on July 25 fixed or will it be adjusted > > accordingly? > > > Ideally we try not to push

Re: [DISCUSS] Update Scala 2.12.7 to 2.12.15

2022-06-07 Thread Martijn Visser
Hi all, @Ran When I created this discussion thread, the latest available version of Scala 2.12 was 2.12.15. I don't see 2.12.16 being released yet, only discussed. I did read that they were planning it in the last couple of weeks but haven't seen any progress since. Do you know more about the

[jira] [Created] (FLINK-27933) Savepoint OperationResult should be serializable

2022-06-07 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-27933: Summary: Savepoint OperationResult should be serializable Key: FLINK-27933 URL: https://issues.apache.org/jira/browse/FLINK-27933 Project: Flink

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

2022-06-07 Thread Jing Ge
got it, thanks Martijn! Best regards, Jing On Tue, Jun 7, 2022 at 10:38 AM Martijn Visser wrote: > Hi Jing, > > Yes, since this Flink Community workspace is treated as one company. So > everyone you want to invite, should considered a 'coworker'. > > Best regards, > > Martijn > > Op za 4 jun.

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

2022-06-07 Thread Martijn Visser
Hi Jing, Yes, since this Flink Community workspace is treated as one company. So everyone you want to invite, should considered a 'coworker'. Best regards, Martijn Op za 4 jun. 2022 om 20:02 schreef Jing Ge : > Hi Xingtong, > > While inviting new members, there are two options: "From another

Re: [VOTE] FLIP-224: Blocklist Mechanism

2022-06-07 Thread Zhu Zhu
Hi Chesnay, For your information, one major goal of blocklist mechanism is to support FLIP-168(speculative execution of batch jobs). When speculative execution happens, it needs to keep the existing tasks running and launch speculative tasks on other nodes. We have heard request of speculative

[jira] [Created] (FLINK-27932) align noWatermarks and withWatemarkAlignment API in PyFlink

2022-06-07 Thread Juntao Hu (Jira)
Juntao Hu created FLINK-27932: - Summary: align noWatermarks and withWatemarkAlignment API in PyFlink Key: FLINK-27932 URL: https://issues.apache.org/jira/browse/FLINK-27932 Project: Flink Issue

[jira] [Created] (FLINK-27931) Introduce the SqlGateway to assemble all components

2022-06-07 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27931: - Summary: Introduce the SqlGateway to assemble all components Key: FLINK-27931 URL: https://issues.apache.org/jira/browse/FLINK-27931 Project: Flink Issue

Re: [VOTE] FLIP-224: Blocklist Mechanism

2022-06-07 Thread Chesnay Schepler
I've had some time to think about it and concluded to stick to my -1. While BLOCK_WITH_QUARANTINE is easy to implement (un-register TMs and ignore all RPCs (the latter mostly happens automatically)) it doesn't add a whole lot of value as it's pretty much equivalent with shutting down the TM.

[jira] [Created] (FLINK-27930) Format the timestamp in status to make it explicit

2022-06-07 Thread Aitozi (Jira)
Aitozi created FLINK-27930: -- Summary: Format the timestamp in status to make it explicit Key: FLINK-27930 URL: https://issues.apache.org/jira/browse/FLINK-27930 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-237: Thrift Format Support in Flink

2022-06-07 Thread Jark Wu
Yes. The community recommends keeping content on the wiki page and discuss in the mailing list. Discussions on the google doc are not so visible to the community, and "If it didn’t happen on a mailing list, it didn’t happen." Best, Jark On Tue, 7 Jun 2022 at 14:15, Jing Ge wrote: > Hi Chen, >

[jira] [Created] (FLINK-27929) Drop support for python 3.6

2022-06-07 Thread LuNng Wang (Jira)
LuNng Wang created FLINK-27929: -- Summary: Drop support for python 3.6 Key: FLINK-27929 URL: https://issues.apache.org/jira/browse/FLINK-27929 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-237: Thrift Format Support in Flink

2022-06-07 Thread Jing Ge
Hi Chen, Thanks for driving this! Afaik, the community has the consensus to *Start a [DISCUSS] thread on the Apache mailing list*[1]. I just walked through some existing FLIPs and didn't find any that have been using google doc as the discussion thread. Would you like to follow the current