Re: [VOTE] Release flink-connector-opensearch, release candidate #1

2022-12-16 Thread Martijn Visser
Hi Danny, +1 (binding) - Validated hashes - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven - Verified licenses - Verified web PRs Thanks for the help! Best regards, Martijn On Fri, Dec 16, 2022 at 5:37 PM Danny Cranmer wrote: >

Streaming queries in FTS using Kafka log

2022-12-16 Thread Alexander Sorokoumov
Hello community, I want to ask about streaming queries with Flink Table Store. After reading the documentation on Streaming Queries [1], I was under the impression that only tables with LogStore-over-TableStore and No Changelog Producer need the normalization step since the Kafka log has the

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

2022-12-16 Thread Jun Qin
Congratulations to both of you, Godfrey and Xingbo! > On Nov 25, 2022, at 1:29 AM, Dong Lin wrote: > > Congratulations, Godfrey and Xingbo! > > > On Wed, Nov 23, 2022 at 12:19 PM Dian Fu wrote: > >> Hi everyone, >> >> On behalf of the Apache Flink PMC, I'm very happy to announce that

[jira] [Created] (FLINK-30446) [flink-playgrounds] Use more recent MySQL image built for linux/arm64/v8 & linux/amd64

2022-12-16 Thread Connor Ameres (Jira)
Connor Ameres created FLINK-30446: - Summary: [flink-playgrounds] Use more recent MySQL image built for linux/arm64/v8 & linux/amd64 Key: FLINK-30446 URL: https://issues.apache.org/jira/browse/FLINK-30446

[jira] [Created] (FLINK-30445) Add Flink Web3 Connector

2022-12-16 Thread Junyao Huang (Jira)
Junyao Huang created FLINK-30445: Summary: Add Flink Web3 Connector Key: FLINK-30445 URL: https://issues.apache.org/jira/browse/FLINK-30445 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-30444) State recovery error not handled correctly and always causes JM failure

2022-12-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-30444: -- Summary: State recovery error not handled correctly and always causes JM failure Key: FLINK-30444 URL: https://issues.apache.org/jira/browse/FLINK-30444 Project: Flink

RE: Re: [DICSUSS] Releasing Flink Opensearch Connector v1.0.0

2022-12-16 Thread Payne, Julian
Would be great to include 1.15 as well. +1 On 2022/12/16 10:49:56 Danny Cranmer wrote: > Hey, > > I agree it makes sense for this connector to have a dedicated repository. > > I can confirm that the connector does not compile against Flink 1.15 due to > missing test dependencies that were added

RE: [VOTE] Release flink-connector-opensearch, release candidate #1

2022-12-16 Thread Payne, Julian
+1 On 2022/12/16 12:40:42 Danny Cranmer wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version 1.0.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is

Re: [VOTE] Update Flink's Scala 2.12 support from 2.12.7 to 2.12.16

2022-12-16 Thread Martijn Visser
Hi all, I'm bumping this old vote thread once more. If we want to add Java 17 support at some point, we will need to update our Scala 2.12 version (see https://issues.apache.org/jira/browse/FLINK-25000). As explained before in the discussion thread, that means that we would break binary &

Re: [VOTE] Release flink-connector-opensearch, release candidate #1

2022-12-16 Thread Danny Cranmer
Apologies I messed up the link to "the official Apache source release to be deployed to dist.apache.org" [1] Thanks, Danny [1] https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-1.0.0-rc1 On Fri, Dec 16, 2022 at 2:04 PM Ahmed Hamdy wrote: > Thank you Danny, > > +1

[jira] [Created] (FLINK-30443) Expand list of sensitive keys

2022-12-16 Thread Gunnar Morling (Jira)
Gunnar Morling created FLINK-30443: -- Summary: Expand list of sensitive keys Key: FLINK-30443 URL: https://issues.apache.org/jira/browse/FLINK-30443 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-30442) Update table walkthrough playground for 1.12

2022-12-16 Thread David Anderson (Jira)
David Anderson created FLINK-30442: -- Summary: Update table walkthrough playground for 1.12 Key: FLINK-30442 URL: https://issues.apache.org/jira/browse/FLINK-30442 Project: Flink Issue Type:

[jira] [Created] (FLINK-30440) Update operations playground for 1.16

2022-12-16 Thread David Anderson (Jira)
David Anderson created FLINK-30440: -- Summary: Update operations playground for 1.16 Key: FLINK-30440 URL: https://issues.apache.org/jira/browse/FLINK-30440 Project: Flink Issue Type:

[jira] [Created] (FLINK-30441) Update pyflink walkthrough playground for 1.16

2022-12-16 Thread David Anderson (Jira)
David Anderson created FLINK-30441: -- Summary: Update pyflink walkthrough playground for 1.16 Key: FLINK-30441 URL: https://issues.apache.org/jira/browse/FLINK-30441 Project: Flink Issue

[jira] [Created] (FLINK-30439) Update playgrounds for 1.16

2022-12-16 Thread David Anderson (Jira)
David Anderson created FLINK-30439: -- Summary: Update playgrounds for 1.16 Key: FLINK-30439 URL: https://issues.apache.org/jira/browse/FLINK-30439 Project: Flink Issue Type: Improvement

Re: [VOTE] FLIP-277: Glue catalog support in Flink

2022-12-16 Thread Dong Lin
+1 (binding) Thanks for driving this! On Fri, Dec 16, 2022 at 6:17 PM Samrat Deb wrote: > Hi All, Thanks for all the feedback so far. Based on the discussion[1] we > seem to have a consensus, so I would like to start a vote on FLIP-277[2]. > The vote will last for at least 72 hours (Dec 21st

Re: [VOTE] Release flink-connector-opensearch, release candidate #1

2022-12-16 Thread Ahmed Hamdy
Thank you Danny, +1 (non-binding) * Hashes and Signatures look good * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven * Verified Notice and Licence files On Fri, 16 Dec 2022 at 12:41, Danny Cranmer wrote: > Hi everyone,

Re: [VOTE] Release flink-connector-opensearch, release candidate #1

2022-12-16 Thread Andrey Redko
+1 (non-binding), thanks a lot, Danny! Best Regards, Andriy Redko On Fri, Dec 16, 2022, 7:41 AM Danny Cranmer wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version 1.0.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release

[jira] [Created] (FLINK-30438) The generated schema is not correct when using value.format debezium-avro-confluent

2022-12-16 Thread Jun Qin (Jira)
Jun Qin created FLINK-30438: --- Summary: The generated schema is not correct when using value.format debezium-avro-confluent Key: FLINK-30438 URL: https://issues.apache.org/jira/browse/FLINK-30438 Project:

RE: [VOTE] Release flink-connector-opensearch, release candidate #1

2022-12-16 Thread Payne, Julian
+1 On 2022/12/16 12:40:42 Danny Cranmer wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version 1.0.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is

[jira] [Created] (FLINK-30437) Last-state upgrade combined with state incompatibility causes state loss

2022-12-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-30437: -- Summary: Last-state upgrade combined with state incompatibility causes state loss Key: FLINK-30437 URL: https://issues.apache.org/jira/browse/FLINK-30437 Project: Flink

[VOTE] Release flink-connector-opensearch, release candidate #1

2022-12-16 Thread Danny Cranmer
Hi everyone, Please review and vote on the release candidate #1 for the version 1.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

Re: [DISCUSS] assign SQL Table properties from environment variables

2022-12-16 Thread Martijn Visser
Hi Ferenc, I think this is indeed an important topic to have a discussion about. It probably also makes sense to create a FLIP and have a vote on this. One potential option that we could have is to have a generic option to refer to external data when specifying a connector option, like

Re: [DISCUSS] FLIP-278: Hybrid Source Connector

2022-12-16 Thread Ran Tao
Hi, Martijn, thanks for your comments. Using identifier as child source prefix may be a good way instead of index. i will update the flip to illustrate how we can read from hybrid schema to generate child schemas for the question1. question2 is start position for the next kafka source. But

Re: [VOTE] FLIP-277: Glue catalog support in Flink

2022-12-16 Thread Samrat Deb
Hi Jark , Update flip in the configuration section with a note where glue catalog support is intended to add . Thank you for calling that out . On Fri, Dec 16, 2022 at 4:16 PM Danny Cranmer wrote: > +1 (binding) > > Thanks for driving this Samrat, looking forward to it. > > Danny. > > On

Re: [VOTE] FLIP-277: Glue catalog support in Flink

2022-12-16 Thread yuxia
+1 (non-binding) Thanks for driving it. Best regards, Yuxia - 原始邮件 - 发件人: "Danny Cranmer" 收件人: "dev" 发送时间: 星期五, 2022年 12 月 16日 下午 6:46:12 主题: Re: [VOTE] FLIP-277: Glue catalog support in Flink +1 (binding) Thanks for driving this Samrat, looking forward to it. Danny. On Fri, Dec

[jira] [Created] (FLINK-30436) Integrate Opensearch connector docs into Flink docs v1.16

2022-12-16 Thread Danny Cranmer (Jira)
Danny Cranmer created FLINK-30436: - Summary: Integrate Opensearch connector docs into Flink docs v1.16 Key: FLINK-30436 URL: https://issues.apache.org/jira/browse/FLINK-30436 Project: Flink

Re: [DICSUSS] Releasing Flink Opensearch Connector v1.0.0

2022-12-16 Thread Danny Cranmer
Hey, I agree it makes sense for this connector to have a dedicated repository. I can confirm that the connector does not compile against Flink 1.15 due to missing test dependencies that were added in Flink 1.16. Also there are dependency convergence issues. I will go ahead and create an RC1

Re: [VOTE] FLIP-277: Glue catalog support in Flink

2022-12-16 Thread Danny Cranmer
+1 (binding) Thanks for driving this Samrat, looking forward to it. Danny. On Fri, Dec 16, 2022 at 10:26 AM Jark Wu wrote: > +1 (binding) > > It would be better to mention in the FLIP which repository and module to > put the Glue Catalog. > > Best, > Jark > > On Fri, 16 Dec 2022 at 18:18,

[jira] [Created] (FLINK-30435) `SHOW CREATE TABLE` shows column comment

2022-12-16 Thread Yubin Li (Jira)
Yubin Li created FLINK-30435: Summary: `SHOW CREATE TABLE` shows column comment Key: FLINK-30435 URL: https://issues.apache.org/jira/browse/FLINK-30435 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-30434) The non-null check for transactionalIdPrefix is unnecessary in the KafkaWriter constructor

2022-12-16 Thread Zhengqi Zhang (Jira)
Zhengqi Zhang created FLINK-30434: - Summary: The non-null check for transactionalIdPrefix is unnecessary in the KafkaWriter constructor Key: FLINK-30434 URL: https://issues.apache.org/jira/browse/FLINK-30434

Re: [VOTE] FLIP-277: Glue catalog support in Flink

2022-12-16 Thread Jark Wu
+1 (binding) It would be better to mention in the FLIP which repository and module to put the Glue Catalog. Best, Jark On Fri, 16 Dec 2022 at 18:18, Samrat Deb wrote: > Hi All, Thanks for all the feedback so far. Based on the discussion[1] we > seem to have a consensus, so I would like to

Re: [DISCUSS] TLS issues

2022-12-16 Thread Martijn Visser
Thanks to Gabor for taking a look :) On Tue, Dec 6, 2022 at 2:42 PM Jean-Damien Hatzenbuhler wrote: > Hello, > > Is there a Flink Community committer that can take a look at the PR > https://issues.apache.org/jira/browse/FLINK-28520? > > Regards, > > On Mon, Sep 19, 2022 at 4:10 PM Martijn

Re: [VOTE] FLIP-277: Improve Catalog API to Support ALTER TABLE syntax

2022-12-16 Thread Samrat Deb
Apologies ! Please ignore this email. I have sent a fresh one correcting the subject :( On Fri, Dec 16, 2022 at 3:45 PM Samrat Deb wrote: > Hi All, Thanks for all the feedback so far. Based on the discussion[1] we > seem to have a consensus, so I would like to start a vote on FLIP-277[2]. > The

[VOTE] FLIP-277: Glue catalog support in Flink

2022-12-16 Thread Samrat Deb
Hi All, Thanks for all the feedback so far. Based on the discussion[1] we seem to have a consensus, so I would like to start a vote on FLIP-277[2]. The vote will last for at least 72 hours (Dec 21st at 11:00 GMT, excluding weekend days) unless there is an objection or insufficient votes. Please

[VOTE] FLIP-277: Improve Catalog API to Support ALTER TABLE syntax

2022-12-16 Thread Samrat Deb
Hi All, Thanks for all the feedback so far. Based on the discussion[1] we seem to have a consensus, so I would like to start a vote on FLIP-277[2]. The vote will last for at least 72 hours (Dec 21st at 11:00 GMT, excluding weekend days) unless there is an objection or insufficient votes. Best,

[DISCUSS] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

2022-12-16 Thread Jing Ge
Hi Dev, I'd like to start a discussion about FLIP-279: Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode[1]. On one side, currently, there is only one ConfigOption SqlClientOptions.DISPLAY_MAX_COLUMN_WIDTH(sql-client.display.max-column-width')

Re: Table API thrift support

2022-12-16 Thread Martijn Visser
Hi Chen, If you want to split these two things, then "We suggest user upgrade hive metastore version to match thrift version given keeping two thrift version leads to hive connector dependencies complicated to manage." seems like something that won't work. This reads like in order for you to

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2022-12-16 Thread Shammon FY
Hi Piotr, I found there may be several points in our discussion, it will cause misunderstanding between us when we focus on different one. I list each point in our discussion as follows > Point 1: Is "Aligned Checkpoint" the only mechanism to guarantee data consistency in the current Flink

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.3.0 released

2022-12-16 Thread Biao Geng
Congratulations! Thanks a lot for the awesome work! Best regards, Biao Geng Őrhidi Mátyás 于2022年12月15日周四 01:30写道: > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.3.0. > > Release highlights: > >- Upgrade to Fabric8 6.x.x and JOSDK

Re: [DISCUSS] Release new FRocksDB

2022-12-16 Thread Martijn Visser
Hi Yanfei, Big +1 for adding ARM support. It would be great to be able to get tests to pass on Mac M1s. Best regards, Martijn On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi wrote: > Hi Yanfei, > > Makes sense, arm architecture support would be great. Thanks for looking > into this. > > On

Re: Edit permission

2022-12-16 Thread Martijn Visser
Hi Ted, I've granted you access. Best regards, Martijn On Mon, Dec 12, 2022 at 7:23 PM Hao t Chang wrote: > I didn’t provide my cwiki username☹. It’s tedhtchang. >

Re: [DISCUSS] FLIP-278: Hybrid Source Connector

2022-12-16 Thread Martijn Visser
Hi Ran, For completeness, this is a new thread that was already previously started at https://lists.apache.org/thread/xptn2ddzj34q9f5vtbfb62lsybmvcwjq. I'm linking them because I think Timo's comments are relevant to be kept with this discussion thread. I agree with Timo's comments from there

[jira] [Created] (FLINK-30433) HiveDialectITCase#testCreateDatabase fails with "Could not execute CREATE DATABASE"

2022-12-16 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30433: -- Summary: HiveDialectITCase#testCreateDatabase fails with "Could not execute CREATE DATABASE" Key: FLINK-30433 URL: https://issues.apache.org/jira/browse/FLINK-30433

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2022-12-16 Thread Piotr Nowojski
Hi Shammon, > I don't think we can combine watermarks and checkpoint barriers together to > guarantee data consistency. There will be a "Timestamp Barrier" in our > system to "commit data", "single etl failover", "low latency between ETLs" > and "strong data consistency with completed semantics"