[RESULT] [VOTE] Release 1.19.1, release candidate #1
Hi all, I'm happy to announce that we have unanimously approved this release. [1] There are 11 approving votes, 5 of which are binding: * Rui Fan (binding) * Xiqian Yu (non-binding) * Weijie Guo (binding) * Jeyhun (non-binding) * Ahmed Hamdy (non-binding) * Xintong Song (binding) * Matthias Pohl (binding) * Sergey Nuyanzin (non-binding) * Leonard Xu (binding) * Zhongqiang Gong (non-binding) * Hang Ruan (non-binding) There are no disapproving votes. Thanks everyone! [1] https://lists.apache.org/thread/hrptj22y6rjt61flzdzngxdsw134osk4 Regards, Hong
Re: [VOTE] Release 1.19.1, release candidate #1
Thanks all for the testing and votes. The RC is approved and this thread is now closed. See results in [1]. [1] https://lists.apache.org/thread/yqr3jv4wr85brnz2ylzqo9pqn453jqvq Regards, Hong On Tue, Jun 11, 2024 at 9:39 AM Hang Ruan wrote: > +1(non-binding) > > - Verified signatures > - Verified hashsums > - Checked Github release tag > - Source archives with no binary files > - Reviewed the flink-web PR > - Checked the jar build with jdk 1.8 > > Best, > Hang > > gongzhongqiang 于2024年6月11日周二 15:53写道： > > > +1(non-binding) > > > > - Verified signatures and sha512 > > - Checked Github release tag exsit > > - Source archives with no binary files > > - Build the source with jdk8 on ubuntu 22.04 succeed > > - Reviewed the flink-web PR > > > > Best, > > Zhongqiang Gong > > > > Hong Liang 于2024年6月6日周四 23:39写道： > > > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the flink > v1.19.1, > > > as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source release and binary convenience releases to > > be > > > deployed to dist.apache.org [2], which are signed with the key with > > > fingerprint B78A5EA1 [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tag "release-1.19.1-rc1" [5], > > > * website pull request listing the new release and adding announcement > > blog > > > post [6]. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Hong > > > > > > [1] > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354399 > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.19.1-rc1/ > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] > > > > https://repository.apache.org/content/repositories/orgapacheflink-1736/ > > > [5] https://github.com/apache/flink/releases/tag/release-1.19.1-rc1 > > > [6] https://github.com/apache/flink-web/pull/745 > > > > > >
[jira] [Created] (FLINK-35573) [FLIP-447] Upgrade FRocksDB from 6.20.3 to 8.10.0
Yue Ma created FLINK-35573: -- Summary: [FLIP-447] Upgrade FRocksDB from 6.20.3 to 8.10.0 Key: FLINK-35573 URL: https://issues.apache.org/jira/browse/FLINK-35573 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 The FLIP: [https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0|https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0] *_This FLIP proposes upgrading the version of FRocksDB in the Flink Project from 6.20.3 to 8.10.0._* _RocksDBStateBackend is widely used by Flink users in large state scenarios.The last upgrade of FRocksDB was in version Flink-1.14, which mainly supported features such as support arm platform, deleteRange API, period compaction, etc. It has been a long time since then, and RocksDB has now been released to version 8.x. The main motivation for this upgrade is to leverage the features of higher versions of Rocksdb to make Flink RocksDBStateBackend more powerful. While RocksDB is also continuously optimizing and bug fixing, we hope to keep FRocksDB more or less in sync with RocksDB and upgrade it periodically._ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35574) Set up base branch for FrocksDB-8.10
Yue Ma created FLINK-35574: -- Summary: Set up base branch for FrocksDB-8.10 Key: FLINK-35574 URL: https://issues.apache.org/jira/browse/FLINK-35574 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 As the first part of FLINK-35573, we need to prepare a base branch for FRocksDB-8.10.0 first. Mainly, it needs to be checked out from version 8.10.0 of the Rocksdb community. Then check pick the commit which used by Flink from FRocksDB-6.20.3 to 8.10.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35575) FRocksDB supports disabling perf context during compilation
Yue Ma created FLINK-35575: -- Summary: FRocksDB supports disabling perf context during compilation Key: FLINK-35575 URL: https://issues.apache.org/jira/browse/FLINK-35575 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 In FrocksDB 6 thread-local perf-context is disabled by reverting a specific commit (FLINK-19710). However, this creates conflicts and makes upgrading more difficult. We found that disabling *PERF_CONTEXT* can improve the performance of statebenchmark by about 5% and it doesn't create any conflicts. So we plan to supports disabling perf context during compilation in FRocksDB new version -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35576) FRocksdb Cherry pick IngestDB requires commit
Yue Ma created FLINK-35576: -- Summary: FRocksdb Cherry pick IngestDB requires commit Key: FLINK-35576 URL: https://issues.apache.org/jira/browse/FLINK-35576 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 We support the API related to ingest DB in FRocksDb-8.10.0, but many of the fixes related to ingest DB were only integrated in the latest RocksDB version. So we need to add these fixed commit cherryclicks to FRocksDB. Mainly include: [https://github.com/facebook/rocksdb/pull/11646] [https://github.com/facebook/rocksdb/pull/11868] [https://github.com/facebook/rocksdb/pull/11811] [https://github.com/facebook/rocksdb/pull/11381] [https://github.com/facebook/rocksdb/pull/11379] [https://github.com/facebook/rocksdb/pull/11378] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35577) Setup the CI environment for FRocksDB-8.10
Yue Ma created FLINK-35577: -- Summary: Setup the CI environment for FRocksDB-8.10 Key: FLINK-35577 URL: https://issues.apache.org/jira/browse/FLINK-35577 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn
Hi all, Thanks for all your votes, I hereby close the vote and I'll announce the results in a separate email. Best, Yuxin Zhu Zhu 于2024年6月12日周三 11:15写道： > +1 (binding) > > Thanks, > Zhu > > Yuepeng Pan 于2024年6月11日周二 17:04写道： > > > +1 (non-binding) > > > > Best regards, > > Yuepeng Pan > > > > At 2024-06-11 16:34:12, "Rui Fan" <1996fan...@gmail.com> wrote: > > >+1（binding） > > > > > >Best, > > >Rui > > > > > >On Tue, Jun 11, 2024 at 4:14 PM Muhammet Orazov > > > wrote: > > > > > >> +1 (non-binding) > > >> > > >> Thanks Yuxin for driving this! > > >> > > >> Best, > > >> Muhammet > > >> > > >> > > >> On 2024-06-07 08:02, Yuxin Tan wrote: > > >> > Hi everyone, > > >> > > > >> > Thanks for all the feedback about the FLIP-459 Support Flink > > >> > hybrid shuffle integration with Apache Celeborn[1]. > > >> > The discussion thread is here [2]. > > >> > > > >> > I'd like to start a vote for it. The vote will be open for at least > > >> > 72 hours unless there is an objection or insufficient votes. > > >> > > > >> > [1] > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > >> > [2] > https://lists.apache.org/thread/gy7sm7qrf7yrv1rl5f4vtk5fo463ts33 > > >> > > > >> > Best, > > >> > Yuxin > > >> > > >
[jira] [Created] (FLINK-35578) Release Frocksdb-8.10.0 official products
Yue Ma created FLINK-35578: -- Summary: Release Frocksdb-8.10.0 official products Key: FLINK-35578 URL: https://issues.apache.org/jira/browse/FLINK-35578 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35579) Update the FrocksDB version in FLINK
Yue Ma created FLINK-35579: -- Summary: Update the FrocksDB version in FLINK Key: FLINK-35579 URL: https://issues.apache.org/jira/browse/FLINK-35579 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35581) Remove comments from the code related to ingestDB
Yue Ma created FLINK-35581: -- Summary: Remove comments from the code related to ingestDB Key: FLINK-35581 URL: https://issues.apache.org/jira/browse/FLINK-35581 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35580) Fix ingestDB recovery mode related bugs
Yue Ma created FLINK-35580: -- Summary: Fix ingestDB recovery mode related bugs Key: FLINK-35580 URL: https://issues.apache.org/jira/browse/FLINK-35580 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35582) Marking ingestDB as the default recovery mode for rescaling
Yue Ma created FLINK-35582: -- Summary: Marking ingestDB as the default recovery mode for rescaling Key: FLINK-35582 URL: https://issues.apache.org/jira/browse/FLINK-35582 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Affects Versions: 2.0.0 Reporter: Yue Ma Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[RESULT][VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn
Hi, all I am happy to say that FLIP-459 Support Flink hybrid shuffle integration with Apache Celeborn [1] has been accepted. There are 11 votes, of which 4 are binding[2]. Xintong Song (binding) Zhu Zhu (binding) weijie guo (binding) Jim Hughes (non-binding) Jeyhun Karimov (non-binding) Ahmed Hamdy (non-binding) Venkatakrishnan Sowrirajan (non-binding) Junrui Lee (non-binding) Muhammet Orazov (non-binding) Rui Fan (binding) Yuepeng Pan (non-binding) [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn [2] https://lists.apache.org/thread/l64ykk3n8c2gc40gjbowt0ozs0x0jmqm Best, Yuxin
[jira] [Created] (FLINK-35583) Flinker-connector-jdbc is expected to support DDL synchronization for mysql
linux created FLINK-35583: - Summary: Flinker-connector-jdbc is expected to support DDL synchronization for mysql Key: FLINK-35583 URL: https://issues.apache.org/jira/browse/FLINK-35583 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: linux I strongly hope the flinker-connector-jdbc is expected to support DDL synchronization for mysql,This use case is very common and is a feature that many users now expect to have,Hope the official can enhance this function,thanks a lot! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35584) Support Java 21 in flink-docker
Josh England created FLINK-35584: Summary: Support Java 21 in flink-docker Key: FLINK-35584 URL: https://issues.apache.org/jira/browse/FLINK-35584 Project: Flink Issue Type: Improvement Components: flink-docker Reporter: Josh England Support Java 21. Base images are available for 8, 11 and 17 but since Apache flink now supports Java 21 (albeit in Beta) it would be good to have a base image for that too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re:Re: Re: [DISCUSS] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join
Hi Jingsong, Some of the points you mentioned are currently clarified in the updated FLIP. Please check it out. 1. Enabling custom data distribution can be done through the LOOKUP SQL Hint. There are detailed examples provided in the FLIP. 2. We will add the isDeterministic method to the `InputDataPartitioner` interface, which will return true by default. If the `InputDataPartitioner` is not deterministic, the connector developer need to override the isDeterministic method to return false. If the connector developer cannot ensure this protocol, they will need to bear the correctness issues that arise. 3. Yes, this feature will work in batch mode as well. Best regards, Wencong At 2024-06-11 23:47:40, "Jingsong Li" wrote: >Hi all, > >+1 to this FLIP, very thanks all for your proposal. > >isDeterministic looks good to me too. > >We can consider stating the following points: > >1. How to enable custom data distribution? Is it a dynamic hint? Can >you provide an SQL example. > >2. What impact will it have when the mainstream is changelog? Causing >disorder? This may need to be emphasized. > >3. Does this feature work in batch mode too? > >Best, >Jingsong > >On Tue, Jun 11, 2024 at 8:22 PM Wencong Liu wrote: >> >> Hi Lincoln, >> >> >> Thanks for your reply. Weijie and I discussed these two issues offline, >> and here are the results of our discussion: >> 1. When the user utilizes the hash lookup join hint introduced by >> FLIP-204[1], >> the `SupportsLookupCustomShuffle` interface should be ignored. This is >> because >> the hash lookup join hint is directly specified by the user through a SQL >> HINT, >> which is more in line with user intuition. WDYT? >> 2. We agree with the introduction of the `isDeterministic` method. The >> `SupportsLookupCustomShuffle` interface introduces a custom shuffle, which >> can cause ADD/UPDATE_AFTER events (+I, +U) to appear >> after UPDATE_BEFORE/DELETE events (-D, -U), thus breaking the current >> limitations of the Flink Sink Operator[2]. If `isDeterministic` returns >> false and the >> changelog event type is not insert-only, the Planner should not apply the >> shuffle >> provided by `SupportsLookupCustomShuffle`. >> >> >> [1] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-204%3A+Introduce+Hash+Lookup+Join >> [2] >> https://www.ververica.com/blog/flink-sql-secrets-mastering-the-art-of-changelog-event-out-of-orderness >> >> >> Best, >> Wencong >> >> >> >> >> >> >> >> >> >> At 2024-06-11 00:02:57, "Lincoln Lee" wrote: >> >Hi Weijie, >> > >> >Thanks for your proposal, this will be a useful advanced optimization for >> >connector developers! >> > >> >I have two questions: >> > >> >1. FLIP-204[1] hash lookup join hint is mentioned in this FLIP, what's the >> >apply ordering of the two feature? For example, a connector that >> >implements the `SupportsLookupCustomShuffle` interface also has a >> >`SHUFFLE_HASH` lookup join hint specified by the user in sql, what's >> >the expected behavior? >> > >> >2. This FLIP considers the relationship with NDU processing, and I agree >> >with the current choice to prioritize NDU first. However, we should also >> >consider another issue: out-of-orderness of the changelog events in >> >streaming[2]. If the connector developer supplies a non-deterministic >> >partitioner, e.g., a random partitioner for anti-skew purpose, then it'll >> >break the assumption relied by current SQL operators in streaming: the >> >ADD/UDPATE_AFTER events (+I, +U) always occur before its related >> >UDPATE_BEFORE/DELETE events (-D, -U) and they are always >> >processed by the same task even if a data shuffle is involved. So a >> >straightforward approach would be to add method `isDeterministic` to >> >the `InputDataPartitioner` interface to explicitly tell the planner whether >> >the partitioner is deterministic or not(then the planner can reject the >> >non-deterministic custom partitioner for correctness requirements). >> > >> >[1] >> >https://cwiki.apache.org/confluence/display/FLINK/FLIP-204%3A+Introduce+Hash+Lookup+Join >> >[2] >> >https://www.ververica.com/blog/flink-sql-secrets-mastering-the-art-of-changelog-event-out-of-orderness >> > >> > >> >Best, >> >Lincoln Lee >> > >> > >> >Xintong Song 于2024年6月7日周五 13:53写道： >> > >> >> +1 for this proposal. >> >> >> >> This FLIP will make it possible for each lookup join parallel task to only >> >> access and cache a subset of the data. This will significantly improve the >> >> performance and reduce the overhead when using Paimon for the dimension >> >> table. And it's general enough to also be leveraged by other connectors. >> >> >> >> Best, >> >> >> >> Xintong >> >> >> >> >> >> >> >> On Fri, Jun 7, 2024 at 10:01 AM weijie guo >> >> wrote: >> >> >> >> > Hi devs, >> >> > >> >> > >> >> > I'd like to start a discussion about FLIP-462[1]: Support Custom Data >> >> > Distribution for Input Stream of Lookup Join. >> >> > >> >> > >> >> > Lookup Join is an important feature in Flink, It
Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement
Hey Yanquan, I just added the WITH clause to the CTAS syntax in the flip. > If we have a source table with primary keys and partition keys defined, > what is the default behavior if PARTITIONED and DISTRIBUTED not specified > in the CTAS statement, It should not be inherited by default？ You're right. The default behavior does not infer the primary key from the query schema. It only brings column names, column types and nullability constraints. So the reason to support the primary key definition in the create part is important. - Sergio On Tue, Jun 11, 2024 at 9:32 PM Yanquan Lv wrote: > Hi Sergio, thanks for driving it, +1 for this. > > I have some comments: > 1. If we have a source table with primary keys and partition keys defined, > what is the default behavior if PARTITIONED and DISTRIBUTED not specified > in the CTAS statement, It should not be inherited by default？ > 2. I suggest providing a complete syntax that includes table_properties > like FLIP-218. > > > Sergio Pena 于2024年6月12日周三 03:54写道： > > > I just noticed the CREATE TABLE LIKE statement allows the definition of > new > > columns in the CREATE part. The difference > > with this CTAS proposal is that TABLE LIKE appends the new columns at the > > end of the schema instead of adding them > > at the beginning like this proposal and Mysql do. > > > > > create table t1(id int, name string); > > > > create table s1(a int, b string) like t1; > > > > describe s1; > > > > +-+---+--++ > > > | Column Name | Data Type | Nullable | Extras | > > > +-+---+--++ > > > | id | INT | NULL || > > > | name| STRING| NULL || > > > | a | INT | NULL || > > > | b | STRING| NULL || > > > +-+---+--++ > > > > > > > > The CREATE TABLE LIKE also does not let the definition of existing > columns > > in the CREATE part. The statement fails > > that the column already exists. > > > > > create table t1(id int, name string); > > > > > create table s1(id double) like t1; > > > A column named 'id' already exists in the base table. > > > > > > > What do you guys think of making it similar to the CREATE TABLE LIKE? > Seems > > the best approach in order to > > be compatible with it. > > > > - Sergio > > > > On Tue, Jun 11, 2024 at 2:10 PM Sergio Pena wrote: > > > > > Thanks Timo for answering Jeyhun questions. > > > > > > To add info more about your questions Jeyhun. This proposal is not > > > handling NULL/NOT_NULL types. I noticed that > > > the current CTAS impl. (as Timo said) adds this constraint as part of > the > > > resulting schema. And when defining > > > a primary key in the CREATE part, if the resulting schema does not > have a > > > NOT NULL in the column then the CTAS > > > will fail. This is similar to the CREATE TABLE LIKE which expects the > > LIKE > > > table to have a NOT NULL column if > > > the user defines a primary key in the CREATE part. > > > > > > > In some cases, redefining the column types might be redundant, > > especially > > > > when users dont change the column type. A user just wants to change > the > > > > column name from the SELECT clause. Should we also support this > > scenario, > > > > similar to postgres? > > > > > > I looked into Postgres too. Postgres matches the columns based on the > > > order defined in the create and select part. > > > If you want to rename a column in the create part, then that column > > > position must be in the same position as the query column. > > > I didn't like the Postgres approach because it does not let us add > > columns > > > that do not exist in the query schema. > > > > > > i.e. query has schema (a int, b string), now the `a` column is renamed > to > > > `id` because both are in the same position 0 > > > `create table s1(id int) as select a, b from t1`; > > > results in: [id int, b string] > > > > > > I think, if users want to rename then they can use a different alias in > > > the select part. They could also do explicit casting > > > for changing the data types, which now makes it redundant (as you said) > > to > > > allow redefining the query columns again. But > > > perhaps there are cases where explicit casting does not work and just > > > defining the column would? i.e. making a nullable > > > type to not null? I couldn't make `cast(c1 as int not null)` to work > for > > > instance, but it may work in the create part? > > > > > > > Could you also mention the casting rules in the FLIP for this case? > > > > > > I mentioned they're the same as insert/select when doing implicit > > casting. > > > I will search for more info about the insert/select > > > and add the casting rules in the flip.. > > > > > > - Sergio > > > > > > > > > On Tue, Jun 11, 2024 at 12:59 AM Timo Walther > > wrote: > > > > > >> Hi Sergio, > > >> > > >> thanks for proposing this FLIP for finalizing the CTAS statem
Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement
> I just noticed the CREATE TABLE LIKE statement allows the definition > of new columns in the CREATE part. The difference > with this CTAS proposal is that TABLE LIKE appends the new columns at > the end of the schema instead of adding them > at the beginning like this proposal and Mysql do. This should be fine. The LIKE rather "extends from" the right table. Whereas the SELECT in CTAS rather extends the left schema definition. Given that "the extended part" is always appended, we could argue that the current CTAS behavior in the FLIP is acceptable. > If you want to rename a column in the create part, then that column > position must be in the same position as the query column. > I didn't like the Postgres approach because it does not let us add > columns that do not exist in the query schema. Flink offers similar functionality in INSERT INTO. INSERT INTO also allows syntax like: `INSERT INTO (b, c) SELECT * FROM t`. So given that our CTAS can be seen as a CREATE TABLE + INSERT INTO. I would adopt Jeyhun comment in the FLIP. If users don't want to add additional schema parts, the same column reordering should be available as if one would write a INSERT INTO. Regards, Timo On 12.06.24 04:30, Yanquan Lv wrote: Hi Sergio, thanks for driving it, +1 for this. I have some comments: 1. If we have a source table with primary keys and partition keys defined, what is the default behavior if PARTITIONED and DISTRIBUTED not specified in the CTAS statement, It should not be inherited by default？ 2. I suggest providing a complete syntax that includes table_properties like FLIP-218. Sergio Pena 于2024年6月12日周三 03:54写道： I just noticed the CREATE TABLE LIKE statement allows the definition of new columns in the CREATE part. The difference with this CTAS proposal is that TABLE LIKE appends the new columns at the end of the schema instead of adding them at the beginning like this proposal and Mysql do. create table t1(id int, name string); create table s1(a int, b string) like t1; describe s1; +-+---+--++ | Column Name | Data Type | Nullable | Extras | +-+---+--++ | id | INT | NULL || | name| STRING| NULL || | a | INT | NULL || | b | STRING| NULL || +-+---+--++ The CREATE TABLE LIKE also does not let the definition of existing columns in the CREATE part. The statement fails that the column already exists. create table t1(id int, name string); create table s1(id double) like t1; A column named 'id' already exists in the base table. What do you guys think of making it similar to the CREATE TABLE LIKE? Seems the best approach in order to be compatible with it. - Sergio On Tue, Jun 11, 2024 at 2:10 PM Sergio Pena wrote: Thanks Timo for answering Jeyhun questions. To add info more about your questions Jeyhun. This proposal is not handling NULL/NOT_NULL types. I noticed that the current CTAS impl. (as Timo said) adds this constraint as part of the resulting schema. And when defining a primary key in the CREATE part, if the resulting schema does not have a NOT NULL in the column then the CTAS will fail. This is similar to the CREATE TABLE LIKE which expects the LIKE table to have a NOT NULL column if the user defines a primary key in the CREATE part. In some cases, redefining the column types might be redundant, especially when users dont change the column type. A user just wants to change the column name from the SELECT clause. Should we also support this scenario, similar to postgres? I looked into Postgres too. Postgres matches the columns based on the order defined in the create and select part. If you want to rename a column in the create part, then that column position must be in the same position as the query column. I didn't like the Postgres approach because it does not let us add columns that do not exist in the query schema. i.e. query has schema (a int, b string), now the `a` column is renamed to `id` because both are in the same position 0 `create table s1(id int) as select a, b from t1`; results in: [id int, b string] I think, if users want to rename then they can use a different alias in the select part. They could also do explicit casting for changing the data types, which now makes it redundant (as you said) to allow redefining the query columns again. But perhaps there are cases where explicit casting does not work and just defining the column would? i.e. making a nullable type to not null? I couldn't make `cast(c1 as int not null)` to work for instance, but it may work in the create part? Could you also mention the casting rules in the FLIP for this case? I mentioned they're the same as insert/select when doing implicit casting. I will search for more info about the insert/select and add the casting rules in the flip.. - Sergio On T
Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
Congratulations Rui Best Regards Peter Huang On Tue, Jun 11, 2024 at 10:31 PM 王刚 wrote: > Congratulations Rui > > Best Regards, > Gang Wang > > Jacky Lau 于2024年6月11日周二 13:04写道： > > > Congratulations Rui, well deserved! > > > > Regards, > > Jacky Lau > > > > Jeyhun Karimov 于2024年6月11日 周二03:49写道： > > > > > Congratulations Rui, well deserved! > > > > > > Regards, > > > Jeyhun > > > > > > On Mon, Jun 10, 2024, 10:21 Ahmed Hamdy wrote: > > > > > > > Congratulations Rui! > > > > Best Regards > > > > Ahmed Hamdy > > > > > > > > > > > > On Mon, 10 Jun 2024 at 09:10, David Radley > > > > wrote: > > > > > > > > > Congratulations, Rui! > > > > > > > > > > From: Sergey Nuyanzin > > > > > Date: Sunday, 9 June 2024 at 20:33 > > > > > To: dev@flink.apache.org > > > > > Subject: [EXTERNAL] Re: [ANNOUNCE] New Apache Flink PMC Member - > Fan > > > Rui > > > > > Congratulations, Rui! > > > > > > > > > > On Fri, Jun 7, 2024 at 5:36 AM Xia Sun > wrote: > > > > > > > > > > > Congratulations, Rui! > > > > > > > > > > > > Best, > > > > > > Xia > > > > > > > > > > > > Paul Lam 于2024年6月6日周四 11:59写道： > > > > > > > > > > > > > Congrats, Rui! > > > > > > > > > > > > > > Best, > > > > > > > Paul Lam > > > > > > > > > > > > > > > 2024年6月6日 11:02，Junrui Lee 写道： > > > > > > > > > > > > > > > > Congratulations, Rui. > > > > > > > > > > > > > > > > Best, > > > > > > > > Junrui > > > > > > > > > > > > > > > > Hang Ruan 于2024年6月6日周四 10:35写道： > > > > > > > > > > > > > > > >> Congratulations, Rui! > > > > > > > >> > > > > > > > >> Best, > > > > > > > >> Hang > > > > > > > >> > > > > > > > >> Samrat Deb 于2024年6月6日周四 10:28写道： > > > > > > > >> > > > > > > > >>> Congratulations Rui > > > > > > > >>> > > > > > > > >>> Bests, > > > > > > > >>> Samrat > > > > > > > >>> > > > > > > > >>> On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan < > > > tanyuxinw...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > >>> > > > > > > > Congratulations, Rui! > > > > > > > > > > > > > > Best, > > > > > > > Yuxin > > > > > > > > > > > > > > > > > > > > > Xuannan Su 于2024年6月6日周四 09:58写道： > > > > > > > > > > > > > > > Congratulations! > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Xuannan > > > > > > > > > > > > > > > > On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu < > > > > master...@gmail.com > > > > > > > > > > > > > >>> wrote: > > > > > > > >> > > > > > > > >> Congratulations, Rui ! > > > > > > > >> > > > > > > > >> On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee < > > > > > > lincoln.8...@gmail.com > > > > > > > >>> > > > > > > > > wrote: > > > > > > > >> > > > > > > > >>> Congratulations, Rui! > > > > > > > >>> > > > > > > > >>> Best, > > > > > > > >>> Lincoln Lee > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> Lijie Wang 于2024年6月6日周四 > > > 09:11写道： > > > > > > > >>> > > > > > > > Congratulations, Rui! > > > > > > > > > > > > > > Best, > > > > > > > Lijie > > > > > > > > > > > > > > Rodrigo Meneses 于2024年6月5日周三 > > > 21:35写道： > > > > > > > > > > > > > > > All the best > > > > > > > > > > > > > > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng < > > > > > > > xiangyu...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > >> Congratulations, Rui! > > > > > > > >> > > > > > > > >> Regards, > > > > > > > >> Xiangyu Feng > > > > > > > >> > > > > > > > >> Feng Jin 于2024年6月5日周三 > > 20:42写道： > > > > > > > >> > > > > > > > >>> Congratulations, Rui! > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> Best, > > > > > > > >>> Feng Jin > > > > > > > >>> > > > > > > > >>> On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei < > > > > > > > fredia...@gmail.com > > > > > > > >> > > > > > > > wrote: > > > > > > > >>> > > > > > > > Congratulations, Rui! > > > > > > > > > > > > > > Best, > > > > > > > Yanfei > > > > > > > > > > > > > > Luke Chen 于2024年6月5日周三 > 20:08写道： > > > > > > > > > > > > > > > > Congrats, Rui! > > > > > > > > > > > > > > > > Luke > > > > > > > > > > > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun < > > > > > > > >>> jiabao...@apache.org> > > > > > > > >>> wrote: > > > > > > > > > > > > > > > >> Congrats, Rui. Well-deserved! > > > > > > > >> > > > > > > > >> Best, > > > > > > > >> Jiabao > > > > > > > >> > > > > > > > >> Zhanghao Chen > > > > > > > >>> 于2024年6月5日周三 > > > > > > > 19:29写道： > > > > > > > >> > > > > > > > >>> Congrats, Rui! > > > > > > > >>> > > > > > > > >>
Re: Flink Kubernetes Operator 1.9.0 release planning
+1 Thanks Gyula for driving this release! Best Regards Peter Huang On Tue, Jun 11, 2024 at 12:28 PM Márton Balassi wrote: > +1 for cutting the release and Gyula as the release manager. > > On Tue, Jun 11, 2024 at 10:41 AM David Radley > wrote: > > > I agree – thanks for driving this Gyula. > > > > From: Rui Fan <1996fan...@gmail.com> > > Date: Tuesday, 11 June 2024 at 02:52 > > To: dev@flink.apache.org > > Cc: Mate Czagany > > Subject: [EXTERNAL] Re: Flink Kubernetes Operator 1.9.0 release planning > > Thanks Gyula for driving this release! > > > > > I suggest we cut the release branch this week after merging current > > > outstanding smaller PRs. > > > > It makes sense to me. > > > > Best, > > Rui > > > > On Mon, Jun 10, 2024 at 3:05 PM Gyula Fóra wrote: > > > > > Hi all! > > > > > > I want to kick off the discussion / release process for the Flink > > > Kubernetes Operator 1.9.0 version. > > > > > > The last, 1.8.0, version was released in March and since then we have > > had a > > > number of important fixes. Furthermore there are some bigger pieces of > > > outstanding work in the form of open PRs such as the Savepoint CRD work > > > which should only be merged to 1.10.0 to gain more exposure/stability. > > > > > > I suggest we cut the release branch this week after merging current > > > outstanding smaller PRs. > > > > > > I volunteer as the release manager but if someone else would like to do > > it, > > > I would also be happy to assist. > > > > > > Please let me know what you think. > > > > > > Cheers, > > > Gyula > > > > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > > >
[VOTE] FLIP-464: Merge "flink run" and "flink run-application"
Hello devs, I would like to start a vote about FLIP-464 [1]. The FLIP is about to merge back the "flink run-application" functionality to "flink run", so the latter will be capable to deploy jobs in all deployment modes. More details in the FLIP. Discussion thread [2]. The vote will be open for at least 72 hours (until 2024 March 23 14:03 UTC) unless there are any objections or insufficient votes. Thanks,Ferenc [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 [2] https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
Hi, Thank you for driving this Ferenc, +1 (non-binding) Regards, Mate Ferenc Csaky ezt írta (időpont: 2024. jún. 12., Sze, 17:23): > Hello devs, > > I would like to start a vote about FLIP-464 [1]. The FLIP is about to > merge back the > "flink run-application" functionality to "flink run", so the latter will > be capable to deploy jobs in > all deployment modes. More details in the FLIP. Discussion thread [2]. > > The vote will be open for at least 72 hours (until 2024 March 23 14:03 > UTC) unless there > are any objections or insufficient votes. > > Thanks,Ferenc > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > [2] https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
Sounds reasonable, +1 Cheers, Matyas On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany wrote: > Hi, > > Thank you for driving this Ferenc, > +1 (non-binding) > > Regards, > Mate > > Ferenc Csaky ezt írta (időpont: 2024. jún. > 12., Sze, 17:23): > > > Hello devs, > > > > I would like to start a vote about FLIP-464 [1]. The FLIP is about to > > merge back the > > "flink run-application" functionality to "flink run", so the latter will > > be capable to deploy jobs in > > all deployment modes. More details in the FLIP. Discussion thread [2]. > > > > The vote will be open for at least 72 hours (until 2024 March 23 14:03 > > UTC) unless there > > are any objections or insufficient votes. > > > > Thanks,Ferenc > > > > [1] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > > [2] https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g >
Re: Flink Kubernetes Operator 1.9.0 release planning
+1 to release 1.9.0 > On 12. Jun 2024, at 17:10, Peter Huang wrote: > > +1 Thanks Gyula for driving this release! > > > Best Regards > Peter Huang > > On Tue, Jun 11, 2024 at 12:28 PM Márton Balassi > wrote: > >> +1 for cutting the release and Gyula as the release manager. >> >> On Tue, Jun 11, 2024 at 10:41 AM David Radley >> wrote: >> >>> I agree – thanks for driving this Gyula. >>> >>> From: Rui Fan <1996fan...@gmail.com> >>> Date: Tuesday, 11 June 2024 at 02:52 >>> To: dev@flink.apache.org >>> Cc: Mate Czagany >>> Subject: [EXTERNAL] Re: Flink Kubernetes Operator 1.9.0 release planning >>> Thanks Gyula for driving this release! >>> I suggest we cut the release branch this week after merging current outstanding smaller PRs. >>> >>> It makes sense to me. >>> >>> Best, >>> Rui >>> >>> On Mon, Jun 10, 2024 at 3:05 PM Gyula Fóra wrote: >>> Hi all! I want to kick off the discussion / release process for the Flink Kubernetes Operator 1.9.0 version. The last, 1.8.0, version was released in March and since then we have >>> had a number of important fixes. Furthermore there are some bigger pieces of outstanding work in the form of open PRs such as the Savepoint CRD work which should only be merged to 1.10.0 to gain more exposure/stability. I suggest we cut the release branch this week after merging current outstanding smaller PRs. I volunteer as the release manager but if someone else would like to do >>> it, I would also be happy to assist. Please let me know what you think. Cheers, Gyula >>> >>> Unless otherwise stated above: >>> >>> IBM United Kingdom Limited >>> Registered in England and Wales with number 741598 >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >>> >>
Re: [DISCUSS] Proposing an LTS Release for the 1.x Line
Hi folks! 1) Big +1 on being strict with limiting the LTS version to patches and no feature backports. I think that it would open up a big can of worms. What does it even mean to backport a feature to a LTS version? My understanding is that we want to designate a single 1.x version as LTS. But if we backport a feature, we would technically need to bump the minor version and then we are basically back at maintaining features for 1.x. Let me know if I’m misunderstanding something. 2) It also seems fine to me to stick to 1.x and 2.x concretely in the FLIP since it’ll be our first LTS version and we don’t anticipate (as of today) many overlapping LTS versions. It actually makes it clearer to me what we’re talking about. 3) 2 years seems like a good starting point to me. If anything, I’m wondering whether it should be longer given that there are many users on old versions even today. It’s probably best to stick to this number and expand if needed down the line (as opposed to starting with a longer duration and then running into problems). Cheers, Ufuk > On 11. Jun 2024, at 15:44, Alexander Fedulov > wrote: > > Hi Matthias, > > I think we can include this generic semantic into the writeup of the LTS > definition for the Flink website (last item in the Migration Plan). > Talking about 1.x and 2.x feels more natural than about N.x and N+1.x - I'd > prefer not to overcomplicate things here. > Should the gap before the next major release match this one (eight years), > it would be appropriate to reconsider the project stance and vote anew if > required. > > Best, > Alex > > On Mon, 27 May 2024 at 09:02, Matthias Pohl wrote: > >> Hi Alex, >> thanks for creating the FLIP. One question: Is it intentionally done that >> the FLIP only covers the 1.x LTS version? Why don't we make the FLIP >> generic to also apply to other major version upgrades? >> >> Best, >> Matthias >> >> On Sat, May 25, 2024 at 4:55 PM Xintong Song >> wrote: >> I think our starting point should be "We don't backport features, >> unless discussed and agreed on the Dev mailing list". >>> >>> >>> This makes perfect sense. In general, I think we want to encourage users >> to >>> upgrade in order to use the new features. Alternatively, users can also >>> choose to maintain their own custom patches based on the LTS version. I >>> personally don't see why backporting new features to old releases is >>> necessary. In case of exceptions, a mailing list discussion is always a >>> good direction to go. >>> >>> >>> Best, >>> >>> Xintong >>> >>> >>> >>> On Fri, May 24, 2024 at 9:35 PM David Radley >>> wrote: >>> Hi Martjin and Alex, I agree with your summaries, it will be interesting to see what >> requests there might be for back ports. Kind regards, David. From: Alexander Fedulov Date: Friday, 24 May 2024 at 14:07 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] Proposing an LTS Release for the 1.x >>> Line @David > I agree with Martijn that we only put features into version 2. Back porting to v1 should not be business as usual for features, only for security and stability changes. Yep, this choice is explicitly reflected in the FLIP [1] @Martijn > I think our starting point should be "We don't backport features, >>> unless discussed and agreed on the Dev mailing list". I agree - the baseline is that we do not do that. Only if a very >>> compelling argument is made and the community reaches consensus, exceptions could potentially be made, but we should try to avoid them. [1] https://cwiki.apache.org/confluence/x/BApeEg Best, Alex On Fri, 24 May 2024 at 14:38, Martijn Visser >> wrote: > Hi David, > >> If there is a maintainer willing to merge backported features to >> v1, >>> as > it is important to some part of the community, this should be >> allowed, >>> as > different parts of the community have different priorities and >>> timelines, > > I don't think this is a good idea. Backporting a feature can cause >>> issues > in other components that might be outside the span of expertise of >> the > maintainer that backported said feature, causing the overall >> stability >>> to > be degraded. I think our starting point should be "We don't backport > features, unless discussed and agreed on the Dev mailing list". That still > opens up the ability to backport features but makes it clear where >> the bar > lies. > > Best regards, > > Martijn > > On Fri, May 24, 2024 at 11:21 AM David Radley < >> david_rad...@uk.ibm.com > wrote: > >> Hi, >> I agree with Martijn that we only put features into version 2. Back >> porting to v1 should not be business as usual for features, only >> for >> security and stability c
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
+1 (binding) On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás wrote: > Sounds reasonable, > +1 > > Cheers, > Matyas > > > On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany wrote: > > > Hi, > > > > Thank you for driving this Ferenc, > > +1 (non-binding) > > > > Regards, > > Mate > > > > Ferenc Csaky ezt írta (időpont: 2024. jún. > > 12., Sze, 17:23): > > > > > Hello devs, > > > > > > I would like to start a vote about FLIP-464 [1]. The FLIP is about to > > > merge back the > > > "flink run-application" functionality to "flink run", so the latter > will > > > be capable to deploy jobs in > > > all deployment modes. More details in the FLIP. Discussion thread [2]. > > > > > > The vote will be open for at least 72 hours (until 2024 March 23 14:03 > > > UTC) unless there > > > are any objections or insufficient votes. > > > > > > Thanks,Ferenc > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > > > [2] https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g > > >
[jira] [Created] (FLINK-35585) Add documentation for distribution
Jim Hughes created FLINK-35585: -- Summary: Add documentation for distribution Key: FLINK-35585 URL: https://issues.apache.org/jira/browse/FLINK-35585 Project: Flink Issue Type: Sub-task Reporter: Jim Hughes Add documentation for ALTER TABLE, CREATE TABLE, and the sink abilities. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
+1(binding) Best, Xintong On Thu, Jun 13, 2024 at 5:15 AM Márton Balassi wrote: > +1 (binding) > > On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás > wrote: > > > Sounds reasonable, > > +1 > > > > Cheers, > > Matyas > > > > > > On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany wrote: > > > > > Hi, > > > > > > Thank you for driving this Ferenc, > > > +1 (non-binding) > > > > > > Regards, > > > Mate > > > > > > Ferenc Csaky ezt írta (időpont: 2024. > jún. > > > 12., Sze, 17:23): > > > > > > > Hello devs, > > > > > > > > I would like to start a vote about FLIP-464 [1]. The FLIP is about to > > > > merge back the > > > > "flink run-application" functionality to "flink run", so the latter > > will > > > > be capable to deploy jobs in > > > > all deployment modes. More details in the FLIP. Discussion thread > [2]. > > > > > > > > The vote will be open for at least 72 hours (until 2024 March 23 > 14:03 > > > > UTC) unless there > > > > are any objections or insufficient votes. > > > > > > > > Thanks,Ferenc > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > > > > [2] https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g > > > > > >
[jira] [Created] (FLINK-35586) Detected conflict when using Paimon as pipeline sink with parallelism > 1
LvYanquan created FLINK-35586:
-
Summary: Detected conflict when using Paimon as pipeline sink with
parallelism > 1
Key: FLINK-35586
URL: https://issues.apache.org/jira/browse/FLINK-35586
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: LvYanquan
Fix For: cdc-3.2.0
When submit FlinkCDC pipeline job using yaml like:
{code:java}
source:
type: mysql
name: MySQL Source
hostname: 127.0.0.1
port: 3306
username: root
password: 123456
tables: inventory.t1
sink:
type: paimon
name: Paimon Sink
catalog.properties.metastore: filesystem
catalog.properties.warehouse: /mypath
pipeline:
name: MySQL to Paimon Pipeline
parallelism: 2 {code}
I met the following error message:
{code:java}
Caused by: java.lang.RuntimeException: LSM conflicts detected! Give up
committing. Conflict files are:, bucket 0, level 5, file
data-6bcac56a-2df2-4c85-97f2-2db91f6d8099-0.orc, bucket 0, level 5, file
data-351fd27d-4a65-4354-9ce9-c153ba715569-0.orc {code}
And this will cause the task to constantly restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement
Hi, Sergio. Thanks for driving the FLIP. Given we also has REPLACE TABLE AS Statement[1] and it's almost same with CREATE TABLE AS Statement, would you mind also supporting schema definition for REPLACE TABLE AS Statement in this FLIP? It'll be a great to align REPLACE TABLE AS Statement to CREATE TABLE AS Statement [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement Best regards, Yuxia - 原始邮件 - 发件人: "Timo Walther" 收件人: "dev" 发送时间: 星期三, 2024年 6 月 12日 下午 10:19:14 主题: Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement > I just noticed the CREATE TABLE LIKE statement allows the definition > of new columns in the CREATE part. The difference > with this CTAS proposal is that TABLE LIKE appends the new columns at > the end of the schema instead of adding them > at the beginning like this proposal and Mysql do. This should be fine. The LIKE rather "extends from" the right table. Whereas the SELECT in CTAS rather extends the left schema definition. Given that "the extended part" is always appended, we could argue that the current CTAS behavior in the FLIP is acceptable. > If you want to rename a column in the create part, then that column > position must be in the same position as the query column. > I didn't like the Postgres approach because it does not let us add > columns that do not exist in the query schema. Flink offers similar functionality in INSERT INTO. INSERT INTO also allows syntax like: `INSERT INTO (b, c) SELECT * FROM t`. So given that our CTAS can be seen as a CREATE TABLE + INSERT INTO. I would adopt Jeyhun comment in the FLIP. If users don't want to add additional schema parts, the same column reordering should be available as if one would write a INSERT INTO. Regards, Timo On 12.06.24 04:30, Yanquan Lv wrote: > Hi Sergio, thanks for driving it, +1 for this. > > I have some comments: > 1. If we have a source table with primary keys and partition keys defined, > what is the default behavior if PARTITIONED and DISTRIBUTED not specified > in the CTAS statement, It should not be inherited by default？ > 2. I suggest providing a complete syntax that includes table_properties > like FLIP-218. > > > Sergio Pena 于2024年6月12日周三 03:54写道： > >> I just noticed the CREATE TABLE LIKE statement allows the definition of new >> columns in the CREATE part. The difference >> with this CTAS proposal is that TABLE LIKE appends the new columns at the >> end of the schema instead of adding them >> at the beginning like this proposal and Mysql do. >> >>> create table t1(id int, name string); create table s1(a int, b string) like t1; describe s1; >> >> +-+---+--++ >>> | Column Name | Data Type | Nullable | Extras | >>> +-+---+--++ >>> | id | INT | NULL || >>> | name| STRING| NULL || >>> | a | INT | NULL || >>> | b | STRING| NULL || >>> +-+---+--++ >> >> >> >> The CREATE TABLE LIKE also does not let the definition of existing columns >> in the CREATE part. The statement fails >> that the column already exists. >> >>> create table t1(id int, name string); >> >>> create table s1(id double) like t1; >>> A column named 'id' already exists in the base table. >>> >> >> What do you guys think of making it similar to the CREATE TABLE LIKE? Seems >> the best approach in order to >> be compatible with it. >> >> - Sergio >> >> On Tue, Jun 11, 2024 at 2:10 PM Sergio Pena wrote: >> >>> Thanks Timo for answering Jeyhun questions. >>> >>> To add info more about your questions Jeyhun. This proposal is not >>> handling NULL/NOT_NULL types. I noticed that >>> the current CTAS impl. (as Timo said) adds this constraint as part of the >>> resulting schema. And when defining >>> a primary key in the CREATE part, if the resulting schema does not have a >>> NOT NULL in the column then the CTAS >>> will fail. This is similar to the CREATE TABLE LIKE which expects the >> LIKE >>> table to have a NOT NULL column if >>> the user defines a primary key in the CREATE part. >>> In some cases, redefining the column types might be redundant, >> especially when users dont change the column type. A user just wants to change the column name from the SELECT clause. Should we also support this >> scenario, similar to postgres? >>> >>> I looked into Postgres too. Postgres matches the columns based on the >>> order defined in the create and select part. >>> If you want to rename a column in the create part, then that column >>> position must be in the same position as the query column. >>> I didn't like the Postgres approach because it does not let us add >> columns >>> that do not exist in the query schema. >>> >>> i.e. query has schema (a int, b string), now the `a` column is renamed to >>> `id` becaus
Re: Flink Kubernetes Operator 1.9.0 release planning
+1 for the release plan and release manager candidate, thanks Gyula. Best, Leonard > 2024年6月12日 下午11:10，Peter Huang 写道： > > +1 Thanks Gyula for driving this release! > > > Best Regards > Peter Huang > > On Tue, Jun 11, 2024 at 12:28 PM Márton Balassi > wrote: > >> +1 for cutting the release and Gyula as the release manager. >> >> On Tue, Jun 11, 2024 at 10:41 AM David Radley >> wrote: >> >>> I agree – thanks for driving this Gyula. >>> >>> From: Rui Fan <1996fan...@gmail.com> >>> Date: Tuesday, 11 June 2024 at 02:52 >>> To: dev@flink.apache.org >>> Cc: Mate Czagany >>> Subject: [EXTERNAL] Re: Flink Kubernetes Operator 1.9.0 release planning >>> Thanks Gyula for driving this release! >>> I suggest we cut the release branch this week after merging current outstanding smaller PRs. >>> >>> It makes sense to me. >>> >>> Best, >>> Rui >>> >>> On Mon, Jun 10, 2024 at 3:05 PM Gyula Fóra wrote: >>> Hi all! I want to kick off the discussion / release process for the Flink Kubernetes Operator 1.9.0 version. The last, 1.8.0, version was released in March and since then we have >>> had a number of important fixes. Furthermore there are some bigger pieces of outstanding work in the form of open PRs such as the Savepoint CRD work which should only be merged to 1.10.0 to gain more exposure/stability. I suggest we cut the release branch this week after merging current outstanding smaller PRs. I volunteer as the release manager but if someone else would like to do >>> it, I would also be happy to assist. Please let me know what you think. Cheers, Gyula >>> >>> Unless otherwise stated above: >>> >>> IBM United Kingdom Limited >>> Registered in England and Wales with number 741598 >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >>> >>
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
Thanks for driving this! +1(binding) Best regards, Weijie Xintong Song 于2024年6月13日周四 09:04写道： > +1(binding) > > Best, > > Xintong > > > > On Thu, Jun 13, 2024 at 5:15 AM Márton Balassi > wrote: > > > +1 (binding) > > > > On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás > > wrote: > > > > > Sounds reasonable, > > > +1 > > > > > > Cheers, > > > Matyas > > > > > > > > > On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany > wrote: > > > > > > > Hi, > > > > > > > > Thank you for driving this Ferenc, > > > > +1 (non-binding) > > > > > > > > Regards, > > > > Mate > > > > > > > > Ferenc Csaky ezt írta (időpont: 2024. > > jún. > > > > 12., Sze, 17:23): > > > > > > > > > Hello devs, > > > > > > > > > > I would like to start a vote about FLIP-464 [1]. The FLIP is about > to > > > > > merge back the > > > > > "flink run-application" functionality to "flink run", so the latter > > > will > > > > > be capable to deploy jobs in > > > > > all deployment modes. More details in the FLIP. Discussion thread > > [2]. > > > > > > > > > > The vote will be open for at least 72 hours (until 2024 March 23 > > 14:03 > > > > > UTC) unless there > > > > > are any objections or insufficient votes. > > > > > > > > > > Thanks,Ferenc > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > > > > > [2] > https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g > > > > > > > > > >
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
Thanks for driving this. +1 (non-binding) Best, Biao Geng weijie guo 于2024年6月13日周四 09:48写道： > Thanks for driving this! > > +1(binding) > > Best regards, > > Weijie > > > Xintong Song 于2024年6月13日周四 09:04写道： > > > +1(binding) > > > > Best, > > > > Xintong > > > > > > > > On Thu, Jun 13, 2024 at 5:15 AM Márton Balassi > > > wrote: > > > > > +1 (binding) > > > > > > On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás > > > > wrote: > > > > > > > Sounds reasonable, > > > > +1 > > > > > > > > Cheers, > > > > Matyas > > > > > > > > > > > > On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany > > wrote: > > > > > > > > > Hi, > > > > > > > > > > Thank you for driving this Ferenc, > > > > > +1 (non-binding) > > > > > > > > > > Regards, > > > > > Mate > > > > > > > > > > Ferenc Csaky ezt írta (időpont: 2024. > > > jún. > > > > > 12., Sze, 17:23): > > > > > > > > > > > Hello devs, > > > > > > > > > > > > I would like to start a vote about FLIP-464 [1]. The FLIP is > about > > to > > > > > > merge back the > > > > > > "flink run-application" functionality to "flink run", so the > latter > > > > will > > > > > > be capable to deploy jobs in > > > > > > all deployment modes. More details in the FLIP. Discussion thread > > > [2]. > > > > > > > > > > > > The vote will be open for at least 72 hours (until 2024 March 23 > > > 14:03 > > > > > > UTC) unless there > > > > > > are any objections or insufficient votes. > > > > > > > > > > > > Thanks,Ferenc > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > > > > > > [2] > > https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g > > > > > > > > > > > > > > >
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
+1 (non-binding) Best, Junrui Biao Geng 于2024年6月13日周四 09:54写道： > Thanks for driving this. > +1 (non-binding) > > Best, > Biao Geng > > > weijie guo 于2024年6月13日周四 09:48写道： > > > Thanks for driving this! > > > > +1(binding) > > > > Best regards, > > > > Weijie > > > > > > Xintong Song 于2024年6月13日周四 09:04写道： > > > > > +1(binding) > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Thu, Jun 13, 2024 at 5:15 AM Márton Balassi < > balassi.mar...@gmail.com > > > > > > wrote: > > > > > > > +1 (binding) > > > > > > > > On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás < > matyas.orh...@gmail.com > > > > > > > wrote: > > > > > > > > > Sounds reasonable, > > > > > +1 > > > > > > > > > > Cheers, > > > > > Matyas > > > > > > > > > > > > > > > On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Thank you for driving this Ferenc, > > > > > > +1 (non-binding) > > > > > > > > > > > > Regards, > > > > > > Mate > > > > > > > > > > > > Ferenc Csaky ezt írta (időpont: > 2024. > > > > jún. > > > > > > 12., Sze, 17:23): > > > > > > > > > > > > > Hello devs, > > > > > > > > > > > > > > I would like to start a vote about FLIP-464 [1]. The FLIP is > > about > > > to > > > > > > > merge back the > > > > > > > "flink run-application" functionality to "flink run", so the > > latter > > > > > will > > > > > > > be capable to deploy jobs in > > > > > > > all deployment modes. More details in the FLIP. Discussion > thread > > > > [2]. > > > > > > > > > > > > > > The vote will be open for at least 72 hours (until 2024 March > 23 > > > > 14:03 > > > > > > > UTC) unless there > > > > > > > are any objections or insufficient votes. > > > > > > > > > > > > > > Thanks,Ferenc > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > > > > > > > [2] > > > https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
+1 (binding) Thanks for driving this! Best, Zakelly On Thu, Jun 13, 2024 at 10:05 AM Junrui Lee wrote: > +1 (non-binding) > > Best, > Junrui > > Biao Geng 于2024年6月13日周四 09:54写道： > > > Thanks for driving this. > > +1 (non-binding) > > > > Best, > > Biao Geng > > > > > > weijie guo 于2024年6月13日周四 09:48写道： > > > > > Thanks for driving this! > > > > > > +1(binding) > > > > > > Best regards, > > > > > > Weijie > > > > > > > > > Xintong Song 于2024年6月13日周四 09:04写道： > > > > > > > +1(binding) > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Thu, Jun 13, 2024 at 5:15 AM Márton Balassi < > > balassi.mar...@gmail.com > > > > > > > > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás < > > matyas.orh...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Sounds reasonable, > > > > > > +1 > > > > > > > > > > > > Cheers, > > > > > > Matyas > > > > > > > > > > > > > > > > > > On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Thank you for driving this Ferenc, > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > Regards, > > > > > > > Mate > > > > > > > > > > > > > > Ferenc Csaky ezt írta (időpont: > > 2024. > > > > > jún. > > > > > > > 12., Sze, 17:23): > > > > > > > > > > > > > > > Hello devs, > > > > > > > > > > > > > > > > I would like to start a vote about FLIP-464 [1]. The FLIP is > > > about > > > > to > > > > > > > > merge back the > > > > > > > > "flink run-application" functionality to "flink run", so the > > > latter > > > > > > will > > > > > > > > be capable to deploy jobs in > > > > > > > > all deployment modes. More details in the FLIP. Discussion > > thread > > > > > [2]. > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours (until 2024 March > > 23 > > > > > 14:03 > > > > > > > > UTC) unless there > > > > > > > > are any objections or insufficient votes. > > > > > > > > > > > > > > > > Thanks,Ferenc > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > > > > > > > > [2] > > > > https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"
+1 (non-binding) Thanks for driving it! Best, Yuepeng Pan 在 2024-06-13 10:47:11，"Zakelly Lan" 写道： >+1 (binding) > >Thanks for driving this! > > >Best, >Zakelly > >On Thu, Jun 13, 2024 at 10:05 AM Junrui Lee wrote: > >> +1 (non-binding) >> >> Best, >> Junrui >> >> Biao Geng 于2024年6月13日周四 09:54写道： >> >> > Thanks for driving this. >> > +1 (non-binding) >> > >> > Best, >> > Biao Geng >> > >> > >> > weijie guo 于2024年6月13日周四 09:48写道： >> > >> > > Thanks for driving this! >> > > >> > > +1(binding) >> > > >> > > Best regards, >> > > >> > > Weijie >> > > >> > > >> > > Xintong Song 于2024年6月13日周四 09:04写道： >> > > >> > > > +1(binding) >> > > > >> > > > Best, >> > > > >> > > > Xintong >> > > > >> > > > >> > > > >> > > > On Thu, Jun 13, 2024 at 5:15 AM Márton Balassi < >> > balassi.mar...@gmail.com >> > > > >> > > > wrote: >> > > > >> > > > > +1 (binding) >> > > > > >> > > > > On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás < >> > matyas.orh...@gmail.com >> > > > >> > > > > wrote: >> > > > > >> > > > > > Sounds reasonable, >> > > > > > +1 >> > > > > > >> > > > > > Cheers, >> > > > > > Matyas >> > > > > > >> > > > > > >> > > > > > On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany > > >> > > > wrote: >> > > > > > >> > > > > > > Hi, >> > > > > > > >> > > > > > > Thank you for driving this Ferenc, >> > > > > > > +1 (non-binding) >> > > > > > > >> > > > > > > Regards, >> > > > > > > Mate >> > > > > > > >> > > > > > > Ferenc Csaky ezt írta (időpont: >> > 2024. >> > > > > jún. >> > > > > > > 12., Sze, 17:23): >> > > > > > > >> > > > > > > > Hello devs, >> > > > > > > > >> > > > > > > > I would like to start a vote about FLIP-464 [1]. The FLIP is >> > > about >> > > > to >> > > > > > > > merge back the >> > > > > > > > "flink run-application" functionality to "flink run", so the >> > > latter >> > > > > > will >> > > > > > > > be capable to deploy jobs in >> > > > > > > > all deployment modes. More details in the FLIP. Discussion >> > thread >> > > > > [2]. >> > > > > > > > >> > > > > > > > The vote will be open for at least 72 hours (until 2024 March >> > 23 >> > > > > 14:03 >> > > > > > > > UTC) unless there >> > > > > > > > are any objections or insufficient votes. >> > > > > > > > >> > > > > > > > Thanks,Ferenc >> > > > > > > > >> > > > > > > > [1] >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 >> > > > > > > > [2] >> > > > https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >>
[jira] [Created] (FLINK-35587) Job fails with "The read buffer is null in credit-based input channel" on TPC-DS 10TB benchmark
Junrui Li created FLINK-35587: - Summary: Job fails with "The read buffer is null in credit-based input channel" on TPC-DS 10TB benchmark Key: FLINK-35587 URL: https://issues.apache.org/jira/browse/FLINK-35587 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.20.0 Reporter: Junrui Li While running TPC-DS 10TB benchmark on the latest master branch locally, I've encountered a failure in certain queries, specifically query 75, resulting in the error "The read buffer is null in credit-based input channel". Using a binary search approach, I identified the offending commit as FLINK-33668. After reverting FLINK-33668 and subsequent commits, the issue disappears. Re-applying FLINK-33668 to the branch re-introduces the error. -- This message was sent by Atlassian Jira (v8.20.10#820010)
