date:20240415

Re: [DISCUSS] Connector releases for Flink 1.19

2024-04-15 Thread Sergey Nuyanzin

Thanks for volunteering Muhammet!
And thanks Danny for starting the activity.

If I'm not mistaking (please correct me if I'm wrong)

this request is not about version update it is about new releases for
connectors
btw for jdbc connector support of 1.19 and 1.20-SNAPSHOT is already done

I would volunteer for Opensearch connector since currently I'm working on
support of Opensearch v2
and I think it would make sense to have a release after it is done

On Tue, Apr 16, 2024 at 4:29 AM Muhammet Orazov
 wrote:

> Hello Danny,
>
> I would like to work on this task, thanks for initiating it!
>
> I could update the versions on JDBC and Pulsar connectors.
>
> Are you going to create an umbrella issue for it?
>
> Best,
> Muhammet
>
> On 2024-04-15 13:44, Danny Cranmer wrote:
> > Hello all,
> >
> > Flink 1.19 was released on 2024-03-18 [1] and the connectors have not
> > yet
> > caught up. I propose we start releasing the connectors with support for
> > Flink 1.19 as per the connector support guidelines [2].
> >
> > I will make a start on flink-connector-aws, then pickup others in the
> > coming days. Please respond to the thread if you are/want to work on a
> > particular connector to avoid duplicate work.
> >
> > Thanks,
> > Danny
> >
> > [1]
> >
> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Flinkcompatibility
> > [3] https://github.com/apache/flink-connector-aws
>


-- 
Best regards,
Sergey

Re:[ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread Feifan Wang

Congratulations Lincoln !


——

Best regards,

Feifan Wang




At 2024-04-12 15:59:00, "Jark Wu"  wrote:
>Hi everyone,
>
>On behalf of the PMC, I'm very happy to announce that Lincoln Lee has
>joined the Flink PMC!
>
>Lincoln has been an active member of the Apache Flink community for
>many years. He mainly works on Flink SQL component and has driven
>/pushed many FLIPs around SQL, including FLIP-282/373/415/435 in
>the recent versions. He has a great technical vision of Flink SQL and
>participated in plenty of discussions in the dev mailing list. Besides
>that,
>he is community-minded, such as being the release manager of 1.19,
>verifying releases, managing release syncs, writing the release
>announcement etc.
>
>Congratulations and welcome Lincoln!
>
>Best,
>Jark (on behalf of the Flink PMC)

Re:[ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-15 Thread Feifan Wang

Congratulations, Jing!


——

Best regards,

Feifan Wang




At 2024-04-12 16:02:01, "Jark Wu"  wrote:
>Hi everyone,
>
>On behalf of the PMC, I'm very happy to announce that Jing Ge has
>joined the Flink PMC!
>
>Jing has been contributing to Apache Flink for a long time. He continuously
>works on SQL, connectors, Source, and Sink APIs, test, and document
>modules while contributing lots of code and insightful discussions. He is
>one of the maintainers of Flink CI infra. He is also willing to help a lot
>in the
>community work, such as being the release manager for both 1.18 and 1.19,
>verifying releases, and answering questions on the mailing list. Besides
>that,
>he is continuously helping with the expansion of the Flink community and
>has
>given several talks about Flink at many conferences, such as Flink Forward
>2022 and 2023.
>
>Congratulations and welcome Jing!
>
>Best,
>Jark (on behalf of the Flink PMC)

Re:[ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Feifan Wang

Congratulations, Zakelly!——

Best regards,

Feifan Wang




At 2024-04-15 10:50:06, "Yuan Mei"  wrote:
>Hi everyone,
>
>On behalf of the PMC, I'm happy to let you know that Zakelly Lan has become
>a new Flink Committer!
>
>Zakelly has been continuously contributing to the Flink project since 2020,
>with a focus area on Checkpointing, State as well as frocksdb (the default
>on-disk state db).
>
>He leads several FLIPs to improve checkpoints and state APIs, including
>File Merging for Checkpoints and configuration/API reorganizations. He is
>also one of the main contributors to the recent efforts of "disaggregated
>state management for Flink 2.0" and drives the entire discussion in the
>mailing thread, demonstrating outstanding technical depth and breadth of
>knowledge.
>
>Beyond his technical contributions, Zakelly is passionate about helping the
>community in numerous ways. He spent quite some time setting up the Flink
>Speed Center and rebuilding the benchmark pipeline after the original one
>was out of lease. He helps build frocksdb and tests for the upcoming
>frocksdb release (bump rocksdb from 6.20.3->8.10).
>
>Please join me in congratulating Zakelly for becoming an Apache Flink
>committer!
>
>Best,
>Yuan (on behalf of the Flink PMC)

[jira] [Created] (FLINK-35119) UPDATE DataChangeEvent deserialized data is incorrect

2024-04-15 Thread Qishang Zhong (Jira)

Qishang Zhong created FLINK-35119:
-

 Summary: UPDATE DataChangeEvent deserialized data is incorrect
 Key: FLINK-35119
 URL: https://issues.apache.org/jira/browse/FLINK-35119
 Project: Flink
  Issue Type: Bug
  Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: Qishang Zhong


When 
DebeziumChangelogMode is upsert, the before of DataChangeEvent is null, 
deserialized data is incorrect.
 
Add test data  in 
org.apache.flink.cdc.runtime.serializer.event.DataChangeEventSerializerTest
 
{code:java}
DataChangeEvent.updateEvent(
TableId.tableId("namespace", "schema", "table"), null, after, meta) 
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Hang Ruan

Congratulations Zakelly!

Best,
Hang

Yuxin Tan  于2024年4月16日周二 11:04写道：

> Congratulations, Zakelly!
>
> Best,
> Yuxin
>
>
> Xuannan Su  于2024年4月16日周二 10:30写道：
>
> > Congratulations Zakelly!
> >
> > Best regards,
> > Xuannan
> >
> > On Mon, Apr 15, 2024 at 4:31 PM Jing Ge 
> > wrote:
> > >
> > > Congratulations Zakelly!
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Mon, Apr 15, 2024 at 4:26 PM Xia Sun  wrote:
> > >
> > > > Congratulations Zakelly!
> > > >
> > > >  Best,
> > > >  Xia
> > > >
> > > > Leonard Xu  于2024年4月15日周一 16:16写道：
> > > >
> > > > > Congratulations Zakelly!
> > > > >
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > > > 2024年4月15日 下午3:56，Samrat Deb  写道：
> > > > > >
> > > > > > Congratulations Zakelly!
> > > > >
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-15 Thread Hang Ruan

Congratulations, Jing!

Best,
Hang

Yuxin Tan  于2024年4月16日周二 11:07写道：

> Congratulations, Jing!
>
> Best,
> Yuxin
>
>
> Danny Cranmer  于2024年4月15日周一 20:26写道：
>
> > Congrats Jing!
> >
> > Best Regards,
> > Danny
> >
> > On Mon, Apr 15, 2024 at 11:51 AM Swapnal Varma 
> > wrote:
> >
> > > Congratulations, Jing!
> > >
> > > Best,
> > > Swapnal
> > >
> > > On Mon, 15 Apr 2024, 15:14 Jacky Lau,  wrote:
> > >
> > > > Congratulations, Jing!
> > > >
> > > > Best,
> > > > Jacky Lau
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-15 Thread Yuxin Tan

Congratulations, Jing!

Best,
Yuxin


Danny Cranmer  于2024年4月15日周一 20:26写道：

> Congrats Jing!
>
> Best Regards,
> Danny
>
> On Mon, Apr 15, 2024 at 11:51 AM Swapnal Varma 
> wrote:
>
> > Congratulations, Jing!
> >
> > Best,
> > Swapnal
> >
> > On Mon, 15 Apr 2024, 15:14 Jacky Lau,  wrote:
> >
> > > Congratulations, Jing!
> > >
> > > Best,
> > > Jacky Lau
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread Yuxin Tan

Congratulations, Lincoln!

Best,
Yuxin


Xuannan Su  于2024年4月16日周二 10:26写道：

> Congratulations, Lincoln!
>
> Best regards,
> Xuannan
>
> On Tue, Apr 16, 2024 at 10:04 AM Hang Ruan  wrote:
> >
> > Congratulations, Lincoln!
> >
> > Best,
> > Hang
> >
> > yh z  于2024年4月16日周二 09:14写道：
> >
> > > Congratulations, Lincoln!
> > >
> > > Best,
> > > Yunhong (Swuferhong)
> > >
> > >
> > > Swapnal Varma  于2024年4月15日周一 18:50写道：
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Swapnal
> > > >
> > > >
> > > > On Mon, 15 Apr 2024, 15:16 Jacky Lau,  wrote:
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Best,
> > > > > Jacky Lau
> > > > >
> > > > > Jinzhong Li  于2024年4月15日周一 15:45写道：
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > > Best,
> > > > > > Jinzhong Li
> > > > > >
> > > > > > On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu <
> master...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan <
> > > zakelly@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Lincoln!
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zakelly
> > > > > > > >
> > > > > > > > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky
> > > > > >  > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Lincoln!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ferenc
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Friday, April 12th, 2024 at 15:54,
> > > > > lorenzo.affe...@ververica.com
> > > > > > > > .INVALID
> > > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Huge congrats! Well done!
> > > > > > > > > > On Apr 12, 2024 at 13:56 +0200, Ron liu
> ron9@gmail.com,
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Ron
> > > > > > > > > > >
> > > > > > > > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Junrui
> > > > > > > > > > > >
> > > > > > > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五
> > > > 18:29写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards
> > > > > > > > > > > > > > Aleksandr
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best,
> > > > > > > Hangxiang.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Yuxin Tan

Congratulations, Zakelly!

Best,
Yuxin


Xuannan Su  于2024年4月16日周二 10:30写道：

> Congratulations Zakelly!
>
> Best regards,
> Xuannan
>
> On Mon, Apr 15, 2024 at 4:31 PM Jing Ge 
> wrote:
> >
> > Congratulations Zakelly!
> >
> > Best regards,
> > Jing
> >
> > On Mon, Apr 15, 2024 at 4:26 PM Xia Sun  wrote:
> >
> > > Congratulations Zakelly!
> > >
> > >  Best,
> > >  Xia
> > >
> > > Leonard Xu  于2024年4月15日周一 16:16写道：
> > >
> > > > Congratulations Zakelly!
> > > >
> > > >
> > > > Best,
> > > > Leonard
> > > > > 2024年4月15日 下午3:56，Samrat Deb  写道：
> > > > >
> > > > > Congratulations Zakelly!
> > > >
> > > >
> > >
>

Re: Bug report for reading Hive table as streaming source.

2024-04-15 Thread Xiaolong Wang

Reported. JIRA link: https://issues.apache.org/jira/browse/FLINK-35118

On Mon, Apr 15, 2024 at 12:03 PM Xiaolong Wang 
wrote:

> Sure
>
> On Mon, Apr 1, 2024 at 9:28 AM yuxia  wrote:
>
>> Thanks for reporting. Could you please help create a jira about it?
>>
>> Best regards,
>> Yuxia
>>
>> - 原始邮件 -
>> 发件人: "Xiaolong Wang" 
>> 收件人: "dev" 
>> 发送时间: 星期四, 2024年 3 月 28日 下午 5:11:20
>> 主题: Re: Bug report for reading Hive table as streaming source.
>>
>> I think it worth mentioning in the documentation of Hive read that it
>> cannot read a table that has more than 32,767 partitions.
>>
>> On Thu, Mar 28, 2024 at 5:10 PM Xiaolong Wang <
>> xiaolong.w...@smartnews.com>
>> wrote:
>>
>> > Found out the reason:
>> >
>> > It turned out that in Flink, it uses hive’s IMetaStoreClient to fetch
>> > partitions using the following method:
>> >
>> >   List listPartitionNames(String db_name, String tbl_name,
>> > short max_parts) throws MetaException, TException;
>> >
>> > where the max_parts represents the max number of partitions it can fetch
>> > from the Hive metastore.
>> > So the max number of partitions it can fetch is Short.MAX_VALUE, which
>> is
>> > 32767 .
>> >
>> > But the table has a way more partition number than the max value, thus
>> the
>> > list partition operations cannot fetch all partitions, hence it cannot
>> > consume the recent partition.
>> >
>> > On Tue, Mar 26, 2024 at 5:00 PM Xiaolong Wang <
>> xiaolong.w...@smartnews.com>
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> I found a weird bug when reading a Hive table as a streaming source.
>> >>
>> >> In summary, if the first partition is not time related, then the Hive
>> >> table cannot be read as a streaming source.
>> >>
>> >> e.g.
>> >>
>> >> I've a Hive table in the definition of
>> >>
>> >> ```
>> >> CREATE TABLE article (
>> >> id BIGINT,
>> >> edition STRING,
>> >> dt STRING,
>> >> hh STRING
>> >> )
>> >> PARTITIONED BY (edition, dt, hh)
>> >> USING orc;
>> >> ```
>> >> Then I try to query it as a streaming source:
>> >>
>> >> ```
>> >> INSERT INTO kafka_sink
>> >> SELECT id
>> >> FROM article /*+ OPTIONS('streaming-source.enable' = 'true',
>> >> 'streaming-source.partition-order' = 'partition-name',
>> >> 'streaming-source.consume-start-offset' =
>> >> 'edition=en_US/dt=2024-03-26/hh=00') */
>> >> ```
>> >>
>> >> And I see no output in the `kafka_sink`.
>> >>
>> >> Then I defined an external table pointing to the same path but has no
>> >> `edition` partition,
>> >>
>> >> ```
>> >> CREATE TABLE en_article (
>> >> id BIGINT,
>> >> edition STRING,
>> >> dt STRING,
>> >> hh STRING
>> >> )
>> >> PARTITIONED BY (edition, dt, hh)
>> >> LOCATION 's3://xxx/article/edition=en_US'
>> >> USING orc;
>> >> ```
>> >>
>> >> And insert with the following statement:
>> >>
>> >> ```
>> >> INSERT INTO kafka_sink
>> >> SELECT id
>> >> FROM en_article /*+ OPTIONS('streaming-source.enable' = 'true',
>> >> 'streaming-source.partition-order' = 'partition-name',
>> >> 'streaming-source.consume-start-offset' = 'dt=2024-03-26/hh=00') */
>> >> ```
>> >>
>> >> The data is sinked.
>> >>
>> >>
>> >
>>
>

[jira] [Created] (FLINK-35118) StreamingHiveSource cannot track tables that have more than 32,767 partitions

2024-04-15 Thread roland (Jira)

roland created FLINK-35118:
--

 Summary: StreamingHiveSource cannot track tables that have more 
than 32,767 partitions
 Key: FLINK-35118
 URL: https://issues.apache.org/jira/browse/FLINK-35118
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.19.0
Reporter: roland


*Description:*

The Streaming Hive Source cannot track tables that have more than 32,767 
partitions.

 *Root Cause:*

The Streaming Hive Source uses the following lines to get all partitions of a 
table:

([git hub 
link|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HivePartitionFetcherContextBase.java#L130])

HivePartitionFetcherContextBase.java:
{code:java}
    @Override
    public List getComparablePartitionValueList() 
throws Exception {
        List partitionValueList = new ArrayList<>();
        switch (partitionOrder) {
            case PARTITION_NAME:
                List partitionNames =
                        metaStoreClient.listPartitionNames(
                                tablePath.getDatabaseName(),
                                tablePath.getObjectName(),
                                Short.MAX_VALUE);
                for (String partitionName : partitionNames) {
                    
partitionValueList.add(getComparablePartitionByName(partitionName));
                }
                break;
            case CREATE_TIME:
                Map, Long> partValuesToCreateTime = new 
HashMap<>();
                partitionNames =
                        metaStoreClient.listPartitionNames(
                                tablePath.getDatabaseName(),
                                tablePath.getObjectName(),
                                Short.MAX_VALUE); {code}
Where the `metaStoreClient` is a wrapper of the `IMetaStoreClient`, and the 
function `listPartitionNames` can only list no more than `Short.MAX_VALUE` 
partitions, which is 32,767.

 

For tables that have more partitions, the source fails to track new partitions 
and read from it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Connector releases for Flink 1.19

2024-04-15 Thread Muhammet Orazov


Hello Danny,

I would like to work on this task, thanks for initiating it!

I could update the versions on JDBC and Pulsar connectors.

Are you going to create an umbrella issue for it?

Best,
Muhammet

On 2024-04-15 13:44, Danny Cranmer wrote:

Hello all,

Flink 1.19 was released on 2024-03-18 [1] and the connectors have not 
yet

caught up. I propose we start releasing the connectors with support for
Flink 1.19 as per the connector support guidelines [2].

I will make a start on flink-connector-aws, then pickup others in the
coming days. Please respond to the thread if you are/want to work on a
particular connector to avoid duplicate work.

Thanks,
Danny

[1]
https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/
[2]
https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Flinkcompatibility
[3] https://github.com/apache/flink-connector-aws

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Xuannan Su

Congratulations Zakelly!

Best regards,
Xuannan

On Mon, Apr 15, 2024 at 4:31 PM Jing Ge  wrote:
>
> Congratulations Zakelly!
>
> Best regards,
> Jing
>
> On Mon, Apr 15, 2024 at 4:26 PM Xia Sun  wrote:
>
> > Congratulations Zakelly!
> >
> >  Best,
> >  Xia
> >
> > Leonard Xu  于2024年4月15日周一 16:16写道：
> >
> > > Congratulations Zakelly!
> > >
> > >
> > > Best,
> > > Leonard
> > > > 2024年4月15日 下午3:56，Samrat Deb  写道：
> > > >
> > > > Congratulations Zakelly!
> > >
> > >
> >

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread Xuannan Su

Congratulations, Lincoln!

Best regards,
Xuannan

On Tue, Apr 16, 2024 at 10:04 AM Hang Ruan  wrote:
>
> Congratulations, Lincoln!
>
> Best,
> Hang
>
> yh z  于2024年4月16日周二 09:14写道：
>
> > Congratulations, Lincoln!
> >
> > Best,
> > Yunhong (Swuferhong)
> >
> >
> > Swapnal Varma  于2024年4月15日周一 18:50写道：
> >
> > > Congratulations, Lincoln!
> > >
> > > Best,
> > > Swapnal
> > >
> > >
> > > On Mon, 15 Apr 2024, 15:16 Jacky Lau,  wrote:
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Jacky Lau
> > > >
> > > > Jinzhong Li  于2024年4月15日周一 15:45写道：
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Best,
> > > > > Jinzhong Li
> > > > >
> > > > > On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu 
> > > > wrote:
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > > On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan <
> > zakelly@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky
> > > > >  > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Lincoln!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Ferenc
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Friday, April 12th, 2024 at 15:54,
> > > > lorenzo.affe...@ververica.com
> > > > > > > .INVALID
> > > > > > > >  wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Huge congrats! Well done!
> > > > > > > > > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com,
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Ron
> > > > > > > > > >
> > > > > > > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Junrui
> > > > > > > > > > >
> > > > > > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五
> > > 18:29写道：
> > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best Regards
> > > > > > > > > > > > > Aleksandr
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Hangxiang.
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-04-15 Thread Muhammet Orazov


+1 (non-binding)

Thanks Priya for the FLIP and driving it!

Best,
Muhammet

On 2024-04-12 21:56, Dhingra, Priya wrote:

Hi devs,



Thank you to everyone for the feedback on FLIP-438: Amazon SQS Sink 
Connector




I would like to start a vote for it. The vote will be open for at least 
72


hours unless there is an objection or not enough votes.



https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector

Regards
Priya

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

2024-04-15 Thread Muhammet Orazov

Hey Priya,

Having local dockerized integration testing would help devs to run the 
integration tests when they don't have AWS service credentials.

You could have a look to the DynamoDB [1] or Kinesis [2] connector 
integration tests. DynamoDBContainer extends interface from 
Testcontainers [3].

Best,
Muhammet

[1]: 
https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/sink/DynamoDbSinkITCase.java#L74-L78
[2]: 
https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisITCase.java#L87-L89
[3]: 
https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/testutils/DynamoDbContainer.java

On 2024-04-15 20:31, Dhingra, Priya wrote:

Thanks Danny/ Muhammet for the feedback. Updated the FLIP accordingly.

I am not familiar with docker based integration testing but if that is 
what we have used for other AWS services, I can follow the same here. 
If you can share any existing sample code link that would be very 
helpful.

Thanks
Priya

On 4/15/24, 5:22 AM, "Danny Cranmer" > wrote:

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and 
know the content is safe.

Thanks for the FLIP Priya. I have seen many custom implementations of 
SQS

sinks, so having an Apache supported connector will be great.

I agree on the points about implementation detail in the FLIP. There is
still too much code really, we will review that in the PR. The FLIP is
mainly to agree we should add this connector, we can leave the
implementation details for later.

nit: Wrt to versioning, it is actually Flink 1.18 and 1.19 now, since 
1.17
is out of support. But since we are using the flink-connector-aws repo 
we

will use the supported version there.

Would it make sense to use here docker based integration testing using

Testcontainer

+1, we typically use localstack for the Integration tests and hit AWS 
in

the e2e test packages. Having coverage of both will be great.

Thanks,
Danny

On Mon, Apr 15, 2024 at 3:23 AM Muhammet Orazov
mailto:mor+fl...@morazow.com.inva>lid> 
wrote:

Hey Priya,

Thanks for the FLIP and driving it!

One question from my side on test plan:

> End to end integration tests that hit the real Amazon SQS service.
> These tests will be enabled when credentials are defined.

Would it make sense to use here docker based integration testing using
Testcontainer[1], for example, with Localstack[2]? Seems like the SQS
service is provided on the free tier.

[1]: https://java.testcontainers.org/modules/localstack/ 

[2]: https://docs.localstack.cloud/user-guide/aws/sqs/ 

On 2024-04-05 18:16, Dhingra, Priya wrote:
> Hi Dev,
>
> I would like to start a discussion about FLIP-438: Amazon SQS Sink
> Connector<
https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector> 
.

> This FLIP is proposing to add support for AWS SQS sink in
> flink-connector-aws repo.
>
> For more details, see FLIP-438. Looking forward to your feedback.
>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector 

>
> Thanks,
> Priya

[jira] [Created] (FLINK-35117) AsyncScalarFunction has a dependency issue.

2024-04-15 Thread roland (Jira)

roland created FLINK-35117:
--

 Summary: AsyncScalarFunction has a dependency issue.
 Key: FLINK-35117
 URL: https://issues.apache.org/jira/browse/FLINK-35117
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.19.0
 Environment: image: 1.19.0-scala_2.12-java11
Reporter: roland


Hi,
 
I found a ClassNotFound exception when using Flink 1.19's AsyncScalarFunction. 
 
*Stack trace:*
 
{quote}Caused by: java.lang.ClassNotFoundException: 
org.apache.commons.text.StringSubstitutor

at java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]

at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]

at 
org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:150)
 ~[flink-dist-1.19.0.jar:1.19.0]

at 
org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:113)
 ~[flink-dist-1.19.0.jar:1.19.0]

at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]

at 
org.apache.flink.table.planner.codegen.AsyncCodeGenerator.generateProcessCode(AsyncCodeGenerator.java:173)
 ~[?:?]

at 
org.apache.flink.table.planner.codegen.AsyncCodeGenerator.generateFunction(AsyncCodeGenerator.java:77)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecAsyncCalc.getAsyncFunctionOperator(CommonExecAsyncCalc.java:146)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecAsyncCalc.createAsyncOneInputTransformation(CommonExecAsyncCalc.java:126)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecAsyncCalc.translateToPlanInternal(CommonExecAsyncCalc.java:89)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:168)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:259)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecCalc.translateToPlanInternal(CommonExecCalc.java:94)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:168)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:259)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.java:177)
 ~[?:?]

at 
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:168)
 ~[?:?]

at 
org.apache.flink.table.planner.delegation.StreamPlanner.$anonfun$translateToPlan$1(StreamPlanner.scala:85)
 ~[?:?]
{quote}
 
*Root cause:*
 
`PlannerModule` uses a URLClassloader to load class and has an exceptional list 
to load owner owned classes:
 
{code:java}
class PlannerModule {

/**
 * The name of the table planner dependency jar, bundled with 
flink-table-planner-loader module
 * artifact.
 */
static final String FLINK_TABLE_PLANNER_FAT_JAR = "flink-table-planner.jar";

private static final String HINT_USAGE =
"mvn clean package -pl 
flink-table/flink-table-planner,flink-table/flink-table-planner-loader 
-DskipTests";

private static final String[] OWNER_CLASSPATH =
Stream.concat(

Arrays.stream(CoreOptions.PARENT_FIRST_LOGGING_PATTERNS),
Stream.of(
// These packages are shipped either by
// flink-table-runtime or flink-dist itself
"org.codehaus.janino",
"org.codehaus.commons",
"org.apache.commons.lang3",
"org.apache.commons.math3",
// with hive dialect, hadoop jar should be 
in classpath,
// also, we should make it loaded by owner 
classloader,
// otherwise, it'll throw class not found 
exception
// when initialize HiveParser which 
requires hadoop
"org.apache.hadoop"))
                    .toArray(String[]::new);  {code}
But the group of `org.apache.commons.text` is not on the list.

 

*Fix:*

Add `org.apache.commons.text` to the list



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread Hang Ruan

Congratulations, Lincoln!

Best,
Hang

yh z  于2024年4月16日周二 09:14写道：

> Congratulations, Lincoln!
>
> Best,
> Yunhong (Swuferhong)
>
>
> Swapnal Varma  于2024年4月15日周一 18:50写道：
>
> > Congratulations, Lincoln!
> >
> > Best,
> > Swapnal
> >
> >
> > On Mon, 15 Apr 2024, 15:16 Jacky Lau,  wrote:
> >
> > > Congratulations, Lincoln!
> > >
> > > Best,
> > > Jacky Lau
> > >
> > > Jinzhong Li  于2024年4月15日周一 15:45写道：
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Jinzhong Li
> > > >
> > > > On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu 
> > > wrote:
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan <
> zakelly@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky
> > > >  > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > Best,
> > > > > > > Ferenc
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Friday, April 12th, 2024 at 15:54,
> > > lorenzo.affe...@ververica.com
> > > > > > .INVALID
> > > > > > >  wrote:
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Huge congrats! Well done!
> > > > > > > > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com,
> > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Lincoln!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > > > > > >
> > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Junrui
> > > > > > > > > >
> > > > > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五
> > 18:29写道：
> > > > > > > > > >
> > > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > > >
> > > > > > > > > > > > Best Regards
> > > > > > > > > > > > Aleksandr
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best,
> > > > > Hangxiang.
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread yh z

Congratulations, Lincoln!

Best,
Yunhong (Swuferhong)


Swapnal Varma  于2024年4月15日周一 18:50写道：

> Congratulations, Lincoln!
>
> Best,
> Swapnal
>
>
> On Mon, 15 Apr 2024, 15:16 Jacky Lau,  wrote:
>
> > Congratulations, Lincoln!
> >
> > Best,
> > Jacky Lau
> >
> > Jinzhong Li  于2024年4月15日周一 15:45写道：
> >
> > > Congratulations, Lincoln!
> > >
> > > Best,
> > > Jinzhong Li
> > >
> > > On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu 
> > wrote:
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan 
> > > > wrote:
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky
> > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > > Best,
> > > > > > Ferenc
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Friday, April 12th, 2024 at 15:54,
> > lorenzo.affe...@ververica.com
> > > > > .INVALID
> > > > > >  wrote:
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Huge congrats! Well done!
> > > > > > > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com,
> > wrote:
> > > > > > >
> > > > > > > > Congratulations, Lincoln!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > > > > >
> > > > > > > > > Congratulations, Lincoln!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Junrui
> > > > > > > > >
> > > > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五
> 18:29写道：
> > > > > > > > >
> > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > >
> > > > > > > > > > > Best Regards
> > > > > > > > > > > Aleksandr
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Hangxiang.
> > > >
> > >
> >
>

Re: Question around Flink's AdaptiveBatchScheduler

2024-04-15 Thread Venkatakrishnan Sowrirajan

Thanks for adding your thoughts to this discussion.

If we all agree that the source vertex parallelism shouldn't be bound by
the downstream max parallelism
(jobmanager.adaptive-batch-scheduler.max-parallelism)
based on the rationale and the issues described above, I can take a stab at
addressing the issue.

Let me file a ticket to track this issue. Otherwise, I'm looking forward to
hearing more thoughts from others as well, especially Lijie and Junrui who
have more context on the AdaptiveBatchScheduler.

Regards
Venkata krishnan


On Mon, Apr 15, 2024 at 12:54 AM Xia Sun  wrote:

> Hi Venkat,
> I agree that the parallelism of source vertex should not be upper bounded
> by the job's global max parallelism. The case you mentioned, >> High filter
> selectivity with huge amounts of data to read  excellently supports this
> viewpoint. (In fact, in the current implementation, if the source
> parallelism is pre-specified at job create stage, rather than relying on
> the dynamic parallelism inference of the AdaptiveBatchScheduler, the source
> vertex's parallelism can indeed exceed the job's global max parallelism.)
>
> As Lijie and Junrui pointed out, the key issue is "semantic consistency."
> Currently, if a vertex has not set maxParallelism, the
> AdaptiveBatchScheduler will use
> `execution.batch.adaptive.auto-parallelism.max-parallelism` as the vertex's
> maxParallelism. Since the current implementation does not distinguish
> between source vertices and downstream vertices, source vertices are also
> subject to this limitation.
>
> Therefore, I believe that if the issue of "semantic consistency" can be
> well explained in the code and configuration documentation, the
> AdaptiveBatchScheduler should support that the parallelism of source
> vertices can exceed the job's global max parallelism.
>
> Best,
> Xia
>
> Venkatakrishnan Sowrirajan  于2024年4月14日周日 10:31写道：
>
> > Let me state why I think "*jobmanager.adaptive-batch-sche*
> > *duler.default-source-parallelism*" should not be bound by the "
> > *jobmanager.adaptive-batch-sche**duler.max-parallelism*".
> >
> >- Source vertex is unique and does not have any upstream vertices
> >- Downstream vertices read shuffled data partitioned by key, which is
> >not the case for the Source vertex
> >- Limiting source parallelism by downstream vertices' max parallelism
> is
> >incorrect
> >
> > If we say for ""semantic consistency" the source vertex parallelism has
> to
> > be bound by the overall job's max parallelism, it can lead to following
> > issues:
> >
> >- High filter selectivity with huge amounts of data to read - setting
> >high "*jobmanager.adaptive-batch-scheduler.max-parallelism*" so that
> >source parallelism can be set higher can lead to small blocks and
> >sub-optimal performance.
> >- Setting high "*jobmanager.adaptive-batch-scheduler.max-parallelism*"
> >requires careful tuning of network buffer configurations which is
> >unnecessary in cases where it is not required just so that the source
> >parallelism can be set high.
> >
> > Regards
> > Venkata krishnan
> >
> > On Thu, Apr 11, 2024 at 9:30 PM Junrui Lee  wrote:
> >
> > > Hello Venkata krishnan,
> > >
> > > I think the term "semantic inconsistency" defined by
> > > jobmanager.adaptive-batch-scheduler.max-parallelism refers to
> > maintaining a
> > > uniform upper limit on parallelism across all vertices within a job. As
> > the
> > > source vertices are part of the global execution graph, they should
> also
> > > respect this rule to ensure consistent application of parallelism
> > > constraints.
> > >
> > > Best,
> > > Junrui
> > >
> > > Venkatakrishnan Sowrirajan  于2024年4月12日周五 02:10写道：
> > >
> > > > Gentle bump on this question. cc @Becket Qin 
> as
> > > > well.
> > > >
> > > > Regards
> > > > Venkata krishnan
> > > >
> > > >
> > > > On Tue, Mar 12, 2024 at 10:11 PM Venkatakrishnan Sowrirajan <
> > > > vsowr...@asu.edu> wrote:
> > > >
> > > > > Thanks for the response Lijie and Junrui. Sorry for the late reply.
> > Few
> > > > > follow up questions.
> > > > >
> > > > > > Source can actually ignore this limit
> > > > > because it has no upstream, but this will lead to semantic
> > > inconsistency.
> > > > >
> > > > > Lijie, can you please elaborate on the above comment further? What
> do
> > > you
> > > > > mean when you say it will lead to "semantic inconsistency"?
> > > > >
> > > > > > Secondly, we first need to limit the max parallelism of
> > (downstream)
> > > > > vertex, and then we can decide how many subpartitions (upstream
> > vertex)
> > > > > should produce. The limit should be effective, otherwise some
> > > downstream
> > > > > tasks will have no data to process.
> > > > >
> > > > > This makes sense in the context of any other vertices other than
> the
> > > > > source vertex. As you mentioned above ("Source can actually ignore
> > this
> > > > > limit because it has no upstream"), therefore I feel "
> > > > >

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

2024-04-15 Thread Dhingra, Priya

Thanks Danny/ Muhammet for the feedback. Updated the FLIP accordingly.

I am not familiar with docker based integration testing but if that is what we 
have used for other AWS services, I can follow the same here. If you can share 
any existing sample code link that would be very helpful.

Thanks
Priya

On 4/15/24, 5:22 AM, "Danny Cranmer" mailto:dannycran...@apache.org>> wrote:

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Thanks for the FLIP Priya. I have seen many custom implementations of SQS
sinks, so having an Apache supported connector will be great.

I agree on the points about implementation detail in the FLIP. There is
still too much code really, we will review that in the PR. The FLIP is
mainly to agree we should add this connector, we can leave the
implementation details for later.

nit: Wrt to versioning, it is actually Flink 1.18 and 1.19 now, since 1.17
is out of support. But since we are using the flink-connector-aws repo we
will use the supported version there.

> Would it make sense to use here docker based integration testing using
Testcontainer

+1, we typically use localstack for the Integration tests and hit AWS in
the e2e test packages. Having coverage of both will be great.

Thanks,
Danny

On Mon, Apr 15, 2024 at 3:23 AM Muhammet Orazov
mailto:mor+fl...@morazow.com.inva>lid> wrote:

> Hey Priya,
>
> Thanks for the FLIP and driving it!
>
> One question from my side on test plan:
>
> > End to end integration tests that hit the real Amazon SQS service.
> > These tests will be enabled when credentials are defined.
>
> Would it make sense to use here docker based integration testing using
> Testcontainer[1], for example, with Localstack[2]? Seems like the SQS
> service is provided on the free tier.
>
> [1]: https://java.testcontainers.org/modules/localstack/ 
> 
> [2]: https://docs.localstack.cloud/user-guide/aws/sqs/ 
> 
>
> On 2024-04-05 18:16, Dhingra, Priya wrote:
> > Hi Dev,
> >
> > I would like to start a discussion about FLIP-438: Amazon SQS Sink
> > Connector<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector>
>  
> .
>
> > This FLIP is proposing to add support for AWS SQS sink in
> > flink-connector-aws repo.
> >
> > For more details, see FLIP-438. Looking forward to your feedback.
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector
>  
> 
> >
> > Thanks,
> > Priya
>

[jira] [Created] (FLINK-35116) Upgrade JOSDK dependency to 4.8.3

2024-04-15 Thread Jira

Márton Balassi created FLINK-35116:
--

 Summary: Upgrade JOSDK dependency to 4.8.3
 Key: FLINK-35116
 URL: https://issues.apache.org/jira/browse/FLINK-35116
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Márton Balassi
Assignee: Márton Balassi
 Fix For: kubernetes-operator-1.9.0


This bring a much needed fix for the operator HA behaviour:

https://github.com/operator-framework/java-operator-sdk/releases/tag/v4.8.3



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-438: Make Flink's Hadoop and YARN configuration probing consistent

2024-04-15 Thread Venkatakrishnan Sowrirajan

Sorry for the late reply, Ferenc.

I understand the rationale behind the current implementation as the problem
is slightly different b/w yarn (always prefixed with `yarn`) and hadoop (it
is not guaranteed all `hadoop` configs will be prefixed by `hadoop`)
configs.

>From the dev UX perspective, it is confusing and only if you really pay
close attention to the docs it is evident. I understand your point on added
complexity till Flink-3.0 but if we agree it should be made consistent, it
has to be done at some point of time right?

Regards
Venkata krishnan


On Wed, Apr 3, 2024 at 4:51 AM Ferenc Csaky 
wrote:

> Hi Venkata,
>
> Thank you for opening the discussion about this!
>
> After taking a look at the YARN and Hadoop configurations, the
> reason why it was implemented this way is that, in case of YARN,
> every YARN-specific property is prefixed with "yarn.", so to get
> the final, YARN-side property it is enough to remove the "flink."
> prefix.
>
> In case of Hadoop, there are properties that not prefixed with
> "hadoop.", e.g. "dfs.replication" so to identify and get the
> Hadoop-side property it is necessary to duplicate the "hadoop" part
> in the properties.
>
> Taking this into consideration I would personally say -0 to this
> change. IMO the current behavior can be justified as giving
> slightly different solutions to slightly different problems, which
> are well documented. Handling both prefixes would complicate the
> parsing logic until the APIs can be removed, which as it looks at
> the moment would only be possible in Flink 3.0, which probably will
> not happen in the foreseeable future, so I do not see the benefit
> of the added complexity.
>
> Regarding the FLIP, in the "YARN configuration override example"
> part, I think you should present an example that works correctly
> at the moment: "flink.yarn.application.classpath" ->
> "yarn.application.classpath".
>
> Best,
> Ferenc
>
>
> On Friday, March 29th, 2024 at 23:45, Venkatakrishnan Sowrirajan <
> vsowr...@asu.edu> wrote:
>
> >
> >
> > Hi Flink devs,
> >
> > I would like to start a discussion on FLIP-XXX: Make Flink's Hadoop and
> > YARN configuration probing consistent
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1I2jBFI0eVkofAVCAEeajNQRfOqKGJsRfZd54h79AIYc/edit?usp=sharing__;!!IKRxdwAv5BmarQ!d0XJO_mzLCJZNkrjJDMyRGP95zPLW8Cuym88l7CoAUG8aD_KRYJbll3K-q1Ypplyqe6-jcsWq3S8YJqrDMCpK4IhpT4cZPXy$
> .
> >
> > This stems from an earlier discussion thread here
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/l2fh5shbf59fjgbt1h73pmmsqj038ppv__;!!IKRxdwAv5BmarQ!d0XJO_mzLCJZNkrjJDMyRGP95zPLW8Cuym88l7CoAUG8aD_KRYJbll3K-q1Ypplyqe6-jcsWq3S8YJqrDMCpK4IhpW60A99X$
> .
> >
> >
> > This FLIP is proposing to make the configuration probing behavior between
> > Hadoop and YARN configuration to be consistent.
> >
> > Regards
> > Venkata krishnan
>

[jira] [Created] (FLINK-35115) Kinesis connector writes wrong sequence number at stop with savepoint

2024-04-15 Thread Vadim Vararu (Jira)

Vadim Vararu created FLINK-35115:


 Summary: Kinesis connector writes wrong sequence number at stop 
with savepoint
 Key: FLINK-35115
 URL: https://issues.apache.org/jira/browse/FLINK-35115
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Affects Versions: 1.16.3
 Environment: The issue happens in a *Kinesis -> Flink -> Kafka* 
exactly-once setup with:
 * Flink versions checked 1.16.3 and 1.18.1
 * Kinesis connector checked 1.16.3 and 4.2.0-1.18
 * checkpointing configured at 1 minute with EXACTLY_ONCE mode: 
{code:java}
StreamExecutionEnvironment execEnv = 
StreamExecutionEnvironment.getExecutionEnvironment (); 
execEnv.enableCheckpointing (6,EXACTLY_ONCE); execEnv.getCheckpointConfig 
().setCheckpointTimeout (9); execEnv.getCheckpointConfig 
().setCheckpointStorage (CHECKPOINTS_PATH); {code}

 * Kafka sink configured with EXACTLY_ONCE semantic/delivery guarantee:
{code:java}
Properties sinkConfig = new Properties ();
sinkConfig.put ("transaction.timeout.ms", 48);

KafkaSink sink = KafkaSink.builder ()
.setBootstrapServers ("localhost:9092")
.setTransactionalIdPrefix ("test-prefix")
.setDeliverGuarantee (EXACTLY_ONCE)
.setKafkaProducerConfig (sinkConfig)
.setRecordSerializer (
(KafkaRecordSerializationSchema) (element, context, timestamp) 
-> new ProducerRecord<> (
"test-output-topic", null, element.getBytes ()))
.build (); {code}

 * Kinesis consumer defined as: 
{code:java}
FlinkKinesisConsumer flinkKinesisConsumer = new
FlinkKinesisConsumer<> ("test-stream",
new AbstractDeserializationSchema<> () {
@Override
public ByteBuffer deserialize (byte[] bytes) {
// Return
return ByteBuffer.wrap (bytes);
}
}, props); {code}
 
Reporter: Vadim Vararu


Having an exactly-once Kinesis -> Flink -> Kafka job and triggering a 
stop-with-savepoint, Flink duplicates in Kafka all the records between the last 
checkpoint and the savepoint at resume:
 * Event1 is written to Kinesis
 * Event1 is processed by Flink 
 * Event1 is committed to Kafka at the checkpoint
 * 
 * Event2 is written to Kinesis
 * Event2 is processed by Flink
 * Stop with savepoint is triggered manually
 * Event2 is committed to Kafka
 * 
 * Job is resumed from the savepoint
 * *{color:#FF}Event2 is written again to Kafka at the first 
checkpoint{color}*

 

{color:#172b4d}I believe that it's a Kinesis connector issue for 2 
reasons:{color}
 * I've checked the actual Kinesis sequence number in the _metadata file 
generated at stop-with-savepoint and it's the one from the checkpoint before 
the savepoint  instead of being the one of the last record committed to Kafka.
 * I've tested exactly the save job with Kafka as source instead of Kinesis as 
source and the behaviour does not reproduce.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] Connector releases for Flink 1.19

2024-04-15 Thread Danny Cranmer

Hello all,

Flink 1.19 was released on 2024-03-18 [1] and the connectors have not yet
caught up. I propose we start releasing the connectors with support for
Flink 1.19 as per the connector support guidelines [2].

I will make a start on flink-connector-aws, then pickup others in the
coming days. Please respond to the thread if you are/want to work on a
particular connector to avoid duplicate work.

Thanks,
Danny

[1]
https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/
[2]
https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Flinkcompatibility
[3] https://github.com/apache/flink-connector-aws

[jira] [Created] (FLINK-35114) Remove old Table API implementations

2024-04-15 Thread Ferenc Csaky (Jira)

Ferenc Csaky created FLINK-35114:


 Summary: Remove old Table API implementations
 Key: FLINK-35114
 URL: https://issues.apache.org/jira/browse/FLINK-35114
 Project: Flink
  Issue Type: Sub-task
Reporter: Ferenc Csaky


At the moment, the connector has both the old Table sink/source/catalog 
implementations and the matching Dynamic... implementations as well.

Going forward, the deprecated old implementation should be removed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-04-15 Thread Aleksandr Pilipenko

+1 (non-binding),

Best,
Aleksandr

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-15 Thread Danny Cranmer

Congrats Jing!

Best Regards,
Danny

On Mon, Apr 15, 2024 at 11:51 AM Swapnal Varma  wrote:

> Congratulations, Jing!
>
> Best,
> Swapnal
>
> On Mon, 15 Apr 2024, 15:14 Jacky Lau,  wrote:
>
> > Congratulations, Jing!
> >
> > Best,
> > Jacky Lau
> >
>

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-04-15 Thread Danny Cranmer

Thanks for driving this Priya.

+1 (binding)

Danny.

On Sun, Apr 14, 2024 at 12:33 PM Samrat Deb  wrote:

> +1 (non-binding)
>
> Bests,
> Samrat
>
> On Sun, 14 Apr 2024 at 3:38 PM, Ahmed Hamdy  wrote:
>
> > Hi Priya
> > +1 (non-binding)
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Fri, 12 Apr 2024 at 22:56, Dhingra, Priya  >
> > wrote:
> >
> > > Hi devs,
> > >
> > >
> > >
> > > Thank you to everyone for the feedback on FLIP-438: Amazon SQS Sink
> > > Connector<
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector
> > > >
> > >
> > >
> > >
> > > I would like to start a vote for it. The vote will be open for at least
> > 72
> > >
> > > hours unless there is an objection or not enough votes.
> > >
> > >
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector
> > >
> > > Regards
> > > Priya
> > >
> >
>

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

2024-04-15 Thread Danny Cranmer

Thanks for the FLIP Priya. I have seen many custom implementations of SQS
sinks, so having an Apache supported connector will be great.

I agree on the points about implementation detail in the FLIP. There is
still too much code really, we will review that in the PR. The FLIP is
mainly to agree we should add this connector, we can leave the
implementation details for later.

nit: Wrt to versioning, it is actually Flink 1.18 and 1.19 now, since 1.17
is out of support. But since we are using the flink-connector-aws repo we
will use the supported version there.

> Would it make sense to use here docker based integration testing using
Testcontainer

+1, we typically use localstack for the Integration tests and hit AWS in
the e2e test packages. Having coverage of both will be great.

Thanks,
Danny

On Mon, Apr 15, 2024 at 3:23 AM Muhammet Orazov
 wrote:

> Hey Priya,
>
> Thanks for the FLIP and driving it!
>
> One question from my side on test plan:
>
> > End to end integration tests that hit the real Amazon SQS service.
> > These tests will be enabled when credentials are defined.
>
> Would it make sense to use here docker based integration testing using
> Testcontainer[1], for example, with Localstack[2]? Seems like the SQS
> service is provided on the free tier.
>
> [1]: https://java.testcontainers.org/modules/localstack/
> [2]: https://docs.localstack.cloud/user-guide/aws/sqs/
>
> On 2024-04-05 18:16, Dhingra, Priya wrote:
> > Hi Dev,
> >
> > I would like to start a discussion about FLIP-438: Amazon SQS Sink
> > Connector<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector>.
>
> > This FLIP is proposing to add support for AWS SQS sink in
> > flink-connector-aws repo.
> >
> > For more details, see FLIP-438. Looking forward to your feedback.
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector
> >
> > Thanks,
> > Priya
>

[jira] [Created] (FLINK-35113) Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink AWS connectors

2024-04-15 Thread Danny Cranmer (Jira)

Danny Cranmer created FLINK-35113:
-

 Summary: Bump org.apache.commons:commons-compress from 1.25.0 to 
1.26.1 for Flink AWS connectors
 Key: FLINK-35113
 URL: https://issues.apache.org/jira/browse/FLINK-35113
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Kafka
Reporter: Martijn Visser
Assignee: Martijn Visser
 Fix For: kafka-4.0.0, kafka-3.1.1






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-15 Thread David Radley

Hi Martijn,
Thanks for looking at this. I have used the template in a new  Google Doc 
https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing.
  I have significantly reduced the content in the Flip, in line with what I see 
as the template and its usage. If this it too much or too little, I can amend,

Kind regards, David.

From: Martijn Visser 
Date: Friday, 12 April 2024 at 18:11
To: dev@flink.apache.org 
Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi David,

I tried, but the format wasn't as the FLIP template expects, so I ended up
needing to change the entire formatting and that was just too much work to
be honest. If you could make sure that especially the headers match with
the FLIP template, and that all of the contents from the FLIP template is
there, that would make things much easier.

Thanks,

Martijn

On Fri, Apr 12, 2024 at 6:08 PM David Radley 
wrote:

> Hi,
> A gentle nudge. Please could a committer/PMC member raise the Flip for
> this,
>   Kind regards, David.
>
>
> From: David Radley 
> Date: Monday, 8 April 2024 at 09:40
> To: dev@flink.apache.org 
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi,
> I have posted a Google Doc [0] to the mailing list for a discussion thread
> for a Flip proposal to introduce a Apicurio-avro format. The discussions
> have been resolved, please could a committer/PMC member copy the contents
> from the Google Doc, and create a FLIP number for this,. as per the process
> [1],
>   Kind regards, David.
> [0]
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w_edit-3Fusp-3Dsharing=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=pHL2e8waNNtvTDT0a3PQM0bcXrb1Fywv0YW_Ln50jCo=
>
> [1]
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_Flink-2BImprovement-2BProposals-23FlinkImprovementProposals-2DCreateyourOwnFLIP=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=_7fvlZYc-gUtkFEhwSz9utYsgbDrUtkHEToTdhtQvQc=
>
> From: Jeyhun Karimov 
> Date: Friday, 22 March 2024 at 13:05
> To: dev@flink.apache.org 
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> Thanks a lot for clarification.
> Sounds good to me.
>
> Regards,
> Jeyhun
>
> On Fri, Mar 22, 2024 at 10:54 AM David Radley 
> wrote:
>
> > Hi Jeyhun,
> > Thanks for your feedback.
> >
> > So for outbound messages, the message includes the global ID. We register
> > the schema and match on the artifact id. So if the schema then evolved,
> > adding a new  version, the global ID would still be unique and the same
> > version would be targeted. If you wanted to change the Flink table
> > definition in line with a higher version, then you could do this – the
> > artifact id would need to match for it to use the same schema and a
> higher
> > artifact version would need to be provided. I notice that Apicurio has
> > rules around compatibility that you can configure, I suppose if we
> attempt
> > to create an artifact that breaks these rules , then the register schema
> > will fail and the associated operation should fail (e.g. an insert). I
> have
> > not tried this.
> >
> >
> > For inbound messages, using the global id in the header – this targets
> one
> > version of the schema. I can create different messages on the topic built
> > with different schema versions, and I can create different tables in
> Flink,
> > as long as the reader and writer schemas are compatible as per the
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_flink_blob_779459168c46b7b4c600ef52f99a5435f81b9048_flink-2Dformats_flink-2Davro_src_main_java_org_apache_flink_formats_avro_RegistryAvroDeserializationSchema.java-23L109=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=kfPzGTjUx9alvbOMoJoeWEHHQ14qwYxTJXbVWAhYvAc=
> > Then this should work.
> >
> > Does this address your question?
> > Kind regards, David.
> >
> >
> > From: Jeyhun Karimov 
> > Date: Thursday, 21 March 2024 at 21:06
> > To: dev@flink.apache.org 
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > Thanks for the FLIP. +1 for it.
> > I have a minor comment.
> >
> > Can you please elaborate more on mechanisms in place to ensure data
> > consistency and integrity, particularly in the event of schema conflicts?
> > Since each message includes a schema ID for inbound and outbound
> messages,
> > can you elaborate more on message consistency in the context of schema
> > evolution?
> >
> > Regards,
> > Jeyhun
> >
> >
> >
> >
> >
> > On Wed, Mar 20, 2024 at 4:34 PM David Radley 
> wrote:
> >
> > > Thank you very

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-15 Thread Swapnal Varma

Congratulations, Jing!

Best,
Swapnal

On Mon, 15 Apr 2024, 15:14 Jacky Lau,  wrote:

> Congratulations, Jing!
>
> Best,
> Jacky Lau
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread Swapnal Varma

Congratulations, Lincoln!

Best,
Swapnal


On Mon, 15 Apr 2024, 15:16 Jacky Lau,  wrote:

> Congratulations, Lincoln!
>
> Best,
> Jacky Lau
>
> Jinzhong Li  于2024年4月15日周一 15:45写道：
>
> > Congratulations, Lincoln!
> >
> > Best,
> > Jinzhong Li
> >
> > On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu 
> wrote:
> >
> > > Congratulations, Lincoln!
> > >
> > > On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan 
> > > wrote:
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky
> >  > > >
> > > > wrote:
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Best,
> > > > > Ferenc
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Friday, April 12th, 2024 at 15:54,
> lorenzo.affe...@ververica.com
> > > > .INVALID
> > > > >  wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > Huge congrats! Well done!
> > > > > > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com,
> wrote:
> > > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > Best,
> > > > > > > Ron
> > > > > > >
> > > > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > > > >
> > > > > > > > Congratulations, Lincoln!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Junrui
> > > > > > > >
> > > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:29写道：
> > > > > > > >
> > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > >
> > > > > > > > > > Best Regards
> > > > > > > > > > Aleksandr
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Hangxiang.
> > >
> >
>

[jira] [Created] (FLINK-35112) Membership for Row class does not include field names

2024-04-15 Thread Wouter Zorgdrager (Jira)

Wouter Zorgdrager created FLINK-35112:
-

 Summary: Membership for Row class does not include field names
 Key: FLINK-35112
 URL: https://issues.apache.org/jira/browse/FLINK-35112
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.18.1
Reporter: Wouter Zorgdrager


In the Row class in PyFlink I cannot do a membership check for field names. 
This minimal example will show the unexpected behavior:

```

from pyflink.common import Row

row = Row(name="Alice", age=11)
# Expected to be True, but is False
print("name" in row)

person = Row("name", "age")
# This is True, as expected
print('name' in person)

```

The related code in the Row class is:
```
    def __contains__(self, item):
        return item in self._values
```


It should be relatively easy to fix with the following code:
```
    def __contains__(self, item):
        if hasattr(self, "_fields"):
            return item in self._fields
        else:
            return item in self._values
```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35111) Modify the spelling mistakes in the taskmanager html

2024-04-15 Thread JJJJude (Jira)

ude created FLINK-35111:
---

 Summary: Modify the spelling mistakes in the taskmanager html
 Key: FLINK-35111
 URL: https://issues.apache.org/jira/browse/FLINK-35111
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Affects Versions: 1.19.0
Reporter: ude
 Fix For: 1.19.0


Fix the spelling error from "profiler"  to "profiling"



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35110) Modify the spelling mistakes in the taskmanager html

2024-04-15 Thread JJJJude (Jira)

ude created FLINK-35110:
---

 Summary: Modify the spelling mistakes in the taskmanager html
 Key: FLINK-35110
 URL: https://issues.apache.org/jira/browse/FLINK-35110
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Affects Versions: 1.19.0
Reporter: ude
 Fix For: 1.19.0


Fix the spelling error from "profiler"  to "profiling"



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread Jacky Lau

Congratulations, Lincoln!

Best,
Jacky Lau

Jinzhong Li  于2024年4月15日周一 15:45写道：

> Congratulations, Lincoln!
>
> Best,
> Jinzhong Li
>
> On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu  wrote:
>
> > Congratulations, Lincoln!
> >
> > On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan 
> > wrote:
> >
> > > Congratulations, Lincoln!
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky
>  > >
> > > wrote:
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Ferenc
> > > >
> > > >
> > > >
> > > >
> > > > On Friday, April 12th, 2024 at 15:54, lorenzo.affe...@ververica.com
> > > .INVALID
> > > >  wrote:
> > > >
> > > > >
> > > > >
> > > > > Huge congrats! Well done!
> > > > > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com, wrote:
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > > Best,
> > > > > > Ron
> > > > > >
> > > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > Best,
> > > > > > > Junrui
> > > > > > >
> > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:29写道：
> > > > > > >
> > > > > > > > > Congratulations, Lincoln!
> > > > > > > > >
> > > > > > > > > Best Regards
> > > > > > > > > Aleksandr
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-15 Thread Jacky Lau

Congratulations, Jing!

Best,
Jacky Lau

[jira] [Created] (FLINK-35109) Drop support for Flink 1.17 and 1.18 in Flink Kafka connector

2024-04-15 Thread Martijn Visser (Jira)

Martijn Visser created FLINK-35109:
--

 Summary: Drop support for Flink 1.17 and 1.18 in Flink Kafka 
connector
 Key: FLINK-35109
 URL: https://issues.apache.org/jira/browse/FLINK-35109
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Kafka
Reporter: Martijn Visser
 Fix For: kafka-4.0.0


The Flink Kafka connector currently can't compile against Flink 1.20-SNAPSHOT. 
An example failure can be found at 
https://github.com/apache/flink-connector-kafka/actions/runs/8659822490/job/23746484721#step:15:169

The {code:java} TypeSerializerUpgradeTestBase{code} has had issues before, see 
FLINK-32455. See also specifically the comment in 
https://issues.apache.org/jira/browse/FLINK-32455?focusedCommentId=17739785=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17739785

Next to that, there's also FLINK-25509 which can only be supported with Flink 
1.19 and higher. 

So we should:
* Drop support for 1.17 and 1.18
* Refactor the Flink Kafka connector to use the new 
{code:java}MigrationTest{code}

We will support the Flink Kafka connector for Flink 1.18 via the v3.1 branch; 
this change will be a new v4.0 version with support for Flink 1.19 and the 
upcoming Flink 1.20



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-399: Flink Connector Doris

2024-04-15 Thread Samrat Deb

+1 (non binding)

On Mon, 15 Apr 2024 at 2:16 PM, Aleksandr Pilipenko 
wrote:

> +1 (non-binding),
>
> Best,
> Aleksandr
>
> On Mon, 15 Apr 2024 at 03:03, Yanquan Lv  wrote:
>
> > +1 (non-binding), thanks for it.
> >
> > wudi <676366...@qq.com.invalid> 于2024年4月9日周二 10:48写道：
> >
> > > Hi devs,
> > >
> > > I would like to start a vote about FLIP-399 [1]. The FLIP is about
> > > contributing the Flink Doris Connector[2] to the Flink community.
> > > Discussion thread [3].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection
> > or
> > > insufficient votes.
> > >
> > >
> > > Thanks,
> > > Di.Wu
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-399%3A+Flink+Connector+Doris
> > > [2] https://github.com/apache/doris-flink-connector
> > > [3] https://lists.apache.org/thread/p3z4wsw3ftdyfs9p2wd7bbr2gfyl3xnh
> > >
> > >
> >
>

Re: [VOTE] FLIP-399: Flink Connector Doris

2024-04-15 Thread Aleksandr Pilipenko

+1 (non-binding),

Best,
Aleksandr

On Mon, 15 Apr 2024 at 03:03, Yanquan Lv  wrote:

> +1 (non-binding), thanks for it.
>
> wudi <676366...@qq.com.invalid> 于2024年4月9日周二 10:48写道：
>
> > Hi devs,
> >
> > I would like to start a vote about FLIP-399 [1]. The FLIP is about
> > contributing the Flink Doris Connector[2] to the Flink community.
> > Discussion thread [3].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > insufficient votes.
> >
> >
> > Thanks,
> > Di.Wu
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-399%3A+Flink+Connector+Doris
> > [2] https://github.com/apache/doris-flink-connector
> > [3] https://lists.apache.org/thread/p3z4wsw3ftdyfs9p2wd7bbr2gfyl3xnh
> >
> >
>

[jira] [Created] (FLINK-35108) Deployment recovery is triggered on terminal jobs after jm shutdown ttl

2024-04-15 Thread Gyula Fora (Jira)

Gyula Fora created FLINK-35108:
--

 Summary: Deployment recovery is triggered on terminal jobs after 
jm shutdown ttl
 Key: FLINK-35108
 URL: https://issues.apache.org/jira/browse/FLINK-35108
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.8.0, kubernetes-operator-1.7.0
Reporter: Gyula Fora
Assignee: Gyula Fora


The deployment recovery mechanism is incorrectly triggered for terminal jobs 
once the JM deployment is deleted after the TTL period. 

This causes jobs to be resubmitted. This affects only batch jobs.

The workaround is to set 
kubernetes.operator.jm-deployment-recovery.enabled: false

 for batch jobs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Jing Ge

Congratulations Zakelly!

Best regards,
Jing

On Mon, Apr 15, 2024 at 4:26 PM Xia Sun  wrote:

> Congratulations Zakelly!
>
>  Best,
>  Xia
>
> Leonard Xu  于2024年4月15日周一 16:16写道：
>
> > Congratulations Zakelly!
> >
> >
> > Best,
> > Leonard
> > > 2024年4月15日 下午3:56，Samrat Deb  写道：
> > >
> > > Congratulations Zakelly!
> >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Danny Cranmer

Congrats Zakelly!

Danny.

On Mon, Apr 15, 2024 at 9:26 AM Xia Sun  wrote:

> Congratulations Zakelly!
>
>  Best,
>  Xia
>
> Leonard Xu  于2024年4月15日周一 16:16写道：
>
> > Congratulations Zakelly!
> >
> >
> > Best,
> > Leonard
> > > 2024年4月15日 下午3:56，Samrat Deb  写道：
> > >
> > > Congratulations Zakelly!
> >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Xia Sun

Congratulations Zakelly!

 Best,
 Xia

Leonard Xu  于2024年4月15日周一 16:16写道：

> Congratulations Zakelly!
>
>
> Best,
> Leonard
> > 2024年4月15日 下午3:56，Samrat Deb  写道：
> >
> > Congratulations Zakelly!
>
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Leonard Xu

Congratulations Zakelly!


Best,
Leonard
> 2024年4月15日 下午3:56，Samrat Deb  写道：
> 
> Congratulations Zakelly!

[jira] [Created] (FLINK-35107) rename flink-connector-datagen-test module folder to flink-connector-datagen-tests

2024-04-15 Thread xleoken (Jira)

xleoken created FLINK-35107:
---

 Summary: rename flink-connector-datagen-test module folder to 
flink-connector-datagen-tests
 Key: FLINK-35107
 URL: https://issues.apache.org/jira/browse/FLINK-35107
 Project: Flink
  Issue Type: Improvement
Reporter: xleoken






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Samrat Deb

Congratulations Zakelly!

 Bests,
 Samrat

On Mon, 15 Apr 2024 at 1:19 PM, Sergey Nuyanzin  wrote:

> Congratulations,  Zakelly Lan!
>
> On Mon, Apr 15, 2024 at 9:46 AM Jinzhong Li 
> wrote:
>
> > Congratulations, Zakelly!
> >
> > Best,
> > Jinzhong
> >
> > On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu 
> wrote:
> >
> > > Congratulations, Zakelly!
> > >
> > > On Mon, Apr 15, 2024 at 1:58 PM Yun Tang  wrote:
> > >
> > > > Congratulations, Zakelly!
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Yanquan Lv 
> > > > Sent: Monday, April 15, 2024 13:23
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan
> > > >
> > > > Congratulations, Zakelly!
> > > >
> > > > Best,
> > > > YanQuan
> > > >
> > > > Yuan Mei  于2024年4月15日周一 10:51写道：
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > On behalf of the PMC, I'm happy to let you know that Zakelly Lan
> has
> > > > become
> > > > > a new Flink Committer!
> > > > >
> > > > > Zakelly has been continuously contributing to the Flink project
> since
> > > > 2020,
> > > > > with a focus area on Checkpointing, State as well as frocksdb (the
> > > > default
> > > > > on-disk state db).
> > > > >
> > > > > He leads several FLIPs to improve checkpoints and state APIs,
> > including
> > > > > File Merging for Checkpoints and configuration/API reorganizations.
> > He
> > > is
> > > > > also one of the main contributors to the recent efforts of
> > > "disaggregated
> > > > > state management for Flink 2.0" and drives the entire discussion in
> > the
> > > > > mailing thread, demonstrating outstanding technical depth and
> breadth
> > > of
> > > > > knowledge.
> > > > >
> > > > > Beyond his technical contributions, Zakelly is passionate about
> > helping
> > > > the
> > > > > community in numerous ways. He spent quite some time setting up the
> > > Flink
> > > > > Speed Center and rebuilding the benchmark pipeline after the
> original
> > > one
> > > > > was out of lease. He helps build frocksdb and tests for the
> upcoming
> > > > > frocksdb release (bump rocksdb from 6.20.3->8.10).
> > > > >
> > > > > Please join me in congratulating Zakelly for becoming an Apache
> Flink
> > > > > committer!
> > > > >
> > > > > Best,
> > > > > Yuan (on behalf of the Flink PMC)
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Hangxiang.
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: Question around Flink's AdaptiveBatchScheduler

2024-04-15 Thread Xia Sun

Hi Venkat,
I agree that the parallelism of source vertex should not be upper bounded
by the job's global max parallelism. The case you mentioned, >> High filter
selectivity with huge amounts of data to read  excellently supports this
viewpoint. (In fact, in the current implementation, if the source
parallelism is pre-specified at job create stage, rather than relying on
the dynamic parallelism inference of the AdaptiveBatchScheduler, the source
vertex's parallelism can indeed exceed the job's global max parallelism.)

As Lijie and Junrui pointed out, the key issue is "semantic consistency."
Currently, if a vertex has not set maxParallelism, the
AdaptiveBatchScheduler will use
`execution.batch.adaptive.auto-parallelism.max-parallelism` as the vertex's
maxParallelism. Since the current implementation does not distinguish
between source vertices and downstream vertices, source vertices are also
subject to this limitation.

Therefore, I believe that if the issue of "semantic consistency" can be
well explained in the code and configuration documentation, the
AdaptiveBatchScheduler should support that the parallelism of source
vertices can exceed the job's global max parallelism.

Best,
Xia

Venkatakrishnan Sowrirajan  于2024年4月14日周日 10:31写道：

> Let me state why I think "*jobmanager.adaptive-batch-sche*
> *duler.default-source-parallelism*" should not be bound by the "
> *jobmanager.adaptive-batch-sche**duler.max-parallelism*".
>
>- Source vertex is unique and does not have any upstream vertices
>- Downstream vertices read shuffled data partitioned by key, which is
>not the case for the Source vertex
>- Limiting source parallelism by downstream vertices' max parallelism is
>incorrect
>
> If we say for ""semantic consistency" the source vertex parallelism has to
> be bound by the overall job's max parallelism, it can lead to following
> issues:
>
>- High filter selectivity with huge amounts of data to read - setting
>high "*jobmanager.adaptive-batch-scheduler.max-parallelism*" so that
>source parallelism can be set higher can lead to small blocks and
>sub-optimal performance.
>- Setting high "*jobmanager.adaptive-batch-scheduler.max-parallelism*"
>requires careful tuning of network buffer configurations which is
>unnecessary in cases where it is not required just so that the source
>parallelism can be set high.
>
> Regards
> Venkata krishnan
>
> On Thu, Apr 11, 2024 at 9:30 PM Junrui Lee  wrote:
>
> > Hello Venkata krishnan,
> >
> > I think the term "semantic inconsistency" defined by
> > jobmanager.adaptive-batch-scheduler.max-parallelism refers to
> maintaining a
> > uniform upper limit on parallelism across all vertices within a job. As
> the
> > source vertices are part of the global execution graph, they should also
> > respect this rule to ensure consistent application of parallelism
> > constraints.
> >
> > Best,
> > Junrui
> >
> > Venkatakrishnan Sowrirajan  于2024年4月12日周五 02:10写道：
> >
> > > Gentle bump on this question. cc @Becket Qin  as
> > > well.
> > >
> > > Regards
> > > Venkata krishnan
> > >
> > >
> > > On Tue, Mar 12, 2024 at 10:11 PM Venkatakrishnan Sowrirajan <
> > > vsowr...@asu.edu> wrote:
> > >
> > > > Thanks for the response Lijie and Junrui. Sorry for the late reply.
> Few
> > > > follow up questions.
> > > >
> > > > > Source can actually ignore this limit
> > > > because it has no upstream, but this will lead to semantic
> > inconsistency.
> > > >
> > > > Lijie, can you please elaborate on the above comment further? What do
> > you
> > > > mean when you say it will lead to "semantic inconsistency"?
> > > >
> > > > > Secondly, we first need to limit the max parallelism of
> (downstream)
> > > > vertex, and then we can decide how many subpartitions (upstream
> vertex)
> > > > should produce. The limit should be effective, otherwise some
> > downstream
> > > > tasks will have no data to process.
> > > >
> > > > This makes sense in the context of any other vertices other than the
> > > > source vertex. As you mentioned above ("Source can actually ignore
> this
> > > > limit because it has no upstream"), therefore I feel "
> > > > jobmanager.adaptive-batch-scheduler.default-source-parallelism" need
> > not
> > > > be upper bounded by
> > > "jobmanager.adaptive-batch-scheduler.max-parallelism".
> > > >
> > > > Regards
> > > > Venkata krishnan
> > > >
> > > >
> > > > On Thu, Feb 29, 2024 at 2:11 AM Junrui Lee 
> > wrote:
> > > >
> > > >> Hi Venkat,
> > > >>
> > > >> As Lijie mentioned,  in Flink, the parallelism is required to be
> less
> > > than
> > > >> or equal to the maximum parallelism. The config option
> > > >> jobmanager.adaptive-batch-scheduler.max-parallelism and
> > > >> jobmanager.adaptive-batch-scheduler.default-source-parallelism will
> be
> > > set
> > > >> as the source's parallelism and max-parallelism, respectively.
> > > Therefore,
> > > >> the check failed situation you encountered is in line with the
> > > >> expectations.

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Sergey Nuyanzin

Congratulations,  Zakelly Lan!

On Mon, Apr 15, 2024 at 9:46 AM Jinzhong Li 
wrote:

> Congratulations, Zakelly!
>
> Best,
> Jinzhong
>
> On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu  wrote:
>
> > Congratulations, Zakelly!
> >
> > On Mon, Apr 15, 2024 at 1:58 PM Yun Tang  wrote:
> >
> > > Congratulations, Zakelly!
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Yanquan Lv 
> > > Sent: Monday, April 15, 2024 13:23
> > > To: dev@flink.apache.org 
> > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan
> > >
> > > Congratulations, Zakelly!
> > >
> > > Best,
> > > YanQuan
> > >
> > > Yuan Mei  于2024年4月15日周一 10:51写道：
> > >
> > > > Hi everyone,
> > > >
> > > > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has
> > > become
> > > > a new Flink Committer!
> > > >
> > > > Zakelly has been continuously contributing to the Flink project since
> > > 2020,
> > > > with a focus area on Checkpointing, State as well as frocksdb (the
> > > default
> > > > on-disk state db).
> > > >
> > > > He leads several FLIPs to improve checkpoints and state APIs,
> including
> > > > File Merging for Checkpoints and configuration/API reorganizations.
> He
> > is
> > > > also one of the main contributors to the recent efforts of
> > "disaggregated
> > > > state management for Flink 2.0" and drives the entire discussion in
> the
> > > > mailing thread, demonstrating outstanding technical depth and breadth
> > of
> > > > knowledge.
> > > >
> > > > Beyond his technical contributions, Zakelly is passionate about
> helping
> > > the
> > > > community in numerous ways. He spent quite some time setting up the
> > Flink
> > > > Speed Center and rebuilding the benchmark pipeline after the original
> > one
> > > > was out of lease. He helps build frocksdb and tests for the upcoming
> > > > frocksdb release (bump rocksdb from 6.20.3->8.10).
> > > >
> > > > Please join me in congratulating Zakelly for becoming an Apache Flink
> > > > committer!
> > > >
> > > > Best,
> > > > Yuan (on behalf of the Flink PMC)
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
> >
>


-- 
Best regards,
Sergey

[jira] [Created] (FLINK-35106) Kubernetes Operator ignores checkpoint type configuration

2024-04-15 Thread Mate Czagany (Jira)

Mate Czagany created FLINK-35106:


 Summary: Kubernetes Operator ignores checkpoint type configuration
 Key: FLINK-35106
 URL: https://issues.apache.org/jira/browse/FLINK-35106
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.8.0
Reporter: Mate Czagany


There is a configuration for checkpoint type that will be taken if perioid 
checkpointing is enabled or a manual checkpoint is triggered.

However, the configuration value `kubernetes.operator.checkpoint.type` is 
completely ignored when any checkpoint is triggered.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Jinzhong Li

Congratulations, Zakelly!

Best,
Jinzhong

On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu  wrote:

> Congratulations, Zakelly!
>
> On Mon, Apr 15, 2024 at 1:58 PM Yun Tang  wrote:
>
> > Congratulations, Zakelly!
> >
> > Best
> > Yun Tang
> > 
> > From: Yanquan Lv 
> > Sent: Monday, April 15, 2024 13:23
> > To: dev@flink.apache.org 
> > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan
> >
> > Congratulations, Zakelly!
> >
> > Best,
> > YanQuan
> >
> > Yuan Mei  于2024年4月15日周一 10:51写道：
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has
> > become
> > > a new Flink Committer!
> > >
> > > Zakelly has been continuously contributing to the Flink project since
> > 2020,
> > > with a focus area on Checkpointing, State as well as frocksdb (the
> > default
> > > on-disk state db).
> > >
> > > He leads several FLIPs to improve checkpoints and state APIs, including
> > > File Merging for Checkpoints and configuration/API reorganizations. He
> is
> > > also one of the main contributors to the recent efforts of
> "disaggregated
> > > state management for Flink 2.0" and drives the entire discussion in the
> > > mailing thread, demonstrating outstanding technical depth and breadth
> of
> > > knowledge.
> > >
> > > Beyond his technical contributions, Zakelly is passionate about helping
> > the
> > > community in numerous ways. He spent quite some time setting up the
> Flink
> > > Speed Center and rebuilding the benchmark pipeline after the original
> one
> > > was out of lease. He helps build frocksdb and tests for the upcoming
> > > frocksdb release (bump rocksdb from 6.20.3->8.10).
> > >
> > > Please join me in congratulating Zakelly for becoming an Apache Flink
> > > committer!
> > >
> > > Best,
> > > Yuan (on behalf of the Flink PMC)
> > >
> >
>
>
> --
> Best,
> Hangxiang.
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-15 Thread Jinzhong Li

Congratulations, Jing!

Best,
Jinzhong Li

On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu  wrote:

> Congratulations, Jing!
>
> On Mon, Apr 15, 2024 at 11:11 AM spoon_lz  wrote:
>
> >
> >
> > Congratulations!
> >
> > Best,
> > lz
> >
> > | |
> > spoon_lz
> > |
> > |
> > spoon...@126.com
> > |
> >
> >
> >  Replied Message 
> > | From | Biao Geng |
> > | Date | 04/15/2024 10:40 |
> > | To |  |
> > | Subject | Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge |
> > Congrats, Jing!
> >
> > Best,
> > Biao Geng
> >
> > Zakelly Lan  于2024年4月15日周一 10:17写道：
> >
> > Congratulations, Jing!
> >
> >
> > Best,
> > Zakelly
> >
> > On Sat, Apr 13, 2024 at 12:47 AM Ferenc Csaky  >
> > wrote:
> >
> > Congratulations, Jing!
> >
> > Best,
> > Ferenc
> >
> >
> >
> > On Friday, April 12th, 2024 at 13:54, Ron liu 
> > wrote:
> >
> >
> >
> > Congratulations, Jing!
> >
> > Best,
> > Ron
> >
> > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> >
> > Congratulations, Jing!
> >
> > Best,
> > Junrui
> >
> > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:28写道：
> >
> > Congratulations, Jing!
> >
> > Best Regards,
> > Aleksandr
> >
> >
> >
>
> --
> Best,
> Hangxiang.
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Ahmed Hamdy

Congratulations, Zakelly
Best Regards
Ahmed Hamdy


On Mon, 15 Apr 2024 at 07:56, Hangxiang Yu  wrote:

> Congratulations, Zakelly!
>
> On Mon, Apr 15, 2024 at 1:58 PM Yun Tang  wrote:
>
> > Congratulations, Zakelly!
> >
> > Best
> > Yun Tang
> > 
> > From: Yanquan Lv 
> > Sent: Monday, April 15, 2024 13:23
> > To: dev@flink.apache.org 
> > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan
> >
> > Congratulations, Zakelly!
> >
> > Best,
> > YanQuan
> >
> > Yuan Mei  于2024年4月15日周一 10:51写道：
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has
> > become
> > > a new Flink Committer!
> > >
> > > Zakelly has been continuously contributing to the Flink project since
> > 2020,
> > > with a focus area on Checkpointing, State as well as frocksdb (the
> > default
> > > on-disk state db).
> > >
> > > He leads several FLIPs to improve checkpoints and state APIs, including
> > > File Merging for Checkpoints and configuration/API reorganizations. He
> is
> > > also one of the main contributors to the recent efforts of
> "disaggregated
> > > state management for Flink 2.0" and drives the entire discussion in the
> > > mailing thread, demonstrating outstanding technical depth and breadth
> of
> > > knowledge.
> > >
> > > Beyond his technical contributions, Zakelly is passionate about helping
> > the
> > > community in numerous ways. He spent quite some time setting up the
> Flink
> > > Speed Center and rebuilding the benchmark pipeline after the original
> one
> > > was out of lease. He helps build frocksdb and tests for the upcoming
> > > frocksdb release (bump rocksdb from 6.20.3->8.10).
> > >
> > > Please join me in congratulating Zakelly for becoming an Apache Flink
> > > committer!
> > >
> > > Best,
> > > Yuan (on behalf of the Flink PMC)
> > >
> >
>
>
> --
> Best,
> Hangxiang.
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread Jinzhong Li

Congratulations, Lincoln!

Best,
Jinzhong Li

On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu  wrote:

> Congratulations, Lincoln!
>
> On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan 
> wrote:
>
> > Congratulations, Lincoln!
> >
> >
> > Best,
> > Zakelly
> >
> > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky  >
> > wrote:
> >
> > > Congratulations, Lincoln!
> > >
> > > Best,
> > > Ferenc
> > >
> > >
> > >
> > >
> > > On Friday, April 12th, 2024 at 15:54, lorenzo.affe...@ververica.com
> > .INVALID
> > >  wrote:
> > >
> > > >
> > > >
> > > > Huge congrats! Well done!
> > > > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com, wrote:
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Best,
> > > > > Ron
> > > > >
> > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > > Best,
> > > > > > Junrui
> > > > > >
> > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:29写道：
> > > > > >
> > > > > > > > Congratulations, Lincoln!
> > > > > > > >
> > > > > > > > Best Regards
> > > > > > > > Aleksandr
> > >
> >
>
>
> --
> Best,
> Hangxiang.
>

[jira] [Created] (FLINK-35105) Support setting default Autoscaler options at autoscaler standalone level

2024-04-15 Thread Rui Fan (Jira)

Rui Fan created FLINK-35105:
---

 Summary: Support setting default Autoscaler options at autoscaler 
standalone level
 Key: FLINK-35105
 URL: https://issues.apache.org/jira/browse/FLINK-35105
 Project: Flink
  Issue Type: Sub-task
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.9.0


Currently, autoscaler standalone doesn't support set [autoscaler 
options|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.8/docs/operations/configuration/#autoscaler-configuration].
 We must set them at job level when we use autoscaler standalone. It's not 
convenient if platform administrator wanna change the default value for some 
autoscaler options, such as:
 * job.autoscaler.enabled
 * job.autoscaler.metrics.window
 * etc

This Jira supports setting Autoscaler options at autoscaler standalone level, 
it's similar with flink kubernetes operator.

The  autoscaler options of autoscaler standalone will be as the base 
configuration, and the configuration at job-level can override the default 
value provided in the autoscaler standalone.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35104) Add kafka pipeline data source connector

2024-04-15 Thread melin (Jira)

melin created FLINK-35104:
-

 Summary: Add kafka pipeline data source connector
 Key: FLINK-35104
 URL: https://issues.apache.org/jira/browse/FLINK-35104
 Project: Flink
  Issue Type: New Feature
  Components: Flink CDC
Reporter: melin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35103) [Plugin] Enhancing Flink Failure Management in Kubernetes with Dynamic Termination Log Integration

2024-04-15 Thread SwathiChandrashekar (Jira)

SwathiChandrashekar created FLINK-35103:
---

 Summary: [Plugin] Enhancing Flink Failure Management in Kubernetes 
with Dynamic Termination Log Integration
 Key: FLINK-35103
 URL: https://issues.apache.org/jira/browse/FLINK-35103
 Project: Flink
  Issue Type: Improvement
  Components: API / Core
Reporter: SwathiChandrashekar
 Fix For: 1.20.0
 Attachments: Status-pod.png

Currently, whenever we have flink failures, we need to manually do the triaging 
by looking into the flink logs even for the initial analysis. It would have 
been better, if the user/admin directly gets the initial failure information 
even before looking into the logs.

To address this, we've developed a comprehensive solution via a plugin aimed at 
helping fetch the Flink failures, ensuring critical data is preserved for 
subsequent analysis and action.

 

In Kubernetes environments, troubleshooting pod failures can be challenging 
without checking the pod/flink logs. Fortunately, Kubernetes offers a robust 
mechanism to enhance debugging capabilities by leveraging the 
/dev/termination-log file.

[https://kubernetes.io/docs/tasks/debug/debug-application/determine-reason-pod-failure/]

By writing failure information to this log, Kubernetes automatically 
incorporates it into the container status, providing administrators and 
developers with valuable insights into the root cause of failures.

Our solution capitalizes on this Kubernetes feature to seamlessly integrate 
Flink failure reporting within the container ecosystem. Whenever a Flink 
encounters an issue, our plugin dynamically captures and logs the pertinent 
failure information into the /dev/termination-log file. This ensures that 
Kubernetes recognizes and propagates the failure status throughout the 
container ecosystem, enabling efficient monitoring and response mechanisms.

By leveraging Kubernetes' native functionality in this manner, our plugin 
ensures that Flink failure incidents are promptly identified and reflected in 
the pod status. This technical integration streamlines the debugging process, 
empowering operators to swiftly diagnose and address issues, thereby minimizing 
downtime and maximizing system reliability.

 

In-order to make this plugin generic, by default it doesn't do any action.  We 
can configure this by using

*external.log.factory.class : 
org.apache.flink.externalresource.log.K8SSupportTerminationLog*

This will be present in the plugins directory
PFA for the pod status

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-15 Thread Hangxiang Yu

Congratulations, Jing!

On Mon, Apr 15, 2024 at 11:11 AM spoon_lz  wrote:

>
>
> Congratulations!
>
> Best,
> lz
>
> | |
> spoon_lz
> |
> |
> spoon...@126.com
> |
>
>
>  Replied Message 
> | From | Biao Geng |
> | Date | 04/15/2024 10:40 |
> | To |  |
> | Subject | Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge |
> Congrats, Jing!
>
> Best,
> Biao Geng
>
> Zakelly Lan  于2024年4月15日周一 10:17写道：
>
> Congratulations, Jing!
>
>
> Best,
> Zakelly
>
> On Sat, Apr 13, 2024 at 12:47 AM Ferenc Csaky 
> wrote:
>
> Congratulations, Jing!
>
> Best,
> Ferenc
>
>
>
> On Friday, April 12th, 2024 at 13:54, Ron liu 
> wrote:
>
>
>
> Congratulations, Jing!
>
> Best,
> Ron
>
> Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
>
> Congratulations, Jing!
>
> Best,
> Junrui
>
> Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:28写道：
>
> Congratulations, Jing!
>
> Best Regards,
> Aleksandr
>
>
>

-- 
Best,
Hangxiang.

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-15 Thread Hangxiang Yu

Congratulations, Lincoln!

On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan  wrote:

> Congratulations, Lincoln!
>
>
> Best,
> Zakelly
>
> On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky 
> wrote:
>
> > Congratulations, Lincoln!
> >
> > Best,
> > Ferenc
> >
> >
> >
> >
> > On Friday, April 12th, 2024 at 15:54, lorenzo.affe...@ververica.com
> .INVALID
> >  wrote:
> >
> > >
> > >
> > > Huge congrats! Well done!
> > > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com, wrote:
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Ron
> > > >
> > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Best,
> > > > > Junrui
> > > > >
> > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:29写道：
> > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > Best Regards
> > > > > > > Aleksandr
> >
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-15 Thread Hangxiang Yu

Congratulations, Zakelly!

On Mon, Apr 15, 2024 at 1:58 PM Yun Tang  wrote:

> Congratulations, Zakelly!
>
> Best
> Yun Tang
> 
> From: Yanquan Lv 
> Sent: Monday, April 15, 2024 13:23
> To: dev@flink.apache.org 
> Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan
>
> Congratulations, Zakelly!
>
> Best,
> YanQuan
>
> Yuan Mei  于2024年4月15日周一 10:51写道：
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has
> become
> > a new Flink Committer!
> >
> > Zakelly has been continuously contributing to the Flink project since
> 2020,
> > with a focus area on Checkpointing, State as well as frocksdb (the
> default
> > on-disk state db).
> >
> > He leads several FLIPs to improve checkpoints and state APIs, including
> > File Merging for Checkpoints and configuration/API reorganizations. He is
> > also one of the main contributors to the recent efforts of "disaggregated
> > state management for Flink 2.0" and drives the entire discussion in the
> > mailing thread, demonstrating outstanding technical depth and breadth of
> > knowledge.
> >
> > Beyond his technical contributions, Zakelly is passionate about helping
> the
> > community in numerous ways. He spent quite some time setting up the Flink
> > Speed Center and rebuilding the benchmark pipeline after the original one
> > was out of lease. He helps build frocksdb and tests for the upcoming
> > frocksdb release (bump rocksdb from 6.20.3->8.10).
> >
> > Please join me in congratulating Zakelly for becoming an Apache Flink
> > committer!
> >
> > Best,
> > Yuan (on behalf of the Flink PMC)
> >
>


-- 
Best,
Hangxiang.

63 matches

Mail list logo