Flink checkpoint timeout

2023-05-30 Thread Ethan T Yang
Hello all, We recently start to experience Checkpoint timeout randomly. Here are some background information 1. We are on Flink 1.13.1 2. We have been running these type of streaming jobs for years. When checkpoint succeeds, it only take a few seconds. After a week ago, we start to see random c

Re: [DISCUSS] Hive dialect shouldn't fall back to Flink's default dialect

2023-05-30 Thread liu ron
Thanks for your proposal. I even don't notice this fallback behavior, +1. Best, Ron Jingsong Li 于2023年5月30日周二 15:23写道: > +1, the fallback looks weird now, it is outdated. > > But, it is good to provide an option. I don't know if there are some > users who depend on this fallback. > > Best, > Ji

Interaction between idling sources and watermark alignment

2023-05-30 Thread Alexis Sarda-Espinosa
Hello, I see that, in Flink 1.17.1, watermark alignment will be supported (as beta) within a single source's splits and across different sources. I don't see this explicitly mentioned in the documentation, but I assume that the concept of "maximal drift" used for alignment also takes idleness into

Re: [SUMMARY] Flink 1.18 Release Sync 05/30/2023

2023-05-30 Thread Jing Ge via user
Thanks Qingsheng for driving it! @Devs As you might already be aware of, there are many externalizations and new releases of Flink connectors. Once a connector has been externalized successfully, i.e. the related module has been removed in the Flink repo, we will not set a priority higher than maj

Re: [DISCUSS] Status of Statefun Project

2023-05-30 Thread Galen Warren via user
Getting to a resolution here would be great and much appreciated, yes. On Sat, May 27, 2023 at 1:03 AM Salva Alcántara wrote: > Hey Galen, > > I took a look at StateFun some time ago; not using it in production but I > agree that it would be a pity to abandon it. > > As Martijn said, let's be cl

[SUMMARY] Flink 1.18 Release Sync 05/30/2023

2023-05-30 Thread Qingsheng Ren
Hi devs and users, I'd like to share some highlights from the release sync of 1.18 on May 30. 1. @developers please update the progress of your features on 1.18 release wiki page [1] ! That will help us a lot to have an overview of the entire release cycle. 2. We found a JIRA issue (FLINK-18356)

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

2023-05-30 Thread Tamir Sagi
Hey Chesnay, I'm sending a follow up email regarding JDK 17 support. I see the Epic[1] is in progress and frequently updated. I'm curios if there is an ETA or any plan to release a major release with JDK 17 support that is not backward compatible. Thanks, Tamir [1] https://issues.apache.org/

Re: SupportsReadingMetadata flow between table transformations

2023-05-30 Thread Hang Ruan
Hi, Yuval. `SupportsReadingMetadata` make connectors be able to append metadata columns to the schema of table sources. But one operator does not know the columns map to which column in the source. We could try to generate this information when parsing Flink sql. But it is hard for DataStream API

Re: [DISCUSS] Hive dialect shouldn't fall back to Flink's default dialect

2023-05-30 Thread Jingsong Li
+1, the fallback looks weird now, it is outdated. But, it is good to provide an option. I don't know if there are some users who depend on this fallback. Best, Jingsong On Tue, May 30, 2023 at 1:47 PM Rui Li wrote: > > +1, the fallback was just intended as a temporary workaround to run > catal