Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Yunfeng Zhou
Hi Zakelly, Thanks for the quick response! > Actually splitting APIs into two sets ... warn them in runtime. Could you provide some hint on use cases where users need to mix sync and async state operations in spite of the performance regression? This information might help address our concerns

[jira] [Created] (FLINK-34651) The HiveTableSink of Flink does not support writing to S3

2024-03-11 Thread shizhengchao (Jira)
shizhengchao created FLINK-34651: Summary: The HiveTableSink of Flink does not support writing to S3 Key: FLINK-34651 URL: https://issues.apache.org/jira/browse/FLINK-34651 Project: Flink

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Feng Jin
+1 (non-binding) - Verified signatures and checksums - Verified that source does not contain binaries - Build source code successfully - Run a simple sql query successfully Best, Feng Jin On Tue, Mar 12, 2024 at 11:09 AM Ron liu wrote: > +1 (non binding) > > quickly verified: > - verified

Re: [DISCUSS] FLIP-434: Support optimizations for pre-partitioned data sources

2024-03-11 Thread Jim Hughes
Hi Jeyhun, I like the idea! Given FLIP-376[1], I wonder if it'd make sense to generalize FLIP-434 to be about "pre-divided" data to cover "buckets" and "partitions" (and maybe even situations where a data source is partitioned and bucketed). Separate from that, the page mentions TPC-H Q1 as an

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Ron liu
+1 (non binding) quickly verified: - verified that source distribution does not contain binaries - verified checksums - built source code successfully Best, Ron Jeyhun Karimov 于2024年3月12日周二 01:00写道: > +1 (non binding) > > - verified that source distribution does not contain binaries > -

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Yanfei Lei
+1 (non-binding) - Signature and checksum are OK - Source dist doesn't contain unexpected binaries - Build from source successfully, `SocketWindowWordCount` worked well Hangxiang Yu 于2024年3月12日周二 09:59写道: > > +1 (non-binding) > > - Verified signatures and checksums > - Reviewed Web PR > -

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Xuannan Su
Hi Zakelly, Thanks for the quick response. > It will be used in callback chaining cases where some branch within one > callback does nothing. I'm in favor of short phrases to express the > functionalities. Thus I suggest `completedVoidFuture` or `voidFuture`, WDTY? I'd prefer

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

2024-03-11 Thread Hangxiang Yu
Hi, Weijie. Thanks for your answer! > No, Introducing and declaring new state > at runtime is something we want to explicitly disallow. I just thinked about how some operators define their useState() when their real used states may be changed at runtime, e.g. different state types for different

[jira] [Created] (FLINK-34650) Migrate PushProjectIntoLegacyTableSourceScanRule

2024-03-11 Thread Jacky Lau (Jira)
Jacky Lau created FLINK-34650: - Summary: Migrate PushProjectIntoLegacyTableSourceScanRule Key: FLINK-34650 URL: https://issues.apache.org/jira/browse/FLINK-34650 Project: Flink Issue Type:

[jira] [Created] (FLINK-34649) Migrate PushFilterIntoLegacyTableSourceScanRule

2024-03-11 Thread Jacky Lau (Jira)
Jacky Lau created FLINK-34649: - Summary: Migrate PushFilterIntoLegacyTableSourceScanRule Key: FLINK-34649 URL: https://issues.apache.org/jira/browse/FLINK-34649 Project: Flink Issue Type:

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Hangxiang Yu
+1 (non-binding) - Verified signatures and checksums - Reviewed Web PR - Built from source successfully - Ran a wordcount job which worked well On Tue, Mar 12, 2024 at 1:00 AM Jeyhun Karimov wrote: > +1 (non binding) > > - verified that source distribution does not contain binaries > -

[jira] [Created] (FLINK-34648) Avoid RPC time when apply SchemaChangeEvent to external system

2024-03-11 Thread LvYanquan (Jira)
LvYanquan created FLINK-34648: - Summary: Avoid RPC time when apply SchemaChangeEvent to external system Key: FLINK-34648 URL: https://issues.apache.org/jira/browse/FLINK-34648 Project: Flink

[jira] [Created] (FLINK-34647) Path normalization is allocation intensive

2024-03-11 Thread David Schlosnagle (Jira)
David Schlosnagle created FLINK-34647: - Summary: Path normalization is allocation intensive Key: FLINK-34647 URL: https://issues.apache.org/jira/browse/FLINK-34647 Project: Flink Issue

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Jeyhun Karimov
+1 (non binding) - verified that source distribution does not contain binaries - verified signatures and checksums - built source code successfully Regards, Jeyhun On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb wrote: > +1 (non binding) > > - verified signatures and checksums > - ASF headers are

Re: [DISCUSS] FLIP-399: Flink Connector Doris

2024-03-11 Thread Ferenc Csaky
Hi, Thanks for driving this, +1 for the FLIP. Best, Ferenc On Monday, March 11th, 2024 at 15:17, Ahmed Hamdy wrote: > > > Hello, > Thanks for the proposal, +1 for the FLIP. > > Best Regards > Ahmed Hamdy > > > On Mon, 11 Mar 2024 at 15:12, wudi 676366...@qq.com.invalid wrote: > > >

Re: [DISCUSS] Add "Special Thanks" Page on the Flink Website

2024-03-11 Thread Maximilian Michels
Hi Jark, Thanks for clarifying. At first sight, such a page indicated general sponsorship. +1 for a Thank You page to list specific monetary contributions to the project for resources which are actively used or were actively used in the past. Cheers, Max On Fri, Mar 8, 2024 at 11:55 AM Martijn

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

2024-03-11 Thread Maximilian Michels
The FLIP mentions: "The contents described in this FLIP are all new APIs and do not involve compatibility issues." In this thread it looks like the plan is to remove the old state declaration API. I think we should consider keeping the old APIs to avoid breaking too many jobs. The new APIs will

[jira] [Created] (FLINK-34646) AggregateITCase.testDistinctWithRetract timed out

2024-03-11 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34646: - Summary: AggregateITCase.testDistinctWithRetract timed out Key: FLINK-34646 URL: https://issues.apache.org/jira/browse/FLINK-34646 Project: Flink Issue

[jira] [Created] (FLINK-34645) StreamArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount fails

2024-03-11 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34645: - Summary: StreamArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount fails Key: FLINK-34645 URL:

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

2024-03-11 Thread Maximilian Michels
+1 (binding) Max On Thu, Feb 29, 2024 at 4:24 AM Hang Ruan wrote: > > +1 (non-binding) > > Best, > Hang > > weijie guo 于2024年2月29日周四 09:55写道: > > > +1 (binding) > > > > Best regards, > > > > Weijie > > > > > > Feng Jin 于2024年2月29日周四 09:37写道: > > > > > +1 (non-binding) > > > > > > Best, > > >

[jira] [Created] (FLINK-34644) RestServerEndpointITCase.testShouldWaitForHandlersWhenClosing failed with ConnectionClosedException

2024-03-11 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34644: - Summary: RestServerEndpointITCase.testShouldWaitForHandlersWhenClosing failed with ConnectionClosedException Key: FLINK-34644 URL:

Re: [DISCUSS] FLIP-399: Flink Connector Doris

2024-03-11 Thread Ahmed Hamdy
Hello, Thanks for the proposal, +1 for the FLIP. Best Regards Ahmed Hamdy On Mon, 11 Mar 2024 at 15:12, wudi <676366...@qq.com.invalid> wrote: > Hi, Leonard > Thank you for your suggestion. > I referred to other Connectors[1], modified the naming and types of > relevant parameters[2], and also

[jira] [Created] (FLINK-34643) JobIDLoggingITCase failed

2024-03-11 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34643: - Summary: JobIDLoggingITCase failed Key: FLINK-34643 URL: https://issues.apache.org/jira/browse/FLINK-34643 Project: Flink Issue Type: Bug

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Samrat Deb
+1 (non binding) - verified signatures and checksums - ASF headers are present in all expected file - No unexpected binaries files found in the source - Build successful locally - tested basic word count example Bests, Samrat On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy wrote: > Hi Lincoln

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Ahmed Hamdy
Hi Lincoln +1 (non-binding) from me - Verified Checksums & Signatures - Verified Source dists don't contain binaries - Built source successfully - reviewed web PR Best Regards Ahmed Hamdy On Mon, 11 Mar 2024 at 15:18, Lincoln Lee wrote: > Hi Robin, > > Thanks for helping verifying the

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Lincoln Lee
Hi Robin, Thanks for helping verifying the release note[1], FLINK-14879 should not have been included, after confirming this I moved all unresolved non-blocker issues left over from 1.19.0 to 1.20.0 and reconfigured the release note [1]. Best, Lincoln Lee [1]

Re: [DISCUSS] FLIP-399: Flink Connector Doris

2024-03-11 Thread wudi
Hi, Leonard Thank you for your suggestion. I referred to other Connectors[1], modified the naming and types of relevant parameters[2], and also updated FLIP. [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/overview/ [1]

[jira] [Created] (FLINK-34641) Possibility to add Python Files from http/https sources in PythonDriver

2024-03-11 Thread Aleksei Perminov (Jira)
Aleksei Perminov created FLINK-34641: Summary: Possibility to add Python Files from http/https sources in PythonDriver Key: FLINK-34641 URL: https://issues.apache.org/jira/browse/FLINK-34641

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Robin Moffatt
Looking at the release notes [1] it lists `DESCRIBE DATABASE` (FLINK-14879) and `DESCRIBE CATALOG` (FLINK-14690). When I try these in 1.19 RC2 the behaviour is as in 1.18.1, i.e. it is not supported: ``` [INFO] Execute statement succeed. Flink SQL> show catalogs; +-+ |catalog

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-11 Thread Yanfei Lei
Hi Jane, Thanks for the perspective. We plan to adapt SQL operators in Milestone 2, so this part is not fully described. > Would you provide more details on how to adapt to the new API? Two interfaces similar to `setCurrentKey` will be provided: let's say preProcessKey()/postProcessKey()

[jira] [Created] (FLINK-34640) Replace DummyMetricGroup usage with UnregisteredMetricsGroup

2024-03-11 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-34640: Summary: Replace DummyMetricGroup usage with UnregisteredMetricsGroup Key: FLINK-34640 URL: https://issues.apache.org/jira/browse/FLINK-34640 Project: Flink

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Jean-Baptiste Onofré
+1 (non binding) I quickly checked: - signatures and checksums are OK - ASF headers are present in all expected file - No unexpected binaries files found in the source distribution - Build OK on the tag Thanks ! Regards JB On Thu, Mar 7, 2024 at 11:00 AM Lincoln Lee wrote: > > Hi everyone, > >

Re: [DISCUSS] FLIP Suggestion: Externalize Kudu Connector from Bahir

2024-03-11 Thread Ferenc Csaky
Hi Jing, Thank you for your comments! Updated the FLIP with reasoning on the proposed release versions and included them in the headline "Release" field. Best, Ferenc On Sunday, March 10th, 2024 at 16:59, Jing Ge wrote: > > > Hi Ferenc, > > Thanks for the proposal! +1 for it! > >

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-11 Thread Jane Chan
Hi Yanfei, Thanks for driving the discussion! I have a few questions from the SQL perspective (and may have some follow-ups). 1. Currently, there are some mini-batch optimizations[1][2] in SQL that update the state during the prepare snapshot phase, which will explicitly call setCurrentKey in

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Xintong Song
+1 (binding) - verified signature and checksum - verified that source distribution does not contain binaries - built from source and played with example jobs, everything looks fine - reviewed the release announcement PR Best, Xintong On Thu, Mar 7, 2024 at 6:01 PM Lincoln Lee wrote: > Hi

Re: [DISCUSS] FLIP-426: Grouping Remote State Access

2024-03-11 Thread Jinzhong Li
Hi jeyhun, Thanks for your feedback! >> Do we have a fallback mechanism for filesystems that do not support multiget? Currently, the rocksdb multiGet is available only for PosixFileSystem. And the multiGet requires the filesystem to support async read interfaces (e.g. MultiRead, ReadAsync,

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
Hi Xuannan, Thanks for your comments! 1. The name `emptyFuture` seems a little unintuitive, and it is hard > to understand in what use case the `emptyFuture` should be used. If I > understand correctly, it is similar to the > FutureUtils#completedVoidFuture. How about naming it >

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
Hi Yunfeng, Thanks for your comments! +1 for JingGe's suggestion to introduce an AsyncState API, instead of > having both get() and asyncGet() in the same State class. As a > supplement to its benefits, this design could help avoid having users > to use sync and async API in a mixed way (unless

[jira] [Created] (FLINK-34639) Flink CDC: Support DebeziumDeserializationSchema in OceanBase source connector

2024-03-11 Thread He Wang (Jira)
He Wang created FLINK-34639: --- Summary: Flink CDC: Support DebeziumDeserializationSchema in OceanBase source connector Key: FLINK-34639 URL: https://issues.apache.org/jira/browse/FLINK-34639 Project: Flink

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Xuannan Su
Hi Zakelly, Thank you for the proposal! Please find my comments below. 1. The name `emptyFuture` seems a little unintuitive, and it is hard to understand in what use case the `emptyFuture` should be used. If I understand correctly, it is similar to the FutureUtils#completedVoidFuture. How about

退订 unsubscribe

2024-03-11 Thread 95chenjz
- 原始邮件 - 发件人:ss zz 收件人:dev@flink.apache.org 主题:退订 日期:2024年03月11日 13点32分 退订

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Yunfeng Zhou
Hi Zakelly, Thanks for the proposal! The structure of the Async API generally looks good to me. Some comments on the details of the design are as follows. +1 for JingGe's suggestion to introduce an AsyncState API, instead of having both get() and asyncGet() in the same State class. As a

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

2024-03-11 Thread Jinzhong Li
Hi Jing. Thanks for your suggestion. >> I was wondering if SingleStateIterator and RocksDBRestoreOperation are exposed to users even if they are interfaces. I agree that very very few users implement these two interfaces, except for some advanced users who implement their own StateBackend

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-11 Thread Yanfei Lei
Hi Gyula, Thanks for your comments. "Record Context" is a concept similar to the current "Key Context"[1]. which is mutable. > If yes, does this mean that the context itself becomes "immutable" or the > context is switched in the background? "Record Context" is switched in the background, its

[DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

2024-03-11 Thread Ron liu
Hi, Dev Lincoln Lee and I would like to start a discussion about FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines. This FLIP is designed to simplify the development of data processing pipelines. With Dynamic Tables with uniform SQL statements and freshness, users can

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
Hi Jing, Thanks for your comments! Sorry for not making this clear. Actually these APIs are in some newly introduced classes, which are located in a different package name with the original ones. I suggest we name it "State API V2" and the package name will be