Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-30 Thread Lincoln Lee
Thanks Jingsong for the reminder! There is indeed some cleanup work to be done, I'll update the doc today. Best, Lincoln Lee Jingsong Li 于2022年5月31日周二 11:29写道: > Hi Lincoln, > > Some minor comments for FLIP document (Does not affect voting): > > - Maybe we can remove `The current processing

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-30 Thread Jingsong Li
Hi Lincoln, Some minor comments for FLIP document (Does not affect voting): - Maybe we can remove `The current processing flow chart` part, at first impression I thought this was part of `Proposed Changes`, it's right under `Proposed Changes`. - About `We tried two implementations`, I think we

Re: [VOTE] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-30 Thread Jingsong Li
+1 (binding) Thanks for driving. Best, Jingsong On Tue, May 31, 2022 at 11:05 AM Gen Luo wrote: > +1 (non-binding) > > On Mon, May 30, 2022 at 3:50 PM Jark Wu wrote: > > > +1 (binding) > > > > Best, > > Jark > > > > On Mon, 30 May 2022 at 15:40, Lincoln Lee > wrote: > > > > > Dear Flink

Re: [VOTE] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-30 Thread Gen Luo
+1 (non-binding) On Mon, May 30, 2022 at 3:50 PM Jark Wu wrote: > +1 (binding) > > Best, > Jark > > On Mon, 30 May 2022 at 15:40, Lincoln Lee wrote: > > > Dear Flink developers, > > > > Thanks for all your feedback for FLIP-232: Add Retry Support For Async > I/O > > In DataStream API[1] on the

[jira] [Created] (FLINK-27847) SchemaManager supports schema evolution

2022-05-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-27847: Summary: SchemaManager supports schema evolution Key: FLINK-27847 URL: https://issues.apache.org/jira/browse/FLINK-27847 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-27846) Schema evolution for data file reading

2022-05-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-27846: Summary: Schema evolution for data file reading Key: FLINK-27846 URL: https://issues.apache.org/jira/browse/FLINK-27846 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-27845) Introduce Column Type Schema evolution

2022-05-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-27845: Summary: Introduce Column Type Schema evolution Key: FLINK-27845 URL: https://issues.apache.org/jira/browse/FLINK-27845 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-27844) Schema evolution for ManifestFileMeta

2022-05-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-27844: Summary: Schema evolution for ManifestFileMeta Key: FLINK-27844 URL: https://issues.apache.org/jira/browse/FLINK-27844 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-27843) Schema evolution for data file meta

2022-05-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-27843: Summary: Schema evolution for data file meta Key: FLINK-27843 URL: https://issues.apache.org/jira/browse/FLINK-27843 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-30 Thread Xintong Song
#contribution-helps is meant for new contributors to ask non-technical questions. E.g., looking for starter issues or reviewers (just don't DM people). For having a #dev/development, I'm a little concerned this may encourage people to start discussions directly in slack. IMHO, we may want to

Re: Table API thrift support

2022-05-30 Thread Chen Qin
On Mon, May 30, 2022 at 7:35 AM Martijn Visser wrote: > Hi Chen, > > I think the best starting point would be to create a FLIP [1]. One of the > important topics from my point of view is to make sure that such changes > are not only available for SQL users, but are also being considered for >

[DISCUSS] FLIP-236: Asynchronous Job Submission

2022-05-30 Thread Afonso, Nuno
Hello everyone, I would like to start a discussion about FLIP-236: Asynchronous Job Submission [1]. During the processing of a job submission request, Flink runs the “main” method in a previously submitted JAR. This method may contain operations that take arbitrarily long to complete or

Re: Table API thrift support

2022-05-30 Thread yuxia
Thanks for raising it. But I wonder what do you mean by saying "dynamic overwrite schema type information loaded from HiveCatalog". For such case, what does the HiveCatalog store? Best regards, Yuxia - 原始邮件 - 发件人: "Chen Qin" 收件人: "dev" 发送时间: 星期一, 2022年 5 月 30日 上午 1:04:29 主题: Table

[jira] [Created] (FLINK-27842) Rename ndv to granularityNumber

2022-05-30 Thread Jing Ge (Jira)
Jing Ge created FLINK-27842: --- Summary: Rename ndv to granularityNumber Key: FLINK-27842 URL: https://issues.apache.org/jira/browse/FLINK-27842 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-30 Thread Gyula Fóra
Development channel (#dev/development) makes sense to me. It’s related to actual development related questions instead of the contrib process. Gyula On Mon, 30 May 2022 at 19:19, Jing Ge wrote: > Good idea, thanks! Is the channel #contribution-helps a good fit for it? Or > should we just

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-30 Thread Jing Ge
Good idea, thanks! Is the channel #contribution-helps a good fit for it? Or should we just rename it to #development? For me, development is a subset of contribution. Best regards, Jing On Mon, May 30, 2022 at 4:28 PM Robert Metzger wrote: > Thanks a lot for kicking this off. > > Is there a

Re: [DISCUSS] Releasing Flink 1.14.5

2022-05-30 Thread Jing Ge
Hi Xingbo, +1 Thanks for driving this. Best regards, Jing On Mon, May 30, 2022 at 4:51 PM Martijn Visser wrote: > Hi Xingbo, > > +1 to release Flink 1.14.5. > > Best regards, > > Martijn > > Op do 26 mei 2022 om 10:19 schreef Dian Fu : > > > Hi Xingbo, > > > > Thanks for driving this release.

Re: [DISCUSS] Planning Flink 1.16 with the community

2022-05-30 Thread Jing Ge
Thanks Robert for the reminder. Thanks Martijn for sharing the link. Is the feature freeze deadline on July 25 fixed or will it be adjusted accordingly? Best regards, Jing On Mon, May 30, 2022 at 4:56 PM Martijn Visser wrote: > Yes, we will have the release meeting tomorrow. Looking forward

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-05-30 Thread Alexander Smirnov
Hi Jingsong and devs! I agree that custom reloading would be very useful, so I changed recently proposed ReloadTime to customizable ReloadStrategy and its default realization FixedDelayReloadStrategy. I updated the FLIP, you can look at the new design [1]. From my point of view, the disadvantage

[jira] [Created] (FLINK-27841) RocksDB cache miss increase in 1.15

2022-05-30 Thread Feifan Wang (Jira)
Feifan Wang created FLINK-27841: --- Summary: RocksDB cache miss increase in 1.15 Key: FLINK-27841 URL: https://issues.apache.org/jira/browse/FLINK-27841 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-27839) org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.InterruptException

2022-05-30 Thread lijixiang (Jira)
lijixiang created FLINK-27839: - Summary: org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.InterruptException Key: FLINK-27839 URL: https://issues.apache.org/jira/browse/FLINK-27839 Project:

[jira] [Created] (FLINK-27840) org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.DisconnectException

2022-05-30 Thread lijixiang (Jira)
lijixiang created FLINK-27840: - Summary: org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.DisconnectException Key: FLINK-27840 URL: https://issues.apache.org/jira/browse/FLINK-27840 Project:

Re: [DISCUSS] Planning Flink 1.16 with the community

2022-05-30 Thread Martijn Visser
Yes, we will have the release meeting tomorrow. Looking forward to everyone who wants to participate. For those that are looking for the invite link, see https://cwiki.apache.org/confluence/display/FLINK/1.16+Release Best regards, Martijn Op ma 30 mei 2022 om 16:39 schreef Robert Metzger : > I

Re: [DISCUSS] Releasing Flink 1.14.5

2022-05-30 Thread Martijn Visser
Hi Xingbo, +1 to release Flink 1.14.5. Best regards, Martijn Op do 26 mei 2022 om 10:19 schreef Dian Fu : > Hi Xingbo, > > Thanks for driving this release. +1 for 1.14.5 as there are already nearly > 100 commits [1] since 1.14.4. > > Regards, > Dian > > [1]

Re: [DISCUSS] Planning Flink 1.16 with the community

2022-05-30 Thread Robert Metzger
I assume we'll have the next release planning meeting tomorrow? I'm also bringing this up also as a reminder for other folks who might be interested in joining. On Wed, May 11, 2022 at 5:55 AM Xintong Song wrote: > I'd like to kindly remind that, if someone adds / modifies release notes of > a

Re: Table API thrift support

2022-05-30 Thread Martijn Visser
Hi Chen, I think the best starting point would be to create a FLIP [1]. One of the important topics from my point of view is to make sure that such changes are not only available for SQL users, but are also being considered for Table API, DataStream and/or Python. There might be reasons why not

Re: [DISCUSS] Initializing Apache Flink Slack

2022-05-30 Thread Robert Metzger
Thanks a lot for kicking this off. Is there a reason why we haven't set up a #development channel yet? I have a short question that is more suitable for that channel, compared to the dev@ list ;) On Mon, May 23, 2022 at 7:52 AM Xintong Song wrote: > Hi Kyle, > > I've sent an invitation to

[jira] [Created] (FLINK-27838) Support timeout mechansim for executeStatement API

2022-05-30 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27838: - Summary: Support timeout mechansim for executeStatement API Key: FLINK-27838 URL: https://issues.apache.org/jira/browse/FLINK-27838 Project: Flink Issue

[jira] [Created] (FLINK-27837) Support statement set in the SQL Gateway

2022-05-30 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-27837: - Summary: Support statement set in the SQL Gateway Key: FLINK-27837 URL: https://issues.apache.org/jira/browse/FLINK-27837 Project: Flink Issue Type:

Re: About Native Deployment's Autoscaling implementation

2022-05-30 Thread Yang Wang
> > I thought we could enable Adaptive Scheduler, so adding or removing a task > manager is the same as restarting a job when we use an adaptive scheduler. > Do I miss anything ? It is true for standalone mode since adding/removing a TaskManager pod is fully controlled by users(or external

Re: [DISCUSS] FLIP-213: TaskManager's Flame Graphs

2022-05-30 Thread Alexander Fedulov
Hi Jacky, Some performance improvements for Flame Graphs related to how the stack traces are collected were recently merged [1]. Since you are working on a real-life environment with high parallelism, could you maybe run a couple of tests to verify that everything still works as expected after

[jira] [Created] (FLINK-27836) RocksDBMapState iteration may stop too early for var-length prefixes

2022-05-30 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-27836: --- Summary: RocksDBMapState iteration may stop too early for var-length prefixes Key: FLINK-27836 URL: https://issues.apache.org/jira/browse/FLINK-27836 Project: Flink

Re: [DISCUSS] Publish flink-end-to-end-tests-common to Maven

2022-05-30 Thread Jing Ge
Thanks for starting this discussion. I think it makes sense to make a clear definition and goal of ITCases and e2e tests. For me, compared to e2e tests, ITCases are more lightweight and only focus on some specific part of the e2e pipeline. E2e tests cover the whole pipeline and could be considered

Re: [DISCUSS] Publish flink-end-to-end-tests-common to Maven

2022-05-30 Thread Alexander Fedulov
Thanks for the clarifications. I agree then that it indeed makes sense to move the related classes into connector-test-utils for now and reconsider when and if concrete use cases of FlinkContainerTestEnvironment outside of the connectors' scope come up. On Wed, May 25, 2022 at 5:23 PM Chesnay

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

2022-05-30 Thread Piotr Nowojski
Hi Becket, Thanks for summing this up. Just one correction: > Piotr prefers option 2, his opinions are: > e) It is OK that the code itself in option 2 indicates the developers that a feature is optional. We will rely on the documentation to correct that and clarify that the feature is actually

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-05-30 Thread Jark Wu
Hi Jing Ge, What do you mean about the "impact on the block cache used by HBase"? In my understanding, the connector cache and HBase cache are totally two things. The connector cache is a local/client cache, and the HBase cache is a server cache. > does it make sense to have a no-cache solution

Re: [DISCUSS] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-05-30 Thread Lincoln Lee
Thanks Jark for your quick response and the consensus! And I will update the FLIP after Jingsong or other developers confirm that there is no problem. Best, Lincoln Lee Jark Wu 于2022年5月30日周一 15:49写道: > Thanks for the update. > > The unified lookup hint looks good to me. > And thanks for

Re: [VOTE] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-30 Thread Jark Wu
+1 (binding) Best, Jark On Mon, 30 May 2022 at 15:40, Lincoln Lee wrote: > Dear Flink developers, > > Thanks for all your feedback for FLIP-232: Add Retry Support For Async I/O > In DataStream API[1] on the discussion thread[2]. > > I'd like to start a vote for it. The vote will be open for at

Re: [DISCUSS] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-05-30 Thread Jark Wu
Thanks for the update. The unified lookup hint looks good to me. And thanks for explaining ALLOW_UNORDERED. Best, Jark On Mon, 30 May 2022 at 15:31, Lincoln Lee wrote: > Hi Jark & Jingsong, > > Thanks for your feedback! > > 1.) support retry on sync lookup > I also agree with supporting it,

[jira] [Created] (FLINK-27835) Introduce LeafPredicate interface and children method in Predicate

2022-05-30 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-27835: --- Summary: Introduce LeafPredicate interface and children method in Predicate Key: FLINK-27835 URL: https://issues.apache.org/jira/browse/FLINK-27835 Project: Flink

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-30 Thread Lincoln Lee
Hi everyone, I started a vote for this FLIP [1], please vote there or ask additional questions here. [2] [1] https://lists.apache.org/thread/qp9y1k0gldxymzmrso0xgsrwh15n6clc [2] https://lists.apache.org/thread/pgm3bf8vd5vqchlm29n6cro0gz4pbd3g Best, Lincoln Lee Jark Wu 于2022年5月27日周五 14:44写道:

[VOTE] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-30 Thread Lincoln Lee
Dear Flink developers, Thanks for all your feedback for FLIP-232: Add Retry Support For Async I/O In DataStream API[1] on the discussion thread[2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1]

Re: [DISCUSS] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-05-30 Thread Lincoln Lee
Hi Jark & Jingsong, Thanks for your feedback! 1.) support retry on sync lookup I also agree with supporting it, this will be useful for connectors that don't have asynchronous lookup implementations and can also solve the ASYNC non-target problem to some extent(because the retrying is blocking