Re: [DISCUSS] FLIP-371: SinkV2 Committer imporvements

2023-10-09 Thread Péter Váry
Hi everyone, It seems we have no more comments. So I would like to start a vote tomorrow if there are no further things to discuss. Please let me know if you have any concerns, thanks! Best, Peter On Thu, Oct 5, 2023, 12:53 Gabor Somogyi wrote: > Thanks for the efforts Peter! > > I've just a

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

2023-10-09 Thread Jing Ge
Thanks Yun for your clarification. Especially thanks Rui for your informative elaboration. Since we will have two flame graphs, I would suggest updating Flink documentation to help users understand it and know when to use which one. The content provided by Rui is already a really good starting poin

[jira] [Created] (FLINK-33227) Flink sql cep support 'notFollowedBy' semantics

2023-10-09 Thread xiaoran (Jira)
xiaoran created FLINK-33227: --- Summary: Flink sql cep support 'notFollowedBy' semantics Key: FLINK-33227 URL: https://issues.apache.org/jira/browse/FLINK-33227 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

2023-10-09 Thread Yun Tang
Hi Jing, I will answer current questions. > 1. will it replace the current flame graph, i.e. the current flame graph will be deprecated and removed? Although I think the new java profiler introduced in FLIP-375 is more powerful, just as Rui has replied, I don't think it could replace current fl

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

2023-10-09 Thread Rui Fan
Hi Zhu, Thanks for your feedback! >> 2. When it's set to Tasks, how to assign slots to TM? > It's option2 at the moment. However, I think it's just implementation > details and can be changed/refined later. > > As you mentioned in another comment, 'taskmanager.load-balance.mode' is > a user orien

Re: [DISCUSS] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

2023-10-09 Thread Yangze Guo
Thanks for all the comments! I will initiate the vote tomorrow if there is no further discussion. Best, Yangze Guo On Tue, Oct 10, 2023 at 12:20 AM Jing Ge wrote: > > +1 good idea! Looking forward to Flink OLAP! > > Best regards, > Jing > > On Sat, Oct 7, 2023 at 1:36 PM Yangze Guo wrote: >

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

2023-10-09 Thread Rui Fan
Hi Jing, > 1. will it replace the current flame graph, i.e. the current flame graph will be deprecated and removed? I think the current flame graph cannot be removed. As a core contributor to the current flame graph, and I use it almost every week. I would like to clarify the difference between

Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-10-09 Thread Leonard Xu
+1(binding) Best, Leonard > On Oct 9, 2023, at 9:45 PM, Jing Ge wrote: > > +1(binding) > > Best Regards, > Jing > > On Mon, Oct 9, 2023 at 10:40 AM Ahmed Hamdy wrote: > >> +1 (non-binding) >> Best Regards >> Ahmed Hamdy >> >> >> On Mon, 9 Oct 2023 at 09:38, xiangyu feng wrote: >> >>> +1

[jira] [Created] (FLINK-33226) Forbid to drop current database

2023-10-09 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-33226: - Summary: Forbid to drop current database Key: FLINK-33226 URL: https://issues.apache.org/jira/browse/FLINK-33226 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-373: Support Configuring Different State TTLs using SQL Hint

2023-10-09 Thread Feng Jin
Hi Jane, Thank you for proposing this FLIP. I believe that this FLIP will greatly enhance the flexibility of setting state, and by setting different operators' TTL, it can also increase job stability, especially in regular join scenarios. The parameter design is very concise, big +1 for this, and

[jira] [Created] (FLINK-33225) Python API incorrectly passes `JVM_ARGS` as single argument

2023-10-09 Thread Deepyaman Datta (Jira)
Deepyaman Datta created FLINK-33225: --- Summary: Python API incorrectly passes `JVM_ARGS` as single argument Key: FLINK-33225 URL: https://issues.apache.org/jira/browse/FLINK-33225 Project: Flink

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

2023-10-09 Thread Jing Ge
Hi Yu, Hi Yun, Brilliant idea! People are keen to use it. Thanks for your proposal! I was wondering: 1. will it replace the current flame graph, i.e. the current flame graph will be deprecated and removed? 2. does it make sense to provide the performance difference between enable and disable it?

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

2023-10-09 Thread Zakelly Lan
Hi everyone, It seems we're gradually reaching a consensus. So I would like to start a vote after 72 hours if there are no further discussions. Please let me know if you have any concerns, thanks! Best, Zakelly On Sat, Oct 7, 2023 at 4:07 PM Zakelly Lan wrote: > > Hi Jing, > > Sorry for the

Re: Support AWS SDK V2 for Flink's S3 FileSystem

2023-10-09 Thread Matthias Pohl
I would agree with David's proposal as well. Would it make sense to come up with some performance comparisons for the different S3 implementations in the end? ...just to ensure that we're improving things or (at least) don't make things worse. Or is there something like that already somewhere? A

Re: [DISCUSS] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

2023-10-09 Thread Jing Ge
+1 good idea! Looking forward to Flink OLAP! Best regards, Jing On Sat, Oct 7, 2023 at 1:36 PM Yangze Guo wrote: > Thanks for all your comments. > > @Rui > Thanks for the reminder. The "env.java.opts.all" has already taken effect > now. > > > Best, > Yangze Guo > > On Sat, Oct 7, 2023 at 6:45 P

[jira] [Created] (FLINK-33224) Flink Batch Job does not complete when using Kafka topics with empty partitions

2023-10-09 Thread Kanthi Vaidya (Jira)
Kanthi Vaidya created FLINK-33224: - Summary: Flink Batch Job does not complete when using Kafka topics with empty partitions Key: FLINK-33224 URL: https://issues.apache.org/jira/browse/FLINK-33224 Pr

[jira] [Created] (FLINK-33223) MATCH_RECOGNIZE AFTER MATCH clause can not be deserialised from a compiled plan

2023-10-09 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-33223: Summary: MATCH_RECOGNIZE AFTER MATCH clause can not be deserialised from a compiled plan Key: FLINK-33223 URL: https://issues.apache.org/jira/browse/FLINK-33223

Re: [DISCUSS] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

2023-10-09 Thread Péter Váry
Hi Team, Did some experimenting and found the originally proposed solution to be a bit awkward for cases where WithPostCommitTopology was needed but we do not need the WithPreCommitTopology transformation. The flexibility of the new API would be better if we would use a mixin like approach. The in

Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-10-09 Thread Jing Ge
+1(binding) Best Regards, Jing On Mon, Oct 9, 2023 at 10:40 AM Ahmed Hamdy wrote: > +1 (non-binding) > Best Regards > Ahmed Hamdy > > > On Mon, 9 Oct 2023 at 09:38, xiangyu feng wrote: > > > +1 (non-binding) > > > > Feng Jin 于2023年10月9日周一 16:00写道: > > > > > +1 (non-binding) > > > > > > Best,

[jira] [Created] (FLINK-33222) Operator rollback app when it should not

2023-10-09 Thread Nicolas Fraison (Jira)
Nicolas Fraison created FLINK-33222: --- Summary: Operator rollback app when it should not Key: FLINK-33222 URL: https://issues.apache.org/jira/browse/FLINK-33222 Project: Flink Issue Type: Bu

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

2023-10-09 Thread Yu Chen
Hi zhanghao, Yes, agree with you. We'll take Jobmanager into consideration and update the FLIP later! Best, Yu Chen Zhanghao Chen 于2023年10月9日周一 19:22写道: > Hi Yun and Yu, > > Thanks for driving this. This would definitely help users identify > performance bottlenecks, especially for the cases w

[jira] [Created] (FLINK-33221) Add config options for administrator JVM options

2023-10-09 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-33221: - Summary: Add config options for administrator JVM options Key: FLINK-33221 URL: https://issues.apache.org/jira/browse/FLINK-33221 Project: Flink Issue Type

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

2023-10-09 Thread Rui Fan
Thanks Yun and Yu for driving this proposal! It's very useful for troubleshooting why the CPU usage is high. +1 Best, Rui On Mon, Oct 9, 2023 at 7:21 PM Zhanghao Chen wrote: > Hi Yun and Yu, > > Thanks for driving this. This would definitely help users identify > performance bottlenecks, espec

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

2023-10-09 Thread Zhanghao Chen
Hi Yun and Yu, Thanks for driving this. This would definitely help users identify performance bottlenecks, especially for the cases where the bottleneck lies in the system stack (e.g. GC), and big +1 for the downloadable flamegraph to ease sharing. I'm wondering if we could add this for the job

View and TimestampType serde

2023-10-09 Thread Tamas Sule
Hello, I'm working on a persistent Catalog which stores the tables, views etc in a DB. I'd like to get some guidance on the following issue: Firstly, it seems to be quite hard to serialize a view: I found 2 methods to do it, first is to use DescriptorProperties which is also used by the HiveCatalo

Re: [ANNOUNCE] Release 1.18.0, release candidate #1

2023-10-09 Thread Sergey Nuyanzin
Hi Jing, Thanks for looking into this I checked it against opensearch connector [1] and it is green now Also checked against hbase [2] which is now failing because of a different issue however anyway it was able to go further (before it failed at Download Flink binary task) than when it was not p

[jira] [Created] (FLINK-33220) PyFlink support for datagen connector

2023-10-09 Thread Liu Chong (Jira)
Liu Chong created FLINK-33220: - Summary: PyFlink support for datagen connector Key: FLINK-33220 URL: https://issues.apache.org/jira/browse/FLINK-33220 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-33219) Kafka connector has architecture test violation against Flink 1.18

2023-10-09 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-33219: - Summary: Kafka connector has architecture test violation against Flink 1.18 Key: FLINK-33219 URL: https://issues.apache.org/jira/browse/FLINK-33219 Project: Flink

[jira] [Created] (FLINK-33218) First Steps - error when run with zsh

2023-10-09 Thread Robin Moffatt (Jira)
Robin Moffatt created FLINK-33218: - Summary: First Steps - error when run with zsh Key: FLINK-33218 URL: https://issues.apache.org/jira/browse/FLINK-33218 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array

2023-10-09 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-33217: -- Summary: Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array Key: FLINK-33217 URL: https://issues.apache.org/jira/browse/FLINK-33217 Project: Fl

[DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

2023-10-09 Thread Yu Chen
Hi all, Yun Tang and I are opening this thread to discuss our proposal to integrate async-profiler's capabilities for profiling taskmananger (e.g., generating flame graphs) in the Flink Web [1]. Currently, Flink provides ThreadDump and Operator-Level Flame Graphs by sampling task threads. The re

[jira] [Created] (FLINK-33216) turn off the public access of the s3 bucket flink-nightly

2023-10-09 Thread Jing Ge (Jira)
Jing Ge created FLINK-33216: --- Summary: turn off the public access of the s3 bucket flink-nightly Key: FLINK-33216 URL: https://issues.apache.org/jira/browse/FLINK-33216 Project: Flink Issue Type: S

[jira] [Created] (FLINK-33215) [Umbrella] use https://nightlies.apache.org/flink/ as the flink artifact server for connector nightly build

2023-10-09 Thread Jing Ge (Jira)
Jing Ge created FLINK-33215: --- Summary: [Umbrella] use https://nightlies.apache.org/flink/ as the flink artifact server for connector nightly build Key: FLINK-33215 URL: https://issues.apache.org/jira/browse/FLINK-33215

[jira] [Created] (FLINK-33214) Moving away from the Ververica-owned S3 bucket and use Apache Infra's nightly SVN instead

2023-10-09 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-33214: - Summary: Moving away from the Ververica-owned S3 bucket and use Apache Infra's nightly SVN instead Key: FLINK-33214 URL: https://issues.apache.org/jira/browse/FLINK-33214

Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-10-09 Thread Ahmed Hamdy
+1 (non-binding) Best Regards Ahmed Hamdy On Mon, 9 Oct 2023 at 09:38, xiangyu feng wrote: > +1 (non-binding) > > Feng Jin 于2023年10月9日周一 16:00写道: > > > +1 (non-binding) > > > > Best, > > Feng > > > > On Mon, Oct 9, 2023 at 3:12 PM Yangze Guo wrote: > > > > > +1 (binding) > > > > > > Best, > >

Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-10-09 Thread xiangyu feng
+1 (non-binding) Feng Jin 于2023年10月9日周一 16:00写道: > +1 (non-binding) > > Best, > Feng > > On Mon, Oct 9, 2023 at 3:12 PM Yangze Guo wrote: > > > +1 (binding) > > > > Best, > > Yangze Guo > > > > On Mon, Oct 9, 2023 at 2:46 PM Yun Tang wrote: > > > > > > +1 (binding) > > > > > > Best > > > Yun T

Re: [ANNOUNCE] Release 1.18.0, release candidate #1

2023-10-09 Thread Jing Ge
Hi Folks, Ververica has made the files publicly accessible. Could anyone check if any connector build works again? Best regards, Jing On Mon, Oct 9, 2023 at 9:07 AM Jing Ge wrote: > Hi Sergey and devs, > > Thanks for bringing this to our attention. I am open to discuss that. I > have the follo

Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-10-09 Thread Feng Jin
+1 (non-binding) Best, Feng On Mon, Oct 9, 2023 at 3:12 PM Yangze Guo wrote: > +1 (binding) > > Best, > Yangze Guo > > On Mon, Oct 9, 2023 at 2:46 PM Yun Tang wrote: > > > > +1 (binding) > > > > Best > > Yun Tang > > > > From: Weihua Hu > > Sent: Monday, Octob

[jira] [Created] (FLINK-33213) Flink SQL MinMax aggregations without retract messages when `where` condition exist

2023-10-09 Thread Sergey Paryshev (Jira)
Sergey Paryshev created FLINK-33213: --- Summary: Flink SQL MinMax aggregations without retract messages when `where` condition exist Key: FLINK-33213 URL: https://issues.apache.org/jira/browse/FLINK-33213

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

2023-10-09 Thread Zhu Zhu
Thanks for the response, Rui and Yuepeng. >> Rui > 1. The default value is None, right? Exactly. > 2. When it's set to Tasks, how to assign slots to TM? It's option2 at the moment. However, I think it's just implementation details and can be changed/refined later. As you mentioned in another com

Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-10-09 Thread Yangze Guo
+1 (binding) Best, Yangze Guo On Mon, Oct 9, 2023 at 2:46 PM Yun Tang wrote: > > +1 (binding) > > Best > Yun Tang > > From: Weihua Hu > Sent: Monday, October 9, 2023 12:03 > To: dev@flink.apache.org > Subject: Re: [VOTE] FLIP-367: Support Setting Parallelism fo

Re: [ANNOUNCE] Release 1.18.0, release candidate #1

2023-10-09 Thread Jing Ge
Thanks Konstantin for opening the PR, appreciate it! Best regards, Jing On Fri, Oct 6, 2023 at 2:03 PM Konstantin Knauf wrote: > Hi everyone, > > I've just opened a PR for the release announcement [1] and I am looking > forward to reviews and feedback. > > Cheers, > > Konstantin > > [1] https:/

Re: [ANNOUNCE] Release 1.18.0, release candidate #1

2023-10-09 Thread Jing Ge
Hi Sergey and devs, Thanks for bringing this to our attention. I am open to discuss that. I have the following thoughts: 1. Like I already mentioned in many other threads, build issues in downstream repos should not block upstream release. I understand the concern that developers want to have sta