Re: [DISCUSS] FLIP-415: Introduce a new join operator to support minibatch

[Jingsong Li]

Hi all,

This is a relatively large optimization that may pose a significant
risk of bugs, so I like to keep it from being enabled by default for
now.

Best,
Jingsong

On Fri, Jan 12, 2024 at 3:01 PM shuai xu  wrote:
>
> Suppose we currently have a job that joins two CDC sources after 
> de-duplicating them and the output is available for audit analysis, and the 
> user turns off the parameter 
> "table.exec.deduplicate.mini-batch.compact-changes-enabled" to ensure that it 
> does not lose update details. If we don't introduce this parameter, after the 
> user upgrades the version, some update details may be lost due to the 
> mini-batch connection being enabled by default, resulting in distorted audit 
> results.
>
> > 2024年1月11日 16:19，Benchao Li  写道：
> >
> >> the change might not be supposed for the downstream of the job which 
> >> requires details of changelog
> >
> > Could you elaborate on this a bit? I've never met such kinds of
> > requirements before, I'm curious what is the scenario that requires
> > this.
> >
> > shuai xu  于2024年1月11日周四 13:08写道：
> >>
> >> Thanks for your response, Benchao.
> >>
> >> Here is my thought on the newly added option.
> >> Users' current jobs are running on a version without minibatch join. If 
> >> the existing option to enable minibatch join is utilized, then when users' 
> >> jobs are migrated to the new version, the internal behavior of the join 
> >> operation within the jobs will change. Although the semantic of changelog 
> >> emitted by the Join operator is eventual consistency, the change might not 
> >> be supposed for the downstream of the job which requires details of 
> >> changelog. This newly added option also refers to 
> >> 'table.exec.deduplicate.mini-batch.compact-changes-enabled'.
> >>
> >> As for the implementation，The new operator shares the state of the 
> >> original operator and it merely has an additional minibatch for storing 
> >> records to do some optimization. The storage remains consistent, and there 
> >> is minor modification to the computational logic.
> >>
> >> Best,
> >> Xu Shuai
> >>
> >>> 2024年1月10日 22:56，Benchao Li  写道：
> >>>
> >>> Thanks shuai for driving this, mini-batch Join is a very useful
> >>> optimization, +1 for the general idea.
> >>>
> >>> Regarding the configuration
> >>> "table.exec.stream.join.mini-batch-enabled", I'm not sure it's really
> >>> necessary. The semantic of changelog emitted by the Join operator is
> >>> eventual consistency, so there is no much difference between original
> >>> Join and mini-batch Join from this aspect. Besides, introducing more
> >>> options would make it more complex for users, harder to understand and
> >>> maintain, which we should be careful about.
> >>>
> >>> One thing about the implementation, could you make the new operator
> >>> share the same state definition with the original one?
> >>>
> >>> shuai xu  于2024年1月10日周三 21:23写道：
> 
>  Hi devs,
> 
>  I’d like to start a discussion on FLIP-415: Introduce a new join 
>  operator to support minibatch[1].
> 
>  Currently, when performing cascading connections in Flink, there is a 
>  pain point of record amplification. Every record join operator receives 
>  would trigger join process. However, if records of +I and -D matches , 
>  they could be folded to reduce two times of join process. Besides, 
>  records of  -U +U might output 4 records in which two records are 
>  redundant when encountering outer join .
> 
>  To address this issue, this FLIP introduces a new  
>  MiniBatchStreamingJoinOperator to achieve batch processing which could 
>  reduce number of outputting redundant messages and avoid unnecessary 
>  join processes.
>  A new option is added to control the operator to avoid influencing 
>  existing jobs.
> 
>  Please find more details in the FLIP wiki document [1]. Looking
>  forward to your feedback.
> 
>  [1]
>  https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> 
>  Best,
>  Xu Shuai
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Best,
> >>> Benchao Li
> >>
> >
> >
> > --
> >
> > Best,
> > Benchao Li
>
> Best,
> Xu Shuai
>

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[Kurt Yang]

+1 (binding)

Best,
Kurt


On Fri, Jan 12, 2024 at 2:21 PM Hequn Cheng  wrote:

> +1 (binding)
>
> Thanks,
> Hequn
>
> On Fri, Jan 12, 2024 at 2:19 PM godfrey he  wrote:
>
> > +1 (binding)
> >
> > Thanks,
> > Godfrey
> >
> > Zhu Zhu  于2024年1月12日周五 14:10写道：
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Hangxiang Yu  于2024年1月11日周四 14:26写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su 
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Xuannan
> > > > >
> > > > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang 
> wrote:
> > > > > >
> > > > > > +1 (non-binding)--
> > > > > >
> > > > > > Best！
> > > > > > Xuyang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 在 2024-01-11 10:00:11，"Yang Wang"  写道：
> > > > > > >+1 (binding)
> > > > > > >
> > > > > > >
> > > > > > >Best,
> > > > > > >Yang
> > > > > > >
> > > > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron 
> > wrote:
> > > > > > >
> > > > > > >> +1 non-binding
> > > > > > >>
> > > > > > >> Best
> > > > > > >> Ron
> > > > > > >>
> > > > > > >> Matthias Pohl  于2024年1月10日周三
> > > > 23:05写道：
> > > > > > >>
> > > > > > >> > +1 (binding)
> > > > > > >> >
> > > > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam <
> > jam.gz...@gmail.com>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > +1 non-binding
> > > > > > >> > >
> > > > > > >> > > Dawid Wysakowicz  于2024年1月10日周三
> > > > 21:06写道：
> > > > > > >> > >
> > > > > > >> > > > +1 (binding)
> > > > > > >> > > > Best,
> > > > > > >> > > > Dawid
> > > > > > >> > > >
> > > > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski <
> > > > > pnowoj...@apache.org>
> > > > > > >> > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > +1 (binding)
> > > > > > >> > > > >
> > > > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser <
> > > > > martijnvis...@apache.org>
> > > > > > >> > > > > napisał(a):
> > > > > > >> > > > >
> > > > > > >> > > > > > +1 (binding)
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo Huang <
> > > > > hxbks...@gmail.com
> > > > > > >> >
> > > > > > >> > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > +1 (binding)
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Best,
> > > > > > >> > > > > > > Xingbo
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Dian Fu  于2024年1月10日周三
> > 11:35写道：
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > +1 (binding)
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Regards,
> > > > > > >> > > > > > > > Dian
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Wed, Jan 10, 2024 at 5:09 AM Sharath <
> > > > > > >> dsaishar...@gmail.com
> > > > > > >> > >
> > > > > > >> > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > +1 (non-binding)
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Best,
> > > > > > >> > > > > > > > > Sharath
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Tue, Jan 9, 2024 at 1:02 PM Venkata Sanath
> > > > > Muppalla <
> > > > > > >> > > > > > > > sanath...@gmail.com>
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > +1 (non-binding)
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > Sanath
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Tue, Jan 9, 2024 at 11:16 AM Peter Huang
> <
> > > > > > >> > > > > > > > huangzhenqiu0...@gmail.com>
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > +1 (non-binding)
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Best Regards
> > > > > > >> > > > > > > > > > > Peter Huang
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Tue, Jan 9, 2024 at 5:26 AM Jane Chan <
> > > > > > >> > > > > qingyue@gmail.com>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > +1 (non-binding)
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Best,
> > > > > > >> > > > > > > > > > > > Jane
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > On Tue, Jan 9, 2024 at 8:41 PM Lijie
> Wang
> > <
> > > > > > >> > > > > > > > wangdachui9...@gmail.com>
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > +1 (non-binding)
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Best,
> > > > > > >> > > > > > > > > > > > > Lijie
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Jiabao Sun  > > > .invalid>
> > > > > > >> > > > 于2024年1月9日周二
> > > > > > >> > > > > > > > 19:28写道：
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > +1 

Re: [DISCUSS] FLIP-415: Introduce a new join operator to support minibatch

[shuai xu]

Suppose we currently have a job that joins two CDC sources after de-duplicating 
them and the output is available for audit analysis, and the user turns off the 
parameter "table.exec.deduplicate.mini-batch.compact-changes-enabled" to ensure 
that it does not lose update details. If we don't introduce this parameter, 
after the user upgrades the version, some update details may be lost due to the 
mini-batch connection being enabled by default, resulting in distorted audit 
results.

> 2024年1月11日 16:19，Benchao Li  写道：
> 
>> the change might not be supposed for the downstream of the job which 
>> requires details of changelog
> 
> Could you elaborate on this a bit? I've never met such kinds of
> requirements before, I'm curious what is the scenario that requires
> this.
> 
> shuai xu  于2024年1月11日周四 13:08写道：
>> 
>> Thanks for your response, Benchao.
>> 
>> Here is my thought on the newly added option.
>> Users' current jobs are running on a version without minibatch join. If the 
>> existing option to enable minibatch join is utilized, then when users' jobs 
>> are migrated to the new version, the internal behavior of the join operation 
>> within the jobs will change. Although the semantic of changelog emitted by 
>> the Join operator is eventual consistency, the change might not be supposed 
>> for the downstream of the job which requires details of changelog. This 
>> newly added option also refers to 
>> 'table.exec.deduplicate.mini-batch.compact-changes-enabled'.
>> 
>> As for the implementation，The new operator shares the state of the original 
>> operator and it merely has an additional minibatch for storing records to do 
>> some optimization. The storage remains consistent, and there is minor 
>> modification to the computational logic.
>> 
>> Best,
>> Xu Shuai
>> 
>>> 2024年1月10日 22:56，Benchao Li  写道：
>>> 
>>> Thanks shuai for driving this, mini-batch Join is a very useful
>>> optimization, +1 for the general idea.
>>> 
>>> Regarding the configuration
>>> "table.exec.stream.join.mini-batch-enabled", I'm not sure it's really
>>> necessary. The semantic of changelog emitted by the Join operator is
>>> eventual consistency, so there is no much difference between original
>>> Join and mini-batch Join from this aspect. Besides, introducing more
>>> options would make it more complex for users, harder to understand and
>>> maintain, which we should be careful about.
>>> 
>>> One thing about the implementation, could you make the new operator
>>> share the same state definition with the original one?
>>> 
>>> shuai xu  于2024年1月10日周三 21:23写道：
 
 Hi devs,
 
 I’d like to start a discussion on FLIP-415: Introduce a new join operator 
 to support minibatch[1].
 
 Currently, when performing cascading connections in Flink, there is a pain 
 point of record amplification. Every record join operator receives would 
 trigger join process. However, if records of +I and -D matches , they 
 could be folded to reduce two times of join process. Besides, records of  
 -U +U might output 4 records in which two records are redundant when 
 encountering outer join .
 
 To address this issue, this FLIP introduces a new  
 MiniBatchStreamingJoinOperator to achieve batch processing which could 
 reduce number of outputting redundant messages and avoid unnecessary join 
 processes.
 A new option is added to control the operator to avoid influencing 
 existing jobs.
 
 Please find more details in the FLIP wiki document [1]. Looking
 forward to your feedback.
 
 [1]
 https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
 
 Best,
 Xu Shuai
>>> 
>>> 
>>> 
>>> --
>>> 
>>> Best,
>>> Benchao Li
>> 
> 
> 
> -- 
> 
> Best,
> Benchao Li

Best, 
Xu Shuai

Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

[Hang Ruan]

Hi, Mason.

Thanks for driving this FLIP.

The JobManagerOperatorQueryScopeInfo has three fields: jobID, vertexID and
operatorName. So we should use the operator name in the API.
If you think we should use the operator id, there need be more changes
about it.

About the Xuyang's questions, we add both vertexID and operatorID
information because of the operator chain.
A operator chain has a vertexID and contains many different operators. The
operator information helps to distinguish them in the same operator chain.

Best,
Hang


Xuyang  于2024年1月12日周五 10:21写道：

> Hi, Mason.
> Thanks for driving this Flip. I think it's important for external system
> to be able to
> perceive the metric of the operator coordinator. +1 for it.
>
>
> I just have the following minor questions and am looking forward to your
> reply. Please forgive
> me if I have some misunderstandings.
>
>
> 1. IIRC, in a sense, operator ID and vertex ID are the same thing. The
> operator ID can
> be converted from the vertex ID[1]. Therefore, it is somewhat strange to
> have both vertex
> ID and operator ID in a single URL.
>
>
> 2. If I misunderstood the semantics of operator IDs here, then what is the
> relationship
> between vertex ID and operator ID, and do we need a url like
> `/jobs//vertices//operators/`
> to list all operator ids under this vertices？
>
>
>
>
> [1]
> https://github.com/apache/flink/blob/7bebd2d9fac517c28afc24c0c034d77cfe2b43a6/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/OperatorID.java#L40C27-L40C27
>
> --
>
> Best！
> Xuyang
>
>
>
>
>
> At 2024-01-12 04:20:03, "Mason Chen"  wrote:
> >Hi Devs,
> >
> >I'm opening this thread to discuss a short FLIP for exposing
> >JobManagerOperatorMetrics via REST API [1].
> >
> >The current set of REST APIs make it impossible to query coordinator
> >metrics. This FLIP proposes a new REST API to query the
> >JobManagerOperatorMetrics.
> >
> >[1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-417%3A+Expose+JobManagerOperatorMetrics+via+REST+API
> >
> >Best,
> >Mason
>

Re: [VOTE] FLIP-405: Migrate string configuration key to ConfigOption

[Zhu Zhu]

+1 (binding)

Thanks,
Zhu

Xuannan Su  于2024年1月12日周五 14:24写道：

> Hi all,
>
> I would like to clarify the statement regarding the first improvement
> from the previous email, as it was incomplete. To be more specific, we
> will also deprecate the getClass(String key, Class
> defaultValue, ClassLoader classLoader) and setClass(String key,
> Class klazz), as they are intended for internal use only and are no
> longer in use.
>
> To summarize, the first improvement should be as follows:
>
> We will mark the getBytes(String key, byte[] defaultValue) and
> setBytes(String key, byte[] bytes) methods as @Internal, as they are
> intended for internal use only. Additionally, we will
> deprecatesgetClass(String key, Class defaultValue,
> ClassLoader classLoader) and setClass(String key, Class klazz)
> methods, as they are intended for internal use only and are no longer
> in use.
>
> I apologize for any confusion caused by the incomplete information.
>
> Best regards,
> Xuannan
>
> On Fri, Jan 12, 2024 at 1:36 PM Xuannan Su  wrote:
> >
> > Hi all,
> >
> > During voting, we identified two improvements we'd like to make to the
> FLIP:
> >
> > - We will mark the getBytes(String key, byte[] defaultValue) and
> > setBytes(String key, byte[] bytes) methods as @Internal, as they are
> > intended for internal use only.
> > - In addition to marking all getXxx(ConfigOption configOption) methods
> > as @Deprecated, we will also mark the getXxx(ConfigOption
> > configOption, Xxx overrideDefault) methods as @Deprecated. These will
> > be replaced with T get(ConfigOption configOption, T overrideDefault).
> >
> > We had an offline discussion with all the voters and reached a
> > consensus on the above changes. Therefore, we will not initiate
> > another voting thread unless there are objections.
> >
> > Best regards,
> > Xuannan
> >
> >
> > On Tue, Jan 9, 2024 at 11:43 AM Xintong Song 
> wrote:
> > >
> > > +1 (binding)
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Mon, Jan 8, 2024 at 1:48 PM Hang Ruan 
> wrote:
> > >
> > > > +1(non-binding)
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Rui Fan <1996fan...@gmail.com> 于2024年1月8日周一 13:04写道：
> > > >
> > > > > +1(binding)
> > > > >
> > > > > Best,
> > > > > Rui
> > > > >
> > > > > On Mon, Jan 8, 2024 at 1:00 PM Xuannan Su 
> wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Thanks for all the feedback about the FLIP-405: Migrate string
> > > > > > configuration key to ConfigOption [1] [2].
> > > > > >
> > > > > > I'd like to start a vote for it. The vote will be open for at
> least 72
> > > > > > hours(excluding weekends,until Jan 11, 12:00AM GMT) unless there
> is an
> > > > > > objection or an insufficient number of votes.
> > > > > >
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-405%3A+Migrate+string+configuration+key+to+ConfigOption
> > > > > > [2]
> https://lists.apache.org/thread/zfw1b1g3679yn0ppjbsokfrsx9k7ybg0
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Xuannan
> > > > > >
> > > > >
> > > >
>

Re: [VOTE] FLIP-405: Migrate string configuration key to ConfigOption

[Xuannan Su]

Hi all,

I would like to clarify the statement regarding the first improvement
from the previous email, as it was incomplete. To be more specific, we
will also deprecate the getClass(String key, Class
defaultValue, ClassLoader classLoader) and setClass(String key,
Class klazz), as they are intended for internal use only and are no
longer in use.

To summarize, the first improvement should be as follows:

We will mark the getBytes(String key, byte[] defaultValue) and
setBytes(String key, byte[] bytes) methods as @Internal, as they are
intended for internal use only. Additionally, we will
deprecatesgetClass(String key, Class defaultValue,
ClassLoader classLoader) and setClass(String key, Class klazz)
methods, as they are intended for internal use only and are no longer
in use.

I apologize for any confusion caused by the incomplete information.

Best regards,
Xuannan

On Fri, Jan 12, 2024 at 1:36 PM Xuannan Su  wrote:
>
> Hi all,
>
> During voting, we identified two improvements we'd like to make to the FLIP:
>
> - We will mark the getBytes(String key, byte[] defaultValue) and
> setBytes(String key, byte[] bytes) methods as @Internal, as they are
> intended for internal use only.
> - In addition to marking all getXxx(ConfigOption configOption) methods
> as @Deprecated, we will also mark the getXxx(ConfigOption
> configOption, Xxx overrideDefault) methods as @Deprecated. These will
> be replaced with T get(ConfigOption configOption, T overrideDefault).
>
> We had an offline discussion with all the voters and reached a
> consensus on the above changes. Therefore, we will not initiate
> another voting thread unless there are objections.
>
> Best regards,
> Xuannan
>
>
> On Tue, Jan 9, 2024 at 11:43 AM Xintong Song  wrote:
> >
> > +1 (binding)
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Jan 8, 2024 at 1:48 PM Hang Ruan  wrote:
> >
> > > +1(non-binding)
> > >
> > > Best,
> > > Hang
> > >
> > > Rui Fan <1996fan...@gmail.com> 于2024年1月8日周一 13:04写道：
> > >
> > > > +1(binding)
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Mon, Jan 8, 2024 at 1:00 PM Xuannan Su  wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks for all the feedback about the FLIP-405: Migrate string
> > > > > configuration key to ConfigOption [1] [2].
> > > > >
> > > > > I'd like to start a vote for it. The vote will be open for at least 72
> > > > > hours(excluding weekends,until Jan 11, 12:00AM GMT) unless there is an
> > > > > objection or an insufficient number of votes.
> > > > >
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-405%3A+Migrate+string+configuration+key+to+ConfigOption
> > > > > [2] https://lists.apache.org/thread/zfw1b1g3679yn0ppjbsokfrsx9k7ybg0
> > > > >
> > > > >
> > > > > Best,
> > > > > Xuannan
> > > > >
> > > >
> > >

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[Hequn Cheng]

+1 (binding)

Thanks,
Hequn

On Fri, Jan 12, 2024 at 2:19 PM godfrey he  wrote:

> +1 (binding)
>
> Thanks,
> Godfrey
>
> Zhu Zhu  于2024年1月12日周五 14:10写道：
> >
> > +1 (binding)
> >
> > Thanks,
> > Zhu
> >
> > Hangxiang Yu  于2024年1月11日周四 14:26写道：
> >
> > > +1 (non-binding)
> > >
> > > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Xuannan
> > > >
> > > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang  wrote:
> > > > >
> > > > > +1 (non-binding)--
> > > > >
> > > > > Best！
> > > > > Xuyang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 在 2024-01-11 10:00:11，"Yang Wang"  写道：
> > > > > >+1 (binding)
> > > > > >
> > > > > >
> > > > > >Best,
> > > > > >Yang
> > > > > >
> > > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron 
> wrote:
> > > > > >
> > > > > >> +1 non-binding
> > > > > >>
> > > > > >> Best
> > > > > >> Ron
> > > > > >>
> > > > > >> Matthias Pohl  于2024年1月10日周三
> > > 23:05写道：
> > > > > >>
> > > > > >> > +1 (binding)
> > > > > >> >
> > > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam <
> jam.gz...@gmail.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > > +1 non-binding
> > > > > >> > >
> > > > > >> > > Dawid Wysakowicz  于2024年1月10日周三
> > > 21:06写道：
> > > > > >> > >
> > > > > >> > > > +1 (binding)
> > > > > >> > > > Best,
> > > > > >> > > > Dawid
> > > > > >> > > >
> > > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski <
> > > > pnowoj...@apache.org>
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > > +1 (binding)
> > > > > >> > > > >
> > > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser <
> > > > martijnvis...@apache.org>
> > > > > >> > > > > napisał(a):
> > > > > >> > > > >
> > > > > >> > > > > > +1 (binding)
> > > > > >> > > > > >
> > > > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo Huang <
> > > > hxbks...@gmail.com
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > +1 (binding)
> > > > > >> > > > > > >
> > > > > >> > > > > > > Best,
> > > > > >> > > > > > > Xingbo
> > > > > >> > > > > > >
> > > > > >> > > > > > > Dian Fu  于2024年1月10日周三
> 11:35写道：
> > > > > >> > > > > > >
> > > > > >> > > > > > > > +1 (binding)
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Regards,
> > > > > >> > > > > > > > Dian
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Jan 10, 2024 at 5:09 AM Sharath <
> > > > > >> dsaishar...@gmail.com
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Best,
> > > > > >> > > > > > > > > Sharath
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Tue, Jan 9, 2024 at 1:02 PM Venkata Sanath
> > > > Muppalla <
> > > > > >> > > > > > > > sanath...@gmail.com>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > Sanath
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Tue, Jan 9, 2024 at 11:16 AM Peter Huang <
> > > > > >> > > > > > > > huangzhenqiu0...@gmail.com>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Best Regards
> > > > > >> > > > > > > > > > > Peter Huang
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Tue, Jan 9, 2024 at 5:26 AM Jane Chan <
> > > > > >> > > > > qingyue@gmail.com>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > Jane
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Tue, Jan 9, 2024 at 8:41 PM Lijie Wang
> <
> > > > > >> > > > > > > > wangdachui9...@gmail.com>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > Lijie
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Jiabao Sun  > > .invalid>
> > > > > >> > > > 于2024年1月9日周二
> > > > > >> > > > > > > > 19:28写道：
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > > Jiabao
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On 2024/01/09 09:58:04 xiangyu feng
> wrote:
> > > > > >> > > > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> 

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[jincheng sun]

+1 (binding)

Best,
Jincheng Sun


Zhu Zhu  于2024年1月12日周五 14:11写道：

> +1 (binding)
>
> Thanks,
> Zhu
>
> Hangxiang Yu  于2024年1月11日周四 14:26写道：
>
> > +1 (non-binding)
> >
> > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Xuannan
> > >
> > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang  wrote:
> > > >
> > > > +1 (non-binding)--
> > > >
> > > > Best！
> > > > Xuyang
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 在 2024-01-11 10:00:11，"Yang Wang"  写道：
> > > > >+1 (binding)
> > > > >
> > > > >
> > > > >Best,
> > > > >Yang
> > > > >
> > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron  wrote:
> > > > >
> > > > >> +1 non-binding
> > > > >>
> > > > >> Best
> > > > >> Ron
> > > > >>
> > > > >> Matthias Pohl  于2024年1月10日周三
> > 23:05写道：
> > > > >>
> > > > >> > +1 (binding)
> > > > >> >
> > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam 
> > > wrote:
> > > > >> >
> > > > >> > > +1 non-binding
> > > > >> > >
> > > > >> > > Dawid Wysakowicz  于2024年1月10日周三
> > 21:06写道：
> > > > >> > >
> > > > >> > > > +1 (binding)
> > > > >> > > > Best,
> > > > >> > > > Dawid
> > > > >> > > >
> > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski <
> > > pnowoj...@apache.org>
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > > +1 (binding)
> > > > >> > > > >
> > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser <
> > > martijnvis...@apache.org>
> > > > >> > > > > napisał(a):
> > > > >> > > > >
> > > > >> > > > > > +1 (binding)
> > > > >> > > > > >
> > > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo Huang <
> > > hxbks...@gmail.com
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > +1 (binding)
> > > > >> > > > > > >
> > > > >> > > > > > > Best,
> > > > >> > > > > > > Xingbo
> > > > >> > > > > > >
> > > > >> > > > > > > Dian Fu  于2024年1月10日周三
> 11:35写道：
> > > > >> > > > > > >
> > > > >> > > > > > > > +1 (binding)
> > > > >> > > > > > > >
> > > > >> > > > > > > > Regards,
> > > > >> > > > > > > > Dian
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed, Jan 10, 2024 at 5:09 AM Sharath <
> > > > >> dsaishar...@gmail.com
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Best,
> > > > >> > > > > > > > > Sharath
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Tue, Jan 9, 2024 at 1:02 PM Venkata Sanath
> > > Muppalla <
> > > > >> > > > > > > > sanath...@gmail.com>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > Sanath
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Tue, Jan 9, 2024 at 11:16 AM Peter Huang <
> > > > >> > > > > > > > huangzhenqiu0...@gmail.com>
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Best Regards
> > > > >> > > > > > > > > > > Peter Huang
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Tue, Jan 9, 2024 at 5:26 AM Jane Chan <
> > > > >> > > > > qingyue@gmail.com>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > Jane
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Tue, Jan 9, 2024 at 8:41 PM Lijie Wang <
> > > > >> > > > > > > > wangdachui9...@gmail.com>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > Lijie
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Jiabao Sun  > .invalid>
> > > > >> > > > 于2024年1月9日周二
> > > > >> > > > > > > > 19:28写道：
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > > Jiabao
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > On 2024/01/09 09:58:04 xiangyu feng
> wrote:
> > > > >> > > > > > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Regards,
> > > > >> > > > > > > > > > > > > > > Xiangyu Feng
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Danny Cranmer 
> > > 于2024年1月9日周二
> > > > >> > > > 17:50写道：
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > +1 (binding)
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Thanks,
> > > > 

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[godfrey he]

+1 (binding)

Thanks,
Godfrey

Zhu Zhu  于2024年1月12日周五 14:10写道：
>
> +1 (binding)
>
> Thanks,
> Zhu
>
> Hangxiang Yu  于2024年1月11日周四 14:26写道：
>
> > +1 (non-binding)
> >
> > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su  wrote:
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Xuannan
> > >
> > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang  wrote:
> > > >
> > > > +1 (non-binding)--
> > > >
> > > > Best！
> > > > Xuyang
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 在 2024-01-11 10:00:11，"Yang Wang"  写道：
> > > > >+1 (binding)
> > > > >
> > > > >
> > > > >Best,
> > > > >Yang
> > > > >
> > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron  wrote:
> > > > >
> > > > >> +1 non-binding
> > > > >>
> > > > >> Best
> > > > >> Ron
> > > > >>
> > > > >> Matthias Pohl  于2024年1月10日周三
> > 23:05写道：
> > > > >>
> > > > >> > +1 (binding)
> > > > >> >
> > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam 
> > > wrote:
> > > > >> >
> > > > >> > > +1 non-binding
> > > > >> > >
> > > > >> > > Dawid Wysakowicz  于2024年1月10日周三
> > 21:06写道：
> > > > >> > >
> > > > >> > > > +1 (binding)
> > > > >> > > > Best,
> > > > >> > > > Dawid
> > > > >> > > >
> > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski <
> > > pnowoj...@apache.org>
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > > +1 (binding)
> > > > >> > > > >
> > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser <
> > > martijnvis...@apache.org>
> > > > >> > > > > napisał(a):
> > > > >> > > > >
> > > > >> > > > > > +1 (binding)
> > > > >> > > > > >
> > > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo Huang <
> > > hxbks...@gmail.com
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > +1 (binding)
> > > > >> > > > > > >
> > > > >> > > > > > > Best,
> > > > >> > > > > > > Xingbo
> > > > >> > > > > > >
> > > > >> > > > > > > Dian Fu  于2024年1月10日周三 11:35写道：
> > > > >> > > > > > >
> > > > >> > > > > > > > +1 (binding)
> > > > >> > > > > > > >
> > > > >> > > > > > > > Regards,
> > > > >> > > > > > > > Dian
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed, Jan 10, 2024 at 5:09 AM Sharath <
> > > > >> dsaishar...@gmail.com
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Best,
> > > > >> > > > > > > > > Sharath
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Tue, Jan 9, 2024 at 1:02 PM Venkata Sanath
> > > Muppalla <
> > > > >> > > > > > > > sanath...@gmail.com>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > Sanath
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Tue, Jan 9, 2024 at 11:16 AM Peter Huang <
> > > > >> > > > > > > > huangzhenqiu0...@gmail.com>
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Best Regards
> > > > >> > > > > > > > > > > Peter Huang
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Tue, Jan 9, 2024 at 5:26 AM Jane Chan <
> > > > >> > > > > qingyue@gmail.com>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > Jane
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Tue, Jan 9, 2024 at 8:41 PM Lijie Wang <
> > > > >> > > > > > > > wangdachui9...@gmail.com>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > Lijie
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Jiabao Sun  > .invalid>
> > > > >> > > > 于2024年1月9日周二
> > > > >> > > > > > > > 19:28写道：
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > > > > Jiabao
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > On 2024/01/09 09:58:04 xiangyu feng wrote:
> > > > >> > > > > > > > > > > > > > > +1 (non-binding)
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Regards,
> > > > >> > > > > > > > > > > > > > > Xiangyu Feng
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Danny Cranmer 
> > > 于2024年1月9日周二
> > > > >> > > > 17:50写道：
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > +1 (binding)
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Thanks,
> > > > >> > > > > 

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[Zhu Zhu]

+1 (binding)

Thanks,
Zhu

Hangxiang Yu  于2024年1月11日周四 14:26写道：

> +1 (non-binding)
>
> On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su  wrote:
>
> > +1 (non-binding)
> >
> > Best,
> > Xuannan
> >
> > On Thu, Jan 11, 2024 at 10:28 AM Xuyang  wrote:
> > >
> > > +1 (non-binding)--
> > >
> > > Best！
> > > Xuyang
> > >
> > >
> > >
> > >
> > >
> > > 在 2024-01-11 10:00:11，"Yang Wang"  写道：
> > > >+1 (binding)
> > > >
> > > >
> > > >Best,
> > > >Yang
> > > >
> > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron  wrote:
> > > >
> > > >> +1 non-binding
> > > >>
> > > >> Best
> > > >> Ron
> > > >>
> > > >> Matthias Pohl  于2024年1月10日周三
> 23:05写道：
> > > >>
> > > >> > +1 (binding)
> > > >> >
> > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam 
> > wrote:
> > > >> >
> > > >> > > +1 non-binding
> > > >> > >
> > > >> > > Dawid Wysakowicz  于2024年1月10日周三
> 21:06写道：
> > > >> > >
> > > >> > > > +1 (binding)
> > > >> > > > Best,
> > > >> > > > Dawid
> > > >> > > >
> > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski <
> > pnowoj...@apache.org>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > +1 (binding)
> > > >> > > > >
> > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser <
> > martijnvis...@apache.org>
> > > >> > > > > napisał(a):
> > > >> > > > >
> > > >> > > > > > +1 (binding)
> > > >> > > > > >
> > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo Huang <
> > hxbks...@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > +1 (binding)
> > > >> > > > > > >
> > > >> > > > > > > Best,
> > > >> > > > > > > Xingbo
> > > >> > > > > > >
> > > >> > > > > > > Dian Fu  于2024年1月10日周三 11:35写道：
> > > >> > > > > > >
> > > >> > > > > > > > +1 (binding)
> > > >> > > > > > > >
> > > >> > > > > > > > Regards,
> > > >> > > > > > > > Dian
> > > >> > > > > > > >
> > > >> > > > > > > > On Wed, Jan 10, 2024 at 5:09 AM Sharath <
> > > >> dsaishar...@gmail.com
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > >
> > > >> > > > > > > > > Best,
> > > >> > > > > > > > > Sharath
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Tue, Jan 9, 2024 at 1:02 PM Venkata Sanath
> > Muppalla <
> > > >> > > > > > > > sanath...@gmail.com>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks,
> > > >> > > > > > > > > > Sanath
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Tue, Jan 9, 2024 at 11:16 AM Peter Huang <
> > > >> > > > > > > > huangzhenqiu0...@gmail.com>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Best Regards
> > > >> > > > > > > > > > > Peter Huang
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Tue, Jan 9, 2024 at 5:26 AM Jane Chan <
> > > >> > > > > qingyue@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > Jane
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Tue, Jan 9, 2024 at 8:41 PM Lijie Wang <
> > > >> > > > > > > > wangdachui9...@gmail.com>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > Lijie
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Jiabao Sun  .invalid>
> > > >> > > > 于2024年1月9日周二
> > > >> > > > > > > > 19:28写道：
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > > Jiabao
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On 2024/01/09 09:58:04 xiangyu feng wrote:
> > > >> > > > > > > > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Regards,
> > > >> > > > > > > > > > > > > > > Xiangyu Feng
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Danny Cranmer 
> > 于2024年1月9日周二
> > > >> > > > 17:50写道：
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > +1 (binding)
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > Danny
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > On Tue, Jan 9, 2024 at 9:31 AM Feng
> Jin
> > <
> > > >> > > > > > ji...@gmail.com>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > 

Re:Re: [DISCUSS] FLIP-415: Introduce a new join operator to support minibatch

[Xuyang]

Hi, Xu Shuai. Thanks for driving this flip.


The CDC message amplification of cascade join has always been a problem for 
users. Judging from the 
nexmark results, this optimization is very meaningful. I just have the same 
doubts as Benchao, why can't we 
use minibatch join as the default behavior when the user turns on minibatch?


> Although the semantic of changelog emitted by the Join operator is eventual 
> consistency, the change might
not be supposed for the downstream of the job which requires details of 
changelog.


I think if the user adds the minibatch options to his job to enable minibatch, 
he should know that flink will reduce
the amount of data sent to downstream by folding CDC messages as much as 
possible. In scenarios where all
details of CDC records need to be retained, such as just synchronizing data 
with jobs from one db to another db,
users have no reason to enable minibatch.


The only scenario I can think of that requires adding this independent 
minibatch join option is to ensure that the state
is compatible between multiple versions, but we have not promised users state 
compatibility during cross-version upgrades.


Maybe we need to figure it out why does the 
'table.exec.deduplicate.mini-batch.compact-changes-enabled' option need to
be added to deduplicate operator? I think this is the same reason as adding a 
separate parameter to join to control CDC message folding.




--

Best！
Xuyang





在 2024-01-11 16:19:30，"Benchao Li"  写道：
>> the change might not be supposed for the downstream of the job which 
>> requires details of changelog
>
>Could you elaborate on this a bit? I've never met such kinds of
>requirements before, I'm curious what is the scenario that requires
>this.
>
>shuai xu  于2024年1月11日周四 13:08写道：
>>
>> Thanks for your response, Benchao.
>>
>> Here is my thought on the newly added option.
>> Users' current jobs are running on a version without minibatch join. If the 
>> existing option to enable minibatch join is utilized, then when users' jobs 
>> are migrated to the new version, the internal behavior of the join operation 
>> within the jobs will change. Although the semantic of changelog emitted by 
>> the Join operator is eventual consistency, the change might not be supposed 
>> for the downstream of the job which requires details of changelog. This 
>> newly added option also refers to 
>> 'table.exec.deduplicate.mini-batch.compact-changes-enabled'.
>>
>> As for the implementation，The new operator shares the state of the original 
>> operator and it merely has an additional minibatch for storing records to do 
>> some optimization. The storage remains consistent, and there is minor 
>> modification to the computational logic.
>>
>> Best,
>> Xu Shuai
>>
>> > 2024年1月10日 22:56，Benchao Li  写道：
>> >
>> > Thanks shuai for driving this, mini-batch Join is a very useful
>> > optimization, +1 for the general idea.
>> >
>> > Regarding the configuration
>> > "table.exec.stream.join.mini-batch-enabled", I'm not sure it's really
>> > necessary. The semantic of changelog emitted by the Join operator is
>> > eventual consistency, so there is no much difference between original
>> > Join and mini-batch Join from this aspect. Besides, introducing more
>> > options would make it more complex for users, harder to understand and
>> > maintain, which we should be careful about.
>> >
>> > One thing about the implementation, could you make the new operator
>> > share the same state definition with the original one?
>> >
>> > shuai xu  于2024年1月10日周三 21:23写道：
>> >>
>> >> Hi devs,
>> >>
>> >> I’d like to start a discussion on FLIP-415: Introduce a new join operator 
>> >> to support minibatch[1].
>> >>
>> >> Currently, when performing cascading connections in Flink, there is a 
>> >> pain point of record amplification. Every record join operator receives 
>> >> would trigger join process. However, if records of +I and -D matches , 
>> >> they could be folded to reduce two times of join process. Besides, 
>> >> records of  -U +U might output 4 records in which two records are 
>> >> redundant when encountering outer join .
>> >>
>> >> To address this issue, this FLIP introduces a new  
>> >> MiniBatchStreamingJoinOperator to achieve batch processing which could 
>> >> reduce number of outputting redundant messages and avoid unnecessary join 
>> >> processes.
>> >> A new option is added to control the operator to avoid influencing 
>> >> existing jobs.
>> >>
>> >> Please find more details in the FLIP wiki document [1]. Looking
>> >> forward to your feedback.
>> >>
>> >> [1]
>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
>> >>
>> >> Best,
>> >> Xu Shuai
>> >
>> >
>> >
>> > --
>> >
>> > Best,
>> > Benchao Li
>>
>
>
>-- 
>
>Best,
>Benchao Li

Re: [VOTE] FLIP-405: Migrate string configuration key to ConfigOption

[Xuannan Su]

Hi all,

During voting, we identified two improvements we'd like to make to the FLIP:

- We will mark the getBytes(String key, byte[] defaultValue) and
setBytes(String key, byte[] bytes) methods as @Internal, as they are
intended for internal use only.
- In addition to marking all getXxx(ConfigOption configOption) methods
as @Deprecated, we will also mark the getXxx(ConfigOption
configOption, Xxx overrideDefault) methods as @Deprecated. These will
be replaced with T get(ConfigOption configOption, T overrideDefault).

We had an offline discussion with all the voters and reached a
consensus on the above changes. Therefore, we will not initiate
another voting thread unless there are objections.

Best regards,
Xuannan


On Tue, Jan 9, 2024 at 11:43 AM Xintong Song  wrote:
>
> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Mon, Jan 8, 2024 at 1:48 PM Hang Ruan  wrote:
>
> > +1(non-binding)
> >
> > Best,
> > Hang
> >
> > Rui Fan <1996fan...@gmail.com> 于2024年1月8日周一 13:04写道：
> >
> > > +1(binding)
> > >
> > > Best,
> > > Rui
> > >
> > > On Mon, Jan 8, 2024 at 1:00 PM Xuannan Su  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Thanks for all the feedback about the FLIP-405: Migrate string
> > > > configuration key to ConfigOption [1] [2].
> > > >
> > > > I'd like to start a vote for it. The vote will be open for at least 72
> > > > hours(excluding weekends,until Jan 11, 12:00AM GMT) unless there is an
> > > > objection or an insufficient number of votes.
> > > >
> > > >
> > > >
> > > > [1]
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-405%3A+Migrate+string+configuration+key+to+ConfigOption
> > > > [2] https://lists.apache.org/thread/zfw1b1g3679yn0ppjbsokfrsx9k7ybg0
> > > >
> > > >
> > > > Best,
> > > > Xuannan
> > > >
> > >
> >

Re: [FLIP-412] Add the time-consuming span of each stage when starting the Flink job to TraceReporter

[Rui Fan]

The permission is added by Piotr, thank you Piotr.

Best,
Rui

On Thu, Jan 11, 2024 at 9:15 PM Eason Qin  wrote:

> Hi all,
>
> Currently, I am working on the FLIP-412: Add the time-consuming span of
> each stage when starting the Flink job to TraceReporter[1], but I have no
> permission to update the Flink Improvement Proposals space. Can any PMC
> help me add permissions?
>
> My Jira account is easonqin and my email is qinjunzh...@gmail.com.
> My confluence account is eason.qin which is different from the Jira
> account.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-412%3A+Add+the+time-consuming+span+of+each+stage+when+starting+the+Flink+job+to+TraceReporter
>
>
> Thanks!
>

[jira] [Created] (FLINK-34067) Fix javacc warnings in flink-sql-parser

[Jim Hughes (Jira)]

Jim Hughes created FLINK-34067:
--

 Summary: Fix javacc warnings in flink-sql-parser
 Key: FLINK-34067
 URL: https://issues.apache.org/jira/browse/FLINK-34067
 Project: Flink
  Issue Type: Improvement
Reporter: Jim Hughes
Assignee: Jim Hughes


While extending the Flink SQL parser, I noticed these two warnings:

```
[INFO] --- javacc:2.4:javacc (javacc) @ flink-sql-parser ---                    
                             
Java Compiler Compiler Version 4.0 (Parser Generator)                           
                            
(type "javacc" with no arguments for help)                                      
                                                           
Reading from file 
.../flink-table/flink-sql-parser/target/generated-sources/javacc/Parser.jj . . 
.               
Note: UNICODE_INPUT option is specified. Please make sure you create the 
parser/lexer using a Reader with the correct character encoding.  
Warning: Choice conflict involving two expansions at                            
                               
         line 2043, column 13 and line 2052, column 9 respectively.             
                           
         A common prefix is: "IF"                                               
                                                            Consider using a 
lookahead of 2 for earlier expansion.                                           
     
Warning: Choice conflict involving two expansions at                            
                              
         line 2097, column 13 and line 2105, column 8 respectively.             
                            
         A common prefix is: "IF"                                               
                                                   
         Consider using a lookahead of 2 for earlier expansion.     
```

As the warning suggestions, adding `LOOKAHEAD(2)` in a few places addresses the 
warning.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34066) LagFunction throw NPE when input argument are not null

[Yunhong Zheng (Jira)]

Yunhong Zheng created FLINK-34066:
-

 Summary: LagFunction throw NPE when input argument are not null
 Key: FLINK-34066
 URL: https://issues.apache.org/jira/browse/FLINK-34066
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.18.0
Reporter: Yunhong Zheng
 Fix For: 1.19.0


LagFunction throw NPE when input argument are not null.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

[Zhanghao Chen]

Thanks for the input, Piotr. It might still be possible to make it compatible 
with the old snapshots, following the direction of 
FLINK-5290 suggested by Yu. 
I'll discuss with Yu on more details.

Best,
Zhanghao Chen

From: Piotr Nowojski 
Sent: Friday, January 12, 2024 1:55
To: Yu Chen 
Cc: Zhanghao Chen ; dev@flink.apache.org 

Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Hi,

Using unaligned checkpoints is orthogonal to this FLIP.

Yes, unaligned checkpoints are not supported for pointwise connections, so most 
of the cases go away anyway.
It is possible to switch from unchained to chained subtasks by removing a keyBy 
exchange, and this would be
a problem, but that's just one of the things that we claim that unaligned 
checkpoints do not support [1]. But as
I stated above, this is an orthogonal issue to this FLIP.

Regarding the proposal itself, generally speaking it makes sense to me as well. 
However I'm quite worried about
the compatibility and/or migration path. The:
> (v2.0) Make HasherV3 the default hasher, mark HasherV2 deprecated.

step would break the compatibility with Flink 1.xx snapshots. But as this is 
for v2.0, maybe that's not the end of
the world?

Best,
Piotrek

[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/checkpoints_vs_savepoints/#capabilities-and-limitations

czw., 11 sty 2024 o 12:10 Yu Chen 
mailto:yuchen.e...@gmail.com>> napisał(a):
Hi Zhanghao,

Actually, Stefan has done similar compatibility work in the early 
FLINK-5290[1], where he introduced the legacyStreamGraphHashers list for hasher 
backward compatibility.

We have attempted to implement a similar feature in the internal version of 
FLINK and tried to include the new hasher as part of the 
legacyStreamGraphHashers,
which would ensure that the corresponding Operator State could be found at 
restore while ignoring the chaining condition(without changing the default 
hasher).

However, we have found that such a solution may lead to some unexpected 
situations in some cases. While I have no time to find out the root cause 
recently.

If you're interested, I'd be happy to discuss it with you and try to solve the 
problem.

[1] https://issues.apache.org/jira/browse/FLINK-5290

Best,
Yu Chen



2024年1月11日 15:07，Zhanghao Chen 
mailto:zhanghao.c...@outlook.com>> 写道：

Hi Yu,

I haven't thought too much about the compatibility design before. By the nature 
of the problem, it's impossible to make V3 compatible with V2, what we can do 
is to somewhat better inform users when switching the hasher, but I don't have 
any good idea so far. Do you have any suggestions on this?

Best,
Zhanghao Chen

From: Yu Chen mailto:yuchen.e...@gmail.com>>
Sent: Thursday, January 11, 2024 13:52
To: dev@flink.apache.org 
mailto:dev@flink.apache.org>>
Cc: Piotr Nowojski mailto:piotr.nowoj...@gmail.com>>; 
zhanghao.c...@outlook.com 
mailto:zhanghao.c...@outlook.com>>
Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Hi Zhanghao,

Thanks for driving this, that’s really painful for us when we need to switch 
config `pipeline.operator-chaining`.

But I have a Concern, according to FLIP description, modifying `isChainable` 
related code in `StreamGraphHasherV2` will cause the generated operator id to 
be changed, which will result in the user unable to recover from the old state 
(old and new Operator IDs can't be mapped).
Therefore switching Hasher strategy (V2->V3 or V3->V2) will lead to an 
incompatibility, is there any relevant compatibility design considered?

Best,
Yu Chen

2024年1月10日 10:25，Zhanghao Chen 
mailto:zhanghao.c...@outlook.com>> 写道：

Hi David,

Thanks for the comments. AFAIK, unaligned checkpoints are disabled for 
pointwise connections according to [1], let's wait Piotr for confirmation. The 
issue itself is not directly related to this proposal as well. If a user 
manually specifies UIDs for each of the chained operators and has unaligned 
checkpoints enabled, we will encounter the same issue if they decide to break 
the chain on a later restart and try to recover from a retained cp.

[1] 
https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/state/checkpointing_under_backpressure/


Best,
Zhanghao Chen

From: David Morávek mailto:d...@apache.org>>
Sent: Wednesday, January 10, 2024 6:26
To: dev@flink.apache.org 
mailto:dev@flink.apache.org>>; Piotr Nowojski 
mailto:piotr.nowoj...@gmail.com>>
Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Hi Zhanghao,

Thanks for the FLIP. What you're proposing makes a lot of sense 

[jira] [Created] (FLINK-34065) Design AbstractAutoscalerStateStore to support serialize State to String

[Rui Fan (Jira)]

Rui Fan created FLINK-34065:
---

 Summary: Design AbstractAutoscalerStateStore to support serialize 
State to String
 Key: FLINK-34065
 URL: https://issues.apache.org/jira/browse/FLINK-34065
 Project: Flink
  Issue Type: Sub-task
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan


Some logic of {{KubernetesAutoScalerStateStore}} and 
{{JDBCAutoScalerStateStore}} are similar, we can share some common code.
 * {{ConfigMapStore}} and {{JDBCStore}} can be abstracted to 
{{StringStateStore}} interface

 ** They support {{{}put{}}}, {{get}} and {{remove}}
 ** The parameters of {{ConfigMapStore}} are the (JobContext, String key, 
String value).
 ** The parameters of {{JDBCStore}} are the (String jobKey, StateType 
stateType, String value).
 ** We can define a interface {{{}StringStateStore{}}}, and the parameters are 
{{{}(JobContext, StateType stateType, String value){}}}.
 * {{KubernetesAutoScalerStateStore}} and {{JDBCAutoScalerStateStore}} can be 
abstracted to {{AbstractAutoscalerStateStore}}

 ** They support serialize and compress {{Original State}} to String.
 ** {{AbstractAutoscalerStateStore}} can reuse the serialize and compress logic
 ** {{KubernetesAutoScalerStateStore}} support the limitation of stateValue
 ** We can define a parameter for {{{}AbstractAutoscalerStateStore{}}}, the 
limitation is disabled by default, and {{KubernetesAutoScalerStateStore}} can 
enable it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re:[DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

[Xuyang]

Hi, Mason.
Thanks for driving this Flip. I think it's important for external system to be 
able to
perceive the metric of the operator coordinator. +1 for it.


I just have the following minor questions and am looking forward to your reply. 
Please forgive
me if I have some misunderstandings.


1. IIRC, in a sense, operator ID and vertex ID are the same thing. The operator 
ID can
be converted from the vertex ID[1]. Therefore, it is somewhat strange to have 
both vertex
ID and operator ID in a single URL.


2. If I misunderstood the semantics of operator IDs here, then what is the 
relationship
between vertex ID and operator ID, and do we need a url like 
`/jobs//vertices//operators/`
to list all operator ids under this vertices？




[1] 
https://github.com/apache/flink/blob/7bebd2d9fac517c28afc24c0c034d77cfe2b43a6/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/OperatorID.java#L40C27-L40C27

--

Best！
Xuyang





At 2024-01-12 04:20:03, "Mason Chen"  wrote:
>Hi Devs,
>
>I'm opening this thread to discuss a short FLIP for exposing
>JobManagerOperatorMetrics via REST API [1].
>
>The current set of REST APIs make it impossible to query coordinator
>metrics. This FLIP proposes a new REST API to query the
>JobManagerOperatorMetrics.
>
>[1]
>https://cwiki.apache.org/confluence/display/FLINK/FLIP-417%3A+Expose+JobManagerOperatorMetrics+via+REST+API
>
>Best,
>Mason

[ANNOUNCE] Apache Flink-shaded 18.0 released

[Sergey Nuyanzin]

The Apache Flink community is very happy to announce the release of Apache
Flink-shaded 18.0.

The flink-shaded project contains a number of shaded dependencies for
Apache Flink.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353081

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Release Manager

[DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

[Mason Chen]

Hi Devs,

I'm opening this thread to discuss a short FLIP for exposing
JobManagerOperatorMetrics via REST API [1].

The current set of REST APIs make it impossible to query coordinator
metrics. This FLIP proposes a new REST API to query the
JobManagerOperatorMetrics.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-417%3A+Expose+JobManagerOperatorMetrics+via+REST+API

Best,
Mason

[jira] [Created] (FLINK-34064) Expose JobManagerOperatorMetrics via REST API

[Mason Chen (Jira)]

Mason Chen created FLINK-34064:
--

 Summary: Expose JobManagerOperatorMetrics via REST API
 Key: FLINK-34064
 URL: https://issues.apache.org/jira/browse/FLINK-34064
 Project: Flink
  Issue Type: Improvement
Reporter: Mason Chen


Add a REST API to fetch coordinator metrics.

[https://cwiki.apache.org/confluence/display/FLINK/FLIP-417%3A+Expose+JobManagerOperatorMetrics+via+REST+API]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

[Piotr Nowojski]

Hi,

Using unaligned checkpoints is orthogonal to this FLIP.

Yes, unaligned checkpoints are not supported for pointwise connections, so
most of the cases go away anyway.
It is possible to switch from unchained to chained subtasks by removing a
keyBy exchange, and this would be
a problem, but that's just one of the things that we claim that unaligned
checkpoints do not support [1]. But as
I stated above, this is an orthogonal issue to this FLIP.

Regarding the proposal itself, generally speaking it makes sense to me as
well. However I'm quite worried about
the compatibility and/or migration path. The:
> (v2.0) Make HasherV3 the default hasher, mark HasherV2 deprecated.

step would break the compatibility with Flink 1.xx snapshots. But as this
is for v2.0, maybe that's not the end of
the world?

Best,
Piotrek

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/checkpoints_vs_savepoints/#capabilities-and-limitations

czw., 11 sty 2024 o 12:10 Yu Chen  napisał(a):

> Hi Zhanghao,
>
> Actually, Stefan has done similar compatibility work in the early
> FLINK-5290[1], where he introduced the legacyStreamGraphHashers list for
> hasher backward compatibility.
>
> We have attempted to implement a similar feature in the internal version
> of FLINK and tried to include the new hasher as part of the
> legacyStreamGraphHashers,
> which would ensure that the corresponding Operator State could be found at
> restore while ignoring the chaining condition(without changing the default
> hasher).
>
> However, we have found that such a solution may lead to some unexpected
> situations in some cases. While I have no time to find out the root cause
> recently.
>
> If you're interested, I'd be happy to discuss it with you and try to solve
> the problem.
>
> [1] https://issues.apache.org/jira/browse/FLINK-5290
>
> Best,
> Yu Chen
>
>
>
> 2024年1月11日 15:07，Zhanghao Chen  写道：
>
> Hi Yu,
>
> I haven't thought too much about the compatibility design before. By the
> nature of the problem, it's impossible to make V3 compatible with V2, what
> we can do is to somewhat better inform users when switching the hasher, but
> I don't have any good idea so far. Do you have any suggestions on this?
>
> Best,
> Zhanghao Chen
> --
> *From:* Yu Chen 
> *Sent:* Thursday, January 11, 2024 13:52
> *To:* dev@flink.apache.org 
> *Cc:* Piotr Nowojski ; zhanghao.c...@outlook.com
> 
> *Subject:* Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID
> generation for improved state compatibility on parallelism change
>
> Hi Zhanghao,
>
> Thanks for driving this, that’s really painful for us when we need to
> switch config `pipeline.operator-chaining`.
>
> But I have a Concern, according to FLIP description, modifying
> `isChainable` related code in `StreamGraphHasherV2` will cause the
> generated operator id to be changed, which will result in the user unable
> to recover from the old state (old and new Operator IDs can't be mapped).
> Therefore switching Hasher strategy (V2->V3 or V3->V2) will lead to an
> incompatibility, is there any relevant compatibility design considered?
>
> Best,
> Yu Chen
>
> 2024年1月10日 10:25，Zhanghao Chen  写道：
>
> Hi David,
>
> Thanks for the comments. AFAIK, unaligned checkpoints are disabled for
> pointwise connections according to [1], let's wait Piotr for confirmation.
> The issue itself is not directly related to this proposal as well. If a
> user manually specifies UIDs for each of the chained operators and has
> unaligned checkpoints enabled, we will encounter the same issue if they
> decide to break the chain on a later restart and try to recover from a
> retained cp.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/state/checkpointing_under_backpressure/
>
>
> Best,
> Zhanghao Chen
> 
> From: David Morávek 
> Sent: Wednesday, January 10, 2024 6:26
> To: dev@flink.apache.org ; Piotr Nowojski <
> piotr.nowoj...@gmail.com>
> Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation
> for improved state compatibility on parallelism change
>
> Hi Zhanghao,
>
> Thanks for the FLIP. What you're proposing makes a lot of sense +1
>
> Have you thought about how this works with unaligned checkpoints in case
> you go from unchained to chained? I think it should be fine because this
> scenario should only apply to forward/rebalance scenarios where we, as far
> as I recall, force alignment anyway, so there should be no exchanges to
> snapshot. It might just work, but something to double-check. Maybe @Piotr
> Nowojski  could confirm it.
>
> Best,
> D.
>
> On Wed, Jan 3, 2024 at 7:10 AM Zhanghao Chen 
> wrote:
>
> Dear Flink devs,
>
> I'd like to start a discussion on FLIP 411: Chaining-agnostic Operator ID
> generation for improved state compatibility on parallelism change [1].
>
> Currently, when user does not explicitly set operator UIDs, the chaining
> behavior will still affect state compatibility, as the 

Re: [DISCUSS] FLIP-414: Support Retry Mechanism in RocksDBStateDataTransfer

[Piotr Nowojski]

Hi,

Thanks for the proposal. I second the Hangxiang's suggestions.

I think this might be valuable. Instead of retrying the whole checkpoint,
it will be more resource efficient
to retry upload of a single file.

Regarding re-using configuration options, a while back we introduced
`taskmanager.network.retries`
config option. It was hoped to eventually encompass things like this.

My own concern is if we should retry regardless of the exception type, or
should we focus on things like
connection loss/host unreachable? All in all, it would be better to not
retry upload if the failure was:
- `FileSystem` for given schema not found
- authorisation failed
- lack of write rights
- ...

Best,
Piotrek




czw., 11 sty 2024 o 10:35 Hangxiang Yu  napisał(a):

> Thanks for driving this.
> Retry mechanism is common when we want to get or put data by network.
> So I think it will help when checkpoint failure due to temporary network
> problems, of course it may increase a bit overhead for some other reasons.
>
> Some comments and suggestions:
> 1. Since Flink has a checkpoint mechanism to retry failed checkpoint
> coarsely, I think it looks good to me if this fine-grained retry could be
> configurable and don't change the current default mechanism.
> 2. This should work with the checkpoint procedure of all state backends,
> Could we make this config unrelated to a specific state backend (maybe
> execution.checkpointing.xxx)?  Then it could be supported by below state
> backends.
> 3. We may not need to re-implement it. There are some tools supporting the
> Retry mechanism (see RetryingExecutor and RetryPolicy in changelog dstl
> module), it's better to make them become more common tools and reuse them.
>
> On Thu, Jan 11, 2024 at 3:09 PM yue ma  wrote:
>
> > Thanks for driving this effort, xiangyu!
> > The proposal overall LGTM.
> > I just have a small question. There are other places in Flink that
> interact
> > with external storage. Should we consider adding a general retry
> mechanism
> > to them?
> >
> > xiangyu feng  于2024年1月8日周一 11:31写道：
> >
> > > Hi devs,
> > >
> > > I'm opening this thread to discuss FLIP-414: Support Retry Mechanism in
> > > RocksDBStateDataTransfer[1].
> > >
> > > Currently, there is no retry mechanism for downloading and uploading
> > > RocksDB state files. Any jittering of remote filesystem might lead to a
> > > checkpoint failure. By supporting retry mechanism in
> > > `RocksDBStateDataTransfer`, we can significantly reduce the failure
> rate
> > of
> > > checkpoint during asynchronous phrase.
> > >
> > > To make this retry mechanism configurable, we have introduced two
> options
> > > in this FLIP: `state.backend.rocksdb.checkpoint.transfer.retry.times`
> > and `
> > > state.backend.rocksdb.checkpoint.transfer.retry.interval`. The default
> > > behavior remains to be no retry will be performed in order to be
> > consistent
> > > with the original behavior.
> > >
> > > Looking forward to your feedback, thanks.
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-414%3A+Support+Retry+Mechanism+in+RocksDBStateDataTransfer
> > >
> > > Best regards,
> > > Xiangyu Feng
> > >
> >
> >
> > --
> > Best,
> > Yue
> >
>
>
> --
> Best,
> Hangxiang.
>

[jira] [Created] (FLINK-34063) When snapshot compression is enabled, rescaling of a source operator leads to some splits getting lost

[Ivan Burmistrov (Jira)]

Ivan Burmistrov created FLINK-34063:
---

 Summary: When snapshot compression is enabled, rescaling of a 
source operator leads to some splits getting lost
 Key: FLINK-34063
 URL: https://issues.apache.org/jira/browse/FLINK-34063
 Project: Flink
  Issue Type: Bug
 Environment: Can be reproduced in any environment. The most important 
thing is to enable snapshot compression.
Reporter: Ivan Burmistrov
 Attachments: image-2024-01-11-16-27-09-066.png, 
image-2024-01-11-16-30-47-466.png

h2. Backstory

We've been experimenting with Autoscaling on the Flink 1.18 and faced a pretty 
nasty bug. 

The symptoms on our production system were as following. After a while after 
deploying a job with autoscaler it started accumulating Kafka lag, and this 
could only be observed via external lag measurement - from inside Flink 
(measured by
{{_KafkaSourceReader_KafkaConsumer_records_lag_max_}} metric) the lag was OK:
!image-2024-01-11-16-27-09-066.png|width=887,height=263!

After some digging, it turned out that the job has lost some Kafka partitions - 
i.e. it stopped consuming from them, “forgot” about their existence. That’s why 
from the Flink’s perspective everything was fine - the lag was growing on the 
partitions Flink no longer knew about.

This was visible on a metric called “Assigned partitions” 
(KafkaSourceReader_KafkaConsumer_assigned_partitions):
!image-2024-01-11-16-30-47-466.png|width=1046,height=254!


We see on the chart that the job used to know about 20 partitions, and then 
this number got dropped to 16.

This drop has been quickly connected to the job’s scaling events. Or, more 
precisely, to the scaling of the source operator - with almost 100% probability 
any scaling of the source operator led to partitions loss.
h2. Investigation

We've conducted the investigation. We use the latest Kubernetes operator and 
deploy jobs with Native Kubernetes.

The reproducing scenario we used for investigation:
 * Launch a job with source operator parallelism = 4, enable DEBUG logging
 * Wait until it takes the first checkpoint
 * Scale-up the source operator to say 5 (no need to wait for autoscaling, it 
can be done via Flink UI)
 * Wait until the new checkpoint is taken
 * Scale-down the source operator to 3

These simple actions with almost 100% probability led to some partitions get 
lost.

After that we've downloaded all the logs and inspected them. Noticed these 
strange records in logs:


{code:java}
{"timestamp":1704415753166,"is_logging_enabled":"false","logger_id":"org.apache.flink.streaming.api.operators.AbstractStreamOperator","log_level":"INFO","message":"Restoring
 state for 4 split(s) to reader.","service_name":"data-beaver"} 
{"timestamp":1704415753166,"is_logging_enabled":"false","logger_id":"org.apache.flink.connector.base.source.reader.SourceReaderBase","log_level":"INFO","message":"Adding
 split(s) to reader: 
[
[Partition: eventmesh-video-play-v1-6, StartingOffset: 1964306414, 
StoppingOffset: -9223372036854775808], 
[Partition: eventmesh-video-play-v1-19, StartingOffset: 1963002538, 
StoppingOffset: -9223372036854775808], 
[Partition: eventmesh-video-play-v1-6, StartingOffset: 1964306414, 
StoppingOffset: -9223372036854775808], 
[Partition: eventmesh-video-play-v1-19, StartingOffset: 1963002538, 
StoppingOffset: -9223372036854775808]]", "service_name":"data-beaver"}{code}
We see that some task being restored with 4 splits, however actual splits have 
duplicates - we see that in reality 2 unique partitions have been added 
({_}eventmesh-video-play-v1-6{_} and {_}eventmesh-video-play-v1-19{_}).

Digging into the code and the logs a bit more, log lines like this started 
looking suspicious:

 
{code:java}
{"timestamp":1704415753165,"is_logging_enabled":"false","logger_id":"org.apache.flink.runtime.state.TaskStateManagerImpl","log_level":"DEBUG",
 "message":"Operator 156a1ebbc1936f7d4558c8070b35ba93 has remote state 
SubtaskState{operatorStateFromBackend=StateObjectCollection{ 
[OperatorStateHandle{stateNameToPartitionOffsets={SourceReaderState=StateMetaInfo{offsets=[244,
 244], distributionMode=SPLIT_DISTRIBUTE}}, 
delegateStateHandle=ByteStreamStateHandle{handleName='gs://data-beaver/checkpoints/moj-tj-dummy-partition-loss-debug-v1/6e1ba15b1b5bedda64836ff48ed1c264/chk-3/fadb4f23-85dd-4048-b466-94c1c5329dd3',
 dataBytes=328}}, 
OperatorStateHandle{stateNameToPartitionOffsets={SourceReaderState=StateMetaInfo{offsets=[244,
 244], distributionMode=SPLIT_DISTRIBUTE}}, 
delegateStateHandle=ByteStreamStateHandle{handleName='gs://data-beaver/checkpoints/moj-tj-dummy-partition-loss-debug-v1/6e1ba15b1b5bedda64836ff48ed1c264/chk-3/102aa50b-78c2-457e-9a2f-0055f1dbeb98',
 dataBytes=328}}]}, operatorStateFromStream=StateObjectCollection{[]}, 
keyedStateFromBackend=StateObjectCollection{[]}, 
keyedStateFromStream=StateObjectCollection{[]}, 

Re: [DISCUSS] FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving

[Becket Qin]

Hi Qingsheng,

Thanks for the comment. I think the initial idea is to hide the queue
completely from the users, i.e. make FutureCompletingBlockingQueue class
internal. If it is OK to expose the class to the users, then just returning
the queue sounds reasonable to me.

Thanks,

Jiangjie (Becket) Qin

On Wed, Jan 10, 2024 at 10:39 PM Hongshun Wang 
wrote:

> Hi Qingsheng,
>
>
> I agree with you that it would be clearer to have a new interface that
> extracts the SplitFetcher creation and management logic from the current
> SplitFetcherManager. However, extensive modifications to the interface may
> influence a lot and cause compatibility issues. Perhaps we can consider
> doing it later, rather than in this FLIP.
>
>
> Adding a new internal method, SplitFetcherManager#getQueue(), to
> SourceReaderBase seems to be a better option than exposing methods like
> poll and notifyAvailable on SplitFetcherManager.
>
>
> I have taken this valuable suggestion and updated the FLIP accordingly.
>
>
> Thanks,
>
> Hongshun
>
> On Thu, Jan 11, 2024 at 2:09 PM Qingsheng Ren  wrote:
>
>> Hi Hongshun and Becket,
>>
>> Sorry for being late in the discussion! I went through the entire FLIP
>> but I still have some concerns about the new SplitFetcherManager.
>>
>> First of all I agree that we should hide the elementQueue from connector
>> developers. This could simplify the interface exposed to developers so that
>> they can focus on the interaction with external systems.
>>
>> However in the current FLIP, SplitFetcherManager exposes 4 more methods,
>> poll / getAvailabilityFuture / notifyAvailable / noAvailableElement, which
>> are tightly coupled with the implementation of the elementQueue. The naming
>> of these methods look weird to me, like what does it mean to "poll from a
>> SplitFetcherManager" / "notify a SplitFetcherManager available"? To clarify
>> these methods we have to explain to developers that "well we hide a queue
>> inside SplitFetcherMamager and the poll method is actually polling from the
>> queue". I'm afraid these methods will implicitly expose the concept and the
>> implementation of the queue to developers.
>>
>> I think a cleaner solution would be having a new interface that extracts
>> SplitFetcher creating and managing logic from the current
>> SplitFetcherManager, but having too many concepts might make the entire
>> Source API even harder to understand. To make a compromise, I'm considering
>> only exposing constructors of SplitFetcherManager as public APIs, and
>> adding a new internal method SplitFetcherManager#getQueue() for
>> SourceReaderBase (well it's a bit hacky I admit but I think exposing
>> methods like poll and notifyAvailable on SplitFetcherManager is even
>> worth). WDTY?
>>
>> Thanks,
>> Qingsheng
>>
>> On Thu, Dec 21, 2023 at 8:36 AM Becket Qin  wrote:
>>
>>> Hi Hongshun,
>>>
>>> I think the proposal in the FLIP is basically fine. A few minor comments:
>>>
>>> 1. In FLIPs, we define all the user-sensible changes as public
>>> interfaces.
>>> The public interface section should list all of them. So, the code blocks
>>> currently in the proposed changes section should be put into the public
>>> interface section instead.
>>>
>>> 2. It would be good to put all the changes of one class together. For
>>> example, for SplitFetcherManager, we can say:
>>> - Change SplitFetcherManager from Internal to PublicEvolving.
>>> - Deprecate the old constructor exposing the
>>> FutureCompletingBlockingQueue, and add new constructors as replacements
>>> which creates the FutureCompletingBlockingQueue instance internally.
>>> - Add a few new methods to expose the functionality of the internal
>>> FutureCompletingBlockingQueue via the SplitFetcherManager.
>>>And then follows the code block containing all the changes above.
>>> Ideally, the changes should come with something like "// <-- New", so
>>> that it is. easier to be found.
>>>
>>> 3. In the proposed changes section, it would be good to add some more
>>> detailed explanation of the idea behind the public interface changes. So
>>> even people new to Flink can understand better how exactly the interface
>>> changes will help fulfill the motivation. For example, regarding the
>>> constructor signature change, we can say the following. We can mention a
>>> few things in this section:
>>> - By exposing the SplitFetcherManager / SingleThreadFetcheManager, by
>>> implementing addSplits() and removeSplits(), connector developers can
>>> easily create their own threading models in the SourceReaderBase.
>>> - Note that the SplitFetcher constructor is package private, so users
>>> can only create SplitFetchers via
>>> SplitFetcherManager.createSplitFetcher().
>>> This ensures each SplitFetcher is always owned by the
>>> SplitFetcherManager.
>>> - This FLIP essentially embedded the element queue (a
>>> FutureCompletingBlockingQueue) instance into the SplitFetcherManager.
>>> This
>>> hides the element queue from the connector 

[FLIP-412] Add the time-consuming span of each stage when starting the Flink job to TraceReporter

[Eason Qin]

Hi all,

Currently, I am working on the FLIP-412: Add the time-consuming span of
each stage when starting the Flink job to TraceReporter[1], but I have no
permission to update the Flink Improvement Proposals space. Can any PMC
help me add permissions?

My Jira account is easonqin and my email is qinjunzh...@gmail.com.
My confluence account is eason.qin which is different from the Jira account.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-412%3A+Add+the+time-consuming+span+of+each+stage+when+starting+the+Flink+job+to+TraceReporter


Thanks!

Re: [VOTE] FLIP-407: Improve Flink Client performance in interactive scenarios

[Rui Fan]

+1 binding

Best,
Rui


On Thu, 11 Jan 2024 at 19:45, xiangyu feng  wrote:

> Hi all,
>
> I would like to start the vote for FLIP-407: Improve Flink Client
> performance in interactive scenarios[1].
> This FLIP was discussed in this thread [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> Regards,
> Xiangyu
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-407%3A+Improve+Flink+Client+performance+in+interactive+scenarios
> [2] https://lists.apache.org/thread/ccsv66ygffgqbv956bnknbpllj4t24kj
>

Re: [VOTE] Release flink-connector-hive, release candidate #1

[Sergey Nuyanzin]

Great that it is resolved
and thanks a lot for checking

On Thu, Jan 11, 2024 at 8:31 AM Hang Ruan  wrote:

> Hi, Sergey.
>
> Thanks for the quick reply.
>
> I try to package it in other pc with jdk8 and it succeeds. Please ignore
> it. It seems like some errors in my environment.
>
> Best,
> Hang
>
> Sergey Nuyanzin  于2024年1月11日周四 14:31写道：
>
> > Hi Hang
> >
> > thanks for checking
> > yes, it could be packaged with jdk8, moreover jdk8 is checked in ci
> > for instance here ci for the commit tagged with v3.0.0-rc1 [1]
> >
> > the strange thing in the output that you've provided is
> > >org.apache.flink:flink-connector-hive_2.12:jar:3.0.0: Could not find
> > > artifact jdk.tools:jdk.tools:jar:1.8 at specified path
> /Library/Internet
> > > Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/../lib/tools.jar
> >
> > there are no such dependencies in poms,
> > could it happen that there is some specific configuration on the machine
> > you used for that?
> > Can you please check it on another setup?
> >
> >
> > [1]
> https://github.com/apache/flink-connector-hive/actions/runs/7479158667
> >
> >
> > On Thu, Jan 11, 2024 at 4:44 AM Hang Ruan 
> wrote:
> >
> > > Hi, Sergey Nuyanzin.
> > >
> > > Thanks for driving this.
> > >
> > > I try to package the source with jdk8 and it will cause an error as
> > > follows.
> > >
> > > [INFO]
> > >
> 
> > > [INFO] BUILD FAILURE
> > > [INFO]
> > >
> 
> > > [INFO] Total time:  4.621 s
> > > [INFO] Finished at: 2024-01-11T11:34:30+08:00
> > > [INFO]
> > >
> 
> > > [ERROR] Failed to execute goal on project flink-connector-hive_2.12:
> > Could
> > > not resolve dependencies for project
> > > org.apache.flink:flink-connector-hive_2.12:jar:3.0.0: Could not find
> > > artifact jdk.tools:jdk.tools:jar:1.8 at specified path
> /Library/Internet
> > > Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/../lib/tools.jar ->
> [Help
> > 1]
> > > [ERROR]
> > > [ERROR] To see the full stack trace of the errors, re-run Maven with
> the
> > -e
> > > switch.
> > > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > > [ERROR]
> > > [ERROR] For more information about the errors and possible solutions,
> > > please read the following articles:
> > > [ERROR] [Help 1]
> > >
> > >
> >
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> > > [ERROR]
> > > [ERROR] After correcting the problems, you can resume the build with
> the
> > > command
> > > [ERROR]   mvn  -rf :flink-connector-hive_2.12
> > >
> > > I see that the 'Building the Apache Flink Hive Connector from Source'
> > part
> > > in README requires the Java 11. I am not sure whether this could be
> > treated
> > > as an error.
> > > Does the flink-connector-hive support to be packaged with jdk8 now?
> > >
> > > Best,
> > > Hang
> > >
> > > Jiabao Sun  于2024年1月11日周四 11:35写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > - Validated checksum hash
> > > > - Verified signature
> > > > - Verified web PR
> > > > - Verified tags
> > > >
> > > > Best,
> > > > Jiabao
> > > >
> > > >
> > > > > 2024年1月11日 11:25，Hang Ruan  写道：
> > > > >
> > > > > Sorry that I make a mistake. I build the source with Maven and
> jdk11.
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Hang Ruan  于2024年1月11日周四 11:13写道：
> > > > >
> > > > >> +1 (non-binding)
> > > > >>
> > > > >> - Validated checksum hash
> > > > >> - Verified signature
> > > > >> - Verified that no binaries exist in the source archive
> > > > >> - Build the source with Maven and jdk8
> > > > >> - Verified web PR
> > > > >> - Verified that the flink-connector-base is not packaged in hive
> > > > connector
> > > > >>
> > > > >> Best,
> > > > >> Hang
> > > > >>
> > > > >> Sergey Nuyanzin  于2024年1月11日周四 06:19写道：
> > > > >>
> > > > >>> Hi everyone,
> > > > >>> Please review and vote on the release candidate #1 for the
> version
> > > > 3.0.0,
> > > > >>> as follows:
> > > > >>> [ ] +1, Approve the release
> > > > >>> [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > >>>
> > > > >>> This version is compatible with Flink 1.18.x
> > > > >>>
> > > > >>> The complete staging area is available for your review, which
> > > includes:
> > > > >>> * JIRA release notes [1],
> > > > >>> * the official Apache source release to be deployed to
> > > dist.apache.org
> > > > >>> [2],
> > > > >>> which are signed with the key with fingerprint F752 9FAE 2481
> 1A5C
> > > 0DF3
> > > > >>> CA74 1596 BBF0 7268 35D8 [3],
> > > > >>> * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > >>> * source code tag v3.0.0-rc1 [5],
> > > > >>> * website pull request listing the new release [6].
> > > > >>>
> > > > >>> The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > > >>> approval, with 

[VOTE] FLIP-407: Improve Flink Client performance in interactive scenarios

[xiangyu feng]

Hi all,

I would like to start the vote for FLIP-407: Improve Flink Client
performance in interactive scenarios[1].
This FLIP was discussed in this thread [2].

The vote will be open for at least 72 hours unless there is an objection or
insufficient votes.

Regards,
Xiangyu

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-407%3A+Improve+Flink+Client+performance+in+interactive+scenarios
[2] https://lists.apache.org/thread/ccsv66ygffgqbv956bnknbpllj4t24kj

[jira] [Created] (FLINK-34062) Propagate in the surefire-plugin configuration for Java 21

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-34062:
-

 Summary: Propagate  in the surefire-plugin 
configuration for Java 21
 Key: FLINK-34062
 URL: https://issues.apache.org/jira/browse/FLINK-34062
 Project: Flink
  Issue Type: Sub-task
Reporter: Matthias Pohl






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [Discuss] FLIP-407: Improve Flink Client performance in interactive scenarios

[xiangyu feng]

Hi devs,

Thanks for all the feedback. If there are no more comments, I would like to
start a vote for this FLIP, thanks again!

Best,
Xiangyu Feng

Weihua Hu  于2024年1月9日周二 14:45写道：

> Thanks for proposing this FLIP.
>
> Experiments have shown that it significantly enhances the real-time query
> experience.
> +1 for this.
>
> Best,
> Weihua
>
>
> On Mon, Jan 8, 2024 at 5:19 PM Rui Fan <1996fan...@gmail.com> wrote:
>
>> Thanks Xiangyu for the quick update!
>>
>> LGTM
>>
>> Best,
>> Rui
>>
>> On Mon, Jan 8, 2024 at 4:27 PM xiangyu feng  wrote:
>>
>> > Hi Rui and Yong,
>> >
>> > Thx for ur reply.
>> >
>> > My initial attention here is that for short-lived jobs under high QPS: a
>> > fixed delay retry strategy will cause extra resource waste and not
>> flexible
>> > enough, an exponential-backoff strategy might significantly increase the
>> > query latency since the interval time grows too fast. An
>> incremental-delay
>> > strategy could be balanced between resource consumption and short-query
>> > latency.
>> >
>> > With a second thought,  an exponential-delay retry strategy with a
>> > configurable multiplier option can also achieve this goal. By setting
>> the
>> > default value of multiplier to 1, we can be consistent with the original
>> > behavior and reduce the configuration items at the same time.
>> >
>> > I've updated this FLIP accordingly, look forward to your feedback.
>> >
>> > Regards,
>> > Xiangyu Feng
>> >
>> >
>> > Rui Fan <1996fan...@gmail.com> 于2024年1月8日周一 15:29写道：
>> >
>> >> Only one strategy is fine to me.
>> >>
>> >> When the multiplier is set to 1, the exponential-delay will become
>> >> fixed-delay.
>> >> So fixed-delay may not be needed.
>> >>
>> >> Best,
>> >> Rui
>> >>
>> >> On Mon, Jan 8, 2024 at 2:17 PM Yong Fang  wrote:
>> >>
>> >> > I agree with @Rui that the current configuration for Flink Client is
>> a
>> >> > little complex. Can we just provide one strategy with less
>> configuration
>> >> > items for all scenarios?
>> >> >
>> >> > Best,
>> >> > Fang Yong
>> >> >
>> >> > On Mon, Jan 8, 2024 at 11:19 AM Rui Fan <1996fan...@gmail.com>
>> wrote:
>> >> >
>> >> > > Thanks xiangyu for driving this proposal! And sorry for the
>> >> > > late reply.
>> >> > >
>> >> > > Overall looks good to me, I only have some minor questions:
>> >> > >
>> >> > > 1. Do we need to introduce 3 collect strategies in the first
>> version?
>> >> > >
>> >> > > Large and comprehensive configuration items will bring
>> >> > > additional learning costs and usage costs to users. I tend to
>> >> > > provide users with out-of-the-box parameters and 2 collect
>> >> > > strategies may be enough for users.
>> >> > >
>> >> > > IIUC, there is no big difference between exponential-delay and
>> >> > > incremental-delay, especially the default parameters provided.
>> >> > > I wonder could we provide a multiplier for exponential-delay
>> strategy
>> >> > > and removing the incremental-delay strategy?
>> >> > >
>> >> > > Of course, if you think multiplier option is not needed based on
>> >> > > your production experience, it's totally fine for me. Simple is
>> >> better.
>> >> > >
>> >> > > 2. Which strategy do you think is best in mass production?
>> >> > >
>> >> > > I'm working on FLIP-364[1], it's related to Flink failover restart
>> >> > > strategy. IIUC, when one cluster only has a few flink jobs,
>> >> > > fixed-delay is fine. It guarantees minimal latency without too
>> >> > > much stress. But if one cluster has too many jobs, fixed-delay
>> >> > > may not be stable.
>> >> > >
>> >> > > Do you think exponential-delay is better than fixed delay in this
>> >> > > scenario? And which strategy is used in your production for now?
>> >> > > Would you mind sharing it?
>> >> > >
>> >> > > Looking forwarding to your opinion~
>> >> > >
>> >> > > Best,
>> >> > > Rui
>> >> > >
>> >> > > On Sat, Jan 6, 2024 at 5:54 PM xiangyu feng 
>> >> > wrote:
>> >> > >
>> >> > > > Hi all,
>> >> > > >
>> >> > > > Thanks for the comments.
>> >> > > >
>> >> > > > If there is no further comment, we will open the voting thread
>> next
>> >> > week.
>> >> > > >
>> >> > > > Regards,
>> >> > > > Xiangyu
>> >> > > >
>> >> > > > Zhanghao Chen  于2024年1月3日周三 16:46写道：
>> >> > > >
>> >> > > > > Thanks for driving this effort on improving the interactive use
>> >> > > > experience
>> >> > > > > of Flink. The proposal overall looks good to me.
>> >> > > > >
>> >> > > > > Best,
>> >> > > > > Zhanghao Chen
>> >> > > > > 
>> >> > > > > From: xiangyu feng 
>> >> > > > > Sent: Tuesday, December 26, 2023 16:51
>> >> > > > > To: dev@flink.apache.org 
>> >> > > > > Subject: [Discuss] FLIP-407: Improve Flink Client performance
>> in
>> >> > > > > interactive scenarios
>> >> > > > >
>> >> > > > > Hi devs,
>> >> > > > >
>> >> > > > > I'm opening this thread to discuss FLIP-407: Improve Flink
>> Client
>> >> > > > > performance in interactive scenarios. The POC test results and
>> >> design
>> >> > > doc
>> >> > > > > can be found at: 

[jira] [Created] (FLINK-34061) Add explicit exclusion of JDK-related excluded groups in the surefire-plugin config

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-34061:
-

 Summary: Add explicit exclusion of JDK-related excluded groups in 
the surefire-plugin config
 Key: FLINK-34061
 URL: https://issues.apache.org/jira/browse/FLINK-34061
 Project: Flink
  Issue Type: Bug
  Components: Build System
Affects Versions: 1.18.0, 1.19.0
Reporter: Matthias Pohl


The  property of the surefire-plugin doesn't support merging 
groups, i.e. if -Pjava11 and -Pjava17 are set only the @FailsWithJava17 groups 
will be considered. This hasn't been detected because we don't have 
Java11-specific test exclusions anymore. Still, we should fix this in the 
parent pom.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

[Yu Chen]

Hi Zhanghao,

Actually, Stefan has done similar compatibility work in the early 
FLINK-5290[1], where he introduced the legacyStreamGraphHashers list for hasher 
backward compatibility.

We have attempted to implement a similar feature in the internal version of 
FLINK and tried to include the new hasher as part of the 
legacyStreamGraphHashers, 
which would ensure that the corresponding Operator State could be found at 
restore while ignoring the chaining condition(without changing the default 
hasher). 

However, we have found that such a solution may lead to some unexpected 
situations in some cases. While I have no time to find out the root cause 
recently.

If you're interested, I'd be happy to discuss it with you and try to solve the 
problem.

[1] https://issues.apache.org/jira/browse/FLINK-5290

Best,
Yu Chen



> 2024年1月11日 15:07，Zhanghao Chen  写道：
> 
> Hi Yu,
> 
> I haven't thought too much about the compatibility design before. By the 
> nature of the problem, it's impossible to make V3 compatible with V2, what we 
> can do is to somewhat better inform users when switching the hasher, but I 
> don't have any good idea so far. Do you have any suggestions on this?
> 
> Best,
> Zhanghao Chen
> From: Yu Chen 
> Sent: Thursday, January 11, 2024 13:52
> To: dev@flink.apache.org 
> Cc: Piotr Nowojski ; zhanghao.c...@outlook.com 
> 
> Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
> improved state compatibility on parallelism change
>  
> Hi Zhanghao,
> 
> Thanks for driving this, that’s really painful for us when we need to switch 
> config `pipeline.operator-chaining`.
> 
> But I have a Concern, according to FLIP description, modifying `isChainable` 
> related code in `StreamGraphHasherV2` will cause the generated operator id to 
> be changed, which will result in the user unable to recover from the old 
> state (old and new Operator IDs can't be mapped). 
> Therefore switching Hasher strategy (V2->V3 or V3->V2) will lead to an 
> incompatibility, is there any relevant compatibility design considered?
> 
> Best,
> Yu Chen
> 
>> 2024年1月10日 10:25，Zhanghao Chen  写道：
>> 
>> Hi David,
>> 
>> Thanks for the comments. AFAIK, unaligned checkpoints are disabled for 
>> pointwise connections according to [1], let's wait Piotr for confirmation. 
>> The issue itself is not directly related to this proposal as well. If a user 
>> manually specifies UIDs for each of the chained operators and has unaligned 
>> checkpoints enabled, we will encounter the same issue if they decide to 
>> break the chain on a later restart and try to recover from a retained cp.
>> 
>> [1] 
>> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/state/checkpointing_under_backpressure/
>> 
>> 
>> Best,
>> Zhanghao Chen
>> 
>> From: David Morávek 
>> Sent: Wednesday, January 10, 2024 6:26
>> To: dev@flink.apache.org ; Piotr Nowojski 
>> 
>> Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation 
>> for improved state compatibility on parallelism change
>> 
>> Hi Zhanghao,
>> 
>> Thanks for the FLIP. What you're proposing makes a lot of sense +1
>> 
>> Have you thought about how this works with unaligned checkpoints in case
>> you go from unchained to chained? I think it should be fine because this
>> scenario should only apply to forward/rebalance scenarios where we, as far
>> as I recall, force alignment anyway, so there should be no exchanges to
>> snapshot. It might just work, but something to double-check. Maybe @Piotr
>> Nowojski  could confirm it.
>> 
>> Best,
>> D.
>> 
>> On Wed, Jan 3, 2024 at 7:10 AM Zhanghao Chen 
>> wrote:
>> 
>>> Dear Flink devs,
>>> 
>>> I'd like to start a discussion on FLIP 411: Chaining-agnostic Operator ID
>>> generation for improved state compatibility on parallelism change [1].
>>> 
>>> Currently, when user does not explicitly set operator UIDs, the chaining
>>> behavior will still affect state compatibility, as the generation of the
>>> Operator ID is dependent on its chained output nodes. For example, a simple
>>> source->sink DAG with source and sink chained together is state
>>> incompatible with an otherwise identical DAG with source and sink unchained
>>> (either because the parallelisms of the two ops are changed to be unequal
>>> or chaining is disabled). This greatly limits the flexibility to perform
>>> chain-breaking/building for performance tuning.
>>> 
>>> The dependency on chained output nodes for Operator ID generation can be
>>> traced back to Flink 1.2. It is unclear at this point on why chained output
>>> nodes are involved in the algorithm, but the following history background
>>> might be related: prior to Flink 1.3, Flink runtime takes the snapshots by
>>> the operator ID of the first vertex in a chain, so it somewhat makes sense
>>> to include chained output nodes into the algorithm as
>>> chain-breaking/building is expected to break state-compatibility anyway.
>>> 
>>> Given that 

[DISCUSS] [connectors] FileSystem connector - restore from historical checkpoint

[Сергей Парышев]

Hi devs! I have question about filesystem (parquet) sink  connector. When 
compaction is enabled and job restoring from historical checkpoint then job 
canceling with FileNotFoundException: can't find old .uncompacted file, when 
compaction is disabled and restoring from historical checkpoint fresh files 
will not deleted. It means that filesystem connector doesn't provide exactly 
once semantic when max retain checkpoint is greater than 1, right?
 

Re: [DISCUSS] FLIP-414: Support Retry Mechanism in RocksDBStateDataTransfer

[Hangxiang Yu]

Thanks for driving this.
Retry mechanism is common when we want to get or put data by network.
So I think it will help when checkpoint failure due to temporary network
problems, of course it may increase a bit overhead for some other reasons.

Some comments and suggestions:
1. Since Flink has a checkpoint mechanism to retry failed checkpoint
coarsely, I think it looks good to me if this fine-grained retry could be
configurable and don't change the current default mechanism.
2. This should work with the checkpoint procedure of all state backends,
Could we make this config unrelated to a specific state backend (maybe
execution.checkpointing.xxx)?  Then it could be supported by below state
backends.
3. We may not need to re-implement it. There are some tools supporting the
Retry mechanism (see RetryingExecutor and RetryPolicy in changelog dstl
module), it's better to make them become more common tools and reuse them.

On Thu, Jan 11, 2024 at 3:09 PM yue ma  wrote:

> Thanks for driving this effort, xiangyu!
> The proposal overall LGTM.
> I just have a small question. There are other places in Flink that interact
> with external storage. Should we consider adding a general retry mechanism
> to them?
>
> xiangyu feng  于2024年1月8日周一 11:31写道：
>
> > Hi devs,
> >
> > I'm opening this thread to discuss FLIP-414: Support Retry Mechanism in
> > RocksDBStateDataTransfer[1].
> >
> > Currently, there is no retry mechanism for downloading and uploading
> > RocksDB state files. Any jittering of remote filesystem might lead to a
> > checkpoint failure. By supporting retry mechanism in
> > `RocksDBStateDataTransfer`, we can significantly reduce the failure rate
> of
> > checkpoint during asynchronous phrase.
> >
> > To make this retry mechanism configurable, we have introduced two options
> > in this FLIP: `state.backend.rocksdb.checkpoint.transfer.retry.times`
> and `
> > state.backend.rocksdb.checkpoint.transfer.retry.interval`. The default
> > behavior remains to be no retry will be performed in order to be
> consistent
> > with the original behavior.
> >
> > Looking forward to your feedback, thanks.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-414%3A+Support+Retry+Mechanism+in+RocksDBStateDataTransfer
> >
> > Best regards,
> > Xiangyu Feng
> >
>
>
> --
> Best,
> Yue
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-34060) Migrate UserDefinedTableAggFunctions to JavaUserDefinedTableAggFunctions

[Jane Chan (Jira)]

Jane Chan created FLINK-34060:
-

 Summary: Migrate UserDefinedTableAggFunctions to 
JavaUserDefinedTableAggFunctions
 Key: FLINK-34060
 URL: https://issues.apache.org/jira/browse/FLINK-34060
 Project: Flink
  Issue Type: Technical Debt
  Components: Table SQL / Runtime
Affects Versions: 1.19.0
Reporter: Jane Chan
Assignee: Jane Chan


The issue is discovered when testing FLINK-31788.


The Top3 function emits a tuple of (entry.getKey, entry.getKey) 
[UserDefinedTableAggFunctions.scala#L127|https://github.com/apache/flink/blob/907d0f32126b9f8acfc80f3f4098e71cb37f0e37/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/UserDefinedTableAggFunctions.scala#L127],
 which is peculiar.

Meanwhile, consider getting the scala-free goal; it's time to migrate this 
class to the `JavaUserDefinedTableAggFunctions`, and revisit the implementation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-415: Introduce a new join operator to support minibatch

[Benchao Li]

> the change might not be supposed for the downstream of the job which requires 
> details of changelog

Could you elaborate on this a bit? I've never met such kinds of
requirements before, I'm curious what is the scenario that requires
this.

shuai xu  于2024年1月11日周四 13:08写道：
>
> Thanks for your response, Benchao.
>
> Here is my thought on the newly added option.
> Users' current jobs are running on a version without minibatch join. If the 
> existing option to enable minibatch join is utilized, then when users' jobs 
> are migrated to the new version, the internal behavior of the join operation 
> within the jobs will change. Although the semantic of changelog emitted by 
> the Join operator is eventual consistency, the change might not be supposed 
> for the downstream of the job which requires details of changelog. This newly 
> added option also refers to 
> 'table.exec.deduplicate.mini-batch.compact-changes-enabled'.
>
> As for the implementation，The new operator shares the state of the original 
> operator and it merely has an additional minibatch for storing records to do 
> some optimization. The storage remains consistent, and there is minor 
> modification to the computational logic.
>
> Best,
> Xu Shuai
>
> > 2024年1月10日 22:56，Benchao Li  写道：
> >
> > Thanks shuai for driving this, mini-batch Join is a very useful
> > optimization, +1 for the general idea.
> >
> > Regarding the configuration
> > "table.exec.stream.join.mini-batch-enabled", I'm not sure it's really
> > necessary. The semantic of changelog emitted by the Join operator is
> > eventual consistency, so there is no much difference between original
> > Join and mini-batch Join from this aspect. Besides, introducing more
> > options would make it more complex for users, harder to understand and
> > maintain, which we should be careful about.
> >
> > One thing about the implementation, could you make the new operator
> > share the same state definition with the original one?
> >
> > shuai xu  于2024年1月10日周三 21:23写道：
> >>
> >> Hi devs,
> >>
> >> I’d like to start a discussion on FLIP-415: Introduce a new join operator 
> >> to support minibatch[1].
> >>
> >> Currently, when performing cascading connections in Flink, there is a pain 
> >> point of record amplification. Every record join operator receives would 
> >> trigger join process. However, if records of +I and -D matches , they 
> >> could be folded to reduce two times of join process. Besides, records of  
> >> -U +U might output 4 records in which two records are redundant when 
> >> encountering outer join .
> >>
> >> To address this issue, this FLIP introduces a new  
> >> MiniBatchStreamingJoinOperator to achieve batch processing which could 
> >> reduce number of outputting redundant messages and avoid unnecessary join 
> >> processes.
> >> A new option is added to control the operator to avoid influencing 
> >> existing jobs.
> >>
> >> Please find more details in the FLIP wiki document [1]. Looking
> >> forward to your feedback.
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> >>
> >> Best,
> >> Xu Shuai
> >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
>


-- 

Best,
Benchao Li