Re: [DISCUSS] Drop python 3.7 support in 1.19

[Xingbo Huang]

Hi Gyala and Gabor,

Pyflink has provided arm64 wheel packages for Apple silicon since 1.16[1].
The use of Miniconda is only related to ci testing and packaging on linux
platform, and building mac platform wheels are dependent on
cibuildwheel[2]. So I guess you want to run ci tests  on the m1
environment, but the current version of miniconda cannot meet this
requirement, so there is a pre-step that must drop python 3.7?

[1] https://pypi.org/project/apache-flink/1.16.2/#files
[2]
https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml#L30

Best,
Xingbo

Gabor Somogyi  于2023年9月6日周三 14:36写道：

> Hi Xingbo,
>
> *Constraint:*
> I personally not found any miniconda version which provides arm64 support
> together with python 3.7.
> [image: image.png]
>
> At the moment I think new platform support means 3.7 drop.
>
> I fully to agree with Gyula, if we start now maybe we can release it in
> half a year however *3.7 active support already ended in 27 Jun 2020*.
> At the moment any python development/test execution on MacOS M1 is just
> not working as-is just like any kind of python test execution on any ARM
> CPU.
>
> Gains:
> * We can release a working version in half a year hopefully and not
> shifting support to 1+ year
> * MacOS M1 local development would work finally which is essential for
> user engagement
> * It would be possible to execute python tests on ARM64 machines
> * We can shake up the python development story because it's not the most
> loved area
>
> BR,
> G
>
>
> On Wed, Sep 6, 2023 at 8:06 AM Gyula Fóra  wrote:
>
>> Hi Xingbo!
>>
>> I think we have to analyze what we gain by dropping 3.7 and upgrading to a
>> miniconda version with a multiarch support.
>>
>> If this is what we need to get Apple silicon support then I think it's
>> worth doing it already in 1.19. Keep in mind that 1.18 is not even
>> released
>> yet so if we delay this to 1.20 that is basically 1 year from now.
>> Making this change can increase the adoption instantly if we enable new
>> platforms.
>>
>> Cheers,
>> Gyula
>>
>> On Wed, Sep 6, 2023 at 4:46 AM Xingbo Huang  wrote:
>>
>> > Hi Gabor,
>> >
>> > Thanks for bringing this up. In my opinion, it is a bit aggressive to
>> > directly drop Python 3.7 in 1.19. Python 3.7 is still used a lot[1],
>> and as
>> > far as I know, many Pyflink users are still using python 3.7 as their
>> > default interpreter. I prefer to deprecate Python 3.7 in 1.19 just like
>> we
>> > deprecated Python 3.6 in 1.16[2] and dropped Python 3.6 in 1.17[3].
>> >
>> > For the support of Python 3.11, I am very supportive of the
>> implementation
>> > in 1.19 (many users have this appeal, and I originally wanted to
>> support it
>> > in 1.18).
>> >
>> > Regarding the miniconda upgrade, I tend to upgrade miniconda to the
>> latest
>> > version that can support python 3.7 to 3.11 at the same time.
>> >
>> > [1] https://w3techs.com/technologies/history_details/pl-python/3
>> > [2] https://issues.apache.org/jira/browse/FLINK-28195
>> > [3] https://issues.apache.org/jira/browse/FLINK-27929
>> >
>> > Best,
>> > Xingbo
>> >
>> > Jing Ge  于2023年9月5日周二 04:10写道：
>> >
>> > > +1
>> > >
>> > > @Dian should we add support of python 3.11
>> > >
>> > > Best regards,
>> > > Jing
>> > >
>> > > On Mon, Sep 4, 2023 at 3:39 PM Gabor Somogyi <
>> gabor.g.somo...@gmail.com>
>> > > wrote:
>> > >
>> > > > Thanks for all the responses!
>> > > >
>> > > > Based on the suggestions I've created the following jiras and
>> started
>> > to
>> > > > work on them:
>> > > > * https://issues.apache.org/jira/browse/FLINK-33029
>> > > > * https://issues.apache.org/jira/browse/FLINK-33030
>> > > >
>> > > > The reason why I've split them is to separate the concerns and
>> reduce
>> > the
>> > > > amount of code in a PR to help reviewers.
>> > > >
>> > > > BR,
>> > > > G
>> > > >
>> > > >
>> > > > On Mon, Sep 4, 2023 at 12:57 PM Sergey Nuyanzin <
>> snuyan...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > +1,
>> > > > > Thanks for looking into this.
>> > > > >
>> > > > > On Mon, Sep 4, 2023 at 8:38 AM Gyula Fóra 
>> > > wrote:
>> > > > >
>> > > > > > +1
>> > > > > > Thanks for looking into this.
>> > > > > >
>> > > > > > Gyula
>> > > > > >
>> > > > > > On Mon, Sep 4, 2023 at 8:26 AM Matthias Pohl <
>> > matthias.p...@aiven.io
>> > > > > > .invalid>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Thanks Gabor for looking into it. It sounds reasonable to me
>> as
>> > > well.
>> > > > > > >
>> > > > > > > +1
>> > > > > > >
>> > > > > > > On Sun, Sep 3, 2023 at 5:44 PM Márton Balassi <
>> > > > > balassi.mar...@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Gabor,
>> > > > > > > >
>> > > > > > > > Thanks for bringing this up. Similarly to when we dropped
>> > Python
>> > > > 3.6
>> > > > > > due
>> > > > > > > to
>> > > > > > > > its end of life (and added 3.10) in Flink 1.17 [1,2], it
>> makes
>> > > > sense
>> > > > > to
>> > > > > > > > proceed to remove 3.7 and add 3.11 instead.
>> > > > > > > 

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Gyula Fóra]

Thanks Xintong!

Just so I understand correctly, do you suggest adding a metric for
delta(Time) / delta(Count) since the last reporting ?
.TimePerGc or .AverageTime would make sense.
AverageTime may be a bit nicer :)

My only concern is how useful this will be in reality. If there are only
(or several) long pauses then the msPerSec metrics will show it already,
and if there is a single long pause that may not be shown at all if there
are several shorter pauses as well with this metric.

Gyula

On Wed, Sep 6, 2023 at 8:46 AM Xintong Song  wrote:

> Thanks for bringing this up, Gyula.
>
> The proposed changes make sense to me. +1 for them.
>
> In addition to the proposed changes, I wonder if we should also add
> something like timePerGc? This would help understand whether there are long
> pauses, due to GC STW, that may lead to rpc unresponsiveness and heartbeat
> timeouts. Ideally, we'd like to understand the max pause time per STW in a
> recent time window. However, I don't see an easy way to separate the pause
> time of each STW. Deriving the overall time per GC from the existing
> metrics (time-increment / count-increment) seems to be a good alternative.
> WDYT?
>
> Best,
>
> Xintong
>
>
>
> On Wed, Sep 6, 2023 at 2:16 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Thanks for the clarification!
> >
> > By default the meterview measures for 1 minute sounds good to me!
> >
> > +1 for this proposal.
> >
> > Best,
> > Rui
> >
> > On Wed, Sep 6, 2023 at 1:27 PM Gyula Fóra  wrote:
> >
> > > Thanks for the feedback Rui,
> > >
> > > The rates would be computed using the MeterView class (like for any
> other
> > > rate metric), just because we report the value per second it doesn't
> mean
> > > that we measure in a second granularity.
> > > By default the meterview measures for 1 minute and then we calculate
> the
> > > per second rates, but we can increase the timespan if necessary.
> > >
> > > So I don't think we run into this problem in practice and we can keep
> the
> > > metric aligned with other time rate metrics like busyTimeMsPerSec etc.
> > >
> > > Cheers,
> > > Gyula
> > >
> > > On Wed, Sep 6, 2023 at 4:55 AM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > > Hi Gyula,
> > > >
> > > > +1 for this proposal. The current GC metric is really unfriendly.
> > > >
> > > > I have a concern with your proposed rate metric: the rate is
> perSecond
> > > > instead of per minute. I'm unsure whether it's suitable for GC
> metric.
> > > >
> > > > There are two reasons why I suspect perSecond may not be well
> > > > compatible with GC metric:
> > > >
> > > > 1. GCs are usually infrequent and may only occur for a small number
> > > > of time periods within a minute.
> > > >
> > > > Metrics are collected periodically, for example, reported every
> minute.
> > > > If the result reported by the GC metric is 1s/perSecond, it does not
> > > > mean that the GC of the TM is serious, because there may be no GC
> > > > in the remaining 59s.
> > > >
> > > > On the contrary, the GC metric reports 0s/perSecond, which does not
> > > > mean that the GC of the TM is not serious, and the GC may be very
> > > > serious in the remaining 59s.
> > > >
> > > > 2. Stop-the-world may cause the metric to fail(delay) to report
> > > >
> > > > The TM will stop the world during GC, especially full GC. It means
> > > > the metric cannot be collected or reported during full GC.
> > > >
> > > > So the collected GC metric may never be 1s/perSecond. This metric
> > > > may always be good because the metric will only be reported when
> > > > the GC is not severe.
> > > >
> > > >
> > > > If these concerns make sense, how about updating the GC rate
> > > > at minute level?
> > > >
> > > > We can define the type to Gauge for TimeMsPerMiunte, and updating
> > > > this Gauge every second, it is:
> > > > GC Total.Time of current time - GC total time of one miunte ago.
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Tue, Sep 5, 2023 at 11:05 PM Maximilian Michels 
> > > wrote:
> > > >
> > > > > Hi Gyula,
> > > > >
> > > > > +1 The proposed changes make sense and are in line with what is
> > > > > available for other metrics, e.g. number of records processed.
> > > > >
> > > > > -Max
> > > > >
> > > > > On Tue, Sep 5, 2023 at 2:43 PM Gyula Fóra 
> > > wrote:
> > > > > >
> > > > > > Hi Devs,
> > > > > >
> > > > > > I would like to start a discussion on FLIP-361: Improve GC
> Metrics
> > > [1].
> > > > > >
> > > > > > The current Flink GC metrics [2] are not very useful for
> monitoring
> > > > > > purposes as they require post processing logic that is also
> > dependent
> > > > on
> > > > > > the current runtime environment.
> > > > > >
> > > > > > Problems:
> > > > > >  - Total time is not very relevant for long running applications,
> > > only
> > > > > the
> > > > > > rate of change (msPerSec)
> > > > > >  - In most cases it's best to simply aggregate the time/count
> > across
> > > > the
> > > > > > different GabrageCollectors, however the specific co

Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

[Stefan Richter]

+1

Thanks,
Stefan


> On 5. Sep 2023, at 10:45, David Morávek  wrote:
> 
> +1 since there is an alternative, more complete implementation available
> 
> Best,
> D.
> 
> On Sat, Sep 2, 2023 at 12:07 AM David Anderson  > wrote:
> 
>> +1
>> 
>> Keeping the legacy implementation in place is confusing and encourages
>> adoption of something that really shouldn't be used.
>> 
>> Thanks for driving this,
>> David
>> 
>> On Fri, Sep 1, 2023 at 8:45 AM Jing Ge  wrote:
>>> 
>>> Hi Wencong,
>>> 
>>> Thanks for your clarification! +1
>>> 
>>> Best regards,
>>> Jing
>>> 
>>> On Fri, Sep 1, 2023 at 12:36 PM Wencong Liu 
>> wrote:
>>> 
 Hi Jing,
 
 
 Thanks for your reply！
 
 
> Or the "independent module extraction" mentioned in the FLIP does
>> mean an
 independent module in Flink?
 
 
 Yes. If there are submodules in Flink repository needs the iteration
 (currently not),
 we could consider extracting them to a new submodule of Flink.
 
 
> users will have to add one more dependency of Flink ML. If iteration
>> is
 the
 only feature they need, it will look a little bit weird.
 
 
 If users only need to execute iteration jobs, they can simply remove
>> the
 Flink
 dependency and add the necessary dependencies related to Flink ML.
 However,
 they can still utilize the DataStream API as it is also a dependency of
 Flink ML.
 
 
 Keeping an iteration submodule in Flink repository and make Flink ML
 depends it
 is also another solution. But the current implementation of Iteration
>> in
 DataStream
 should be removed definitely due to its Incompleteness.
 
 
 The placement of the Iteration API in the repository is a topic that
>> has
 multiple
 potential solutions. WDYT?
 
 
 Best,
 Wencong
 
 
 
 
 
 
 
 
 
 
 
 At 2023-09-01 17:59:34, "Jing Ge"  wrote:
> Hi Wencong,
> 
> Thanks for the proposal!
> 
> "The Iteration API in DataStream is planned be deprecated in Flink
>> 1.19
 and
> then finally removed in Flink 2.0. For the users that rely on the
 Iteration
> API in DataStream, they will have to migrate to Flink ML."
> - Does it make sense to migrate the iteration module into Flink
>> directly?
> Or the "independent module extraction" mentioned in the FLIP does
>> mean an
> independent module in Flink? Since the iteration will be removed in
>> Flink,
> users will have to add one more dependency of Flink ML. If iteration
>> is
 the
> only feature they need, it will look a little bit weird.
> 
> 
> Best regards,
> Jing
> 
> On Fri, Sep 1, 2023 at 11:05 AM weijie guo >> 
> wrote:
> 
>> Thanks, +1 for this.
>> 
>> Best regards,
>> 
>> Weijie
>> 
>> 
>> Yangze Guo  于2023年9月1日周五 14:29写道：
>> 
>>> +1
>>> 
>>> Thanks for driving this.
>>> 
>>> Best,
>>> Yangze Guo
>>> 
>>> On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <
>> tonysong...@gmail.com>
>> wrote:
 
 +1
 
 Best,
 
 Xintong
 
 
 
 On Fri, Sep 1, 2023 at 1:11 PM Dong Lin 
 wrote:
 
> Thanks Wencong for initiating the discussion.
> 
> +1 for the proposal.
> 
> On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <
>> liuwencle...@163.com
> 
>>> wrote:
> 
>> Hi devs,
>> 
>> I would like to start a discussion on FLIP-357: Deprecate
 Iteration
>>> API
> of
>> DataStream [1].
>> 
>> Currently, the Iteration API of DataStream is incomplete.
>> For
>>> instance,
> it
>> lacks support
>> for iteration in sync mode and exactly once semantics.
>> Additionally,
>>> it
>> does not offer the
>> ability to set iteration termination conditions. As a
>> result,
 it's
>>> hard
>> for developers to
>> build an iteration pipeline by DataStream in the practical
>>> applications
>> such as machine learning.
>> 
>> FLIP-176: Unified Iteration to Support Algorithms [2] has
>> introduced
>>> a
>> unified iteration library
>> in the Flink ML repository. This library addresses all the
 issues
>>> present
>> in the Iteration API of
>> DataStream and could provide solution for all the iteration
>>> use-cases.
>> However, maintaining two
>> separate implementations of iteration in both the Flink
 repository
>>> and
> the
>> Flink ML repository
>> would introduce unnecessary complexity and make it
>> difficult to
>>> maintain
>> the Iteration API.
>> 
>> As such I propose deprec

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Matt Wang]

Hi Gyula,

+1 for this proposal.

Do we need to add a metric to record the count of different 
collectors? Now there is only a total count. For example, 
for G1, there is no way to distinguish whether it is the 
young generation or the old generation.



--

Best,
Matt Wang


 Replied Message 
| From | Gyula Fóra |
| Date | 09/6/2023 15:03 |
| To |  |
| Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
Thanks Xintong!

Just so I understand correctly, do you suggest adding a metric for
delta(Time) / delta(Count) since the last reporting ?
.TimePerGc or .AverageTime would make sense.
AverageTime may be a bit nicer :)

My only concern is how useful this will be in reality. If there are only
(or several) long pauses then the msPerSec metrics will show it already,
and if there is a single long pause that may not be shown at all if there
are several shorter pauses as well with this metric.

Gyula

On Wed, Sep 6, 2023 at 8:46 AM Xintong Song  wrote:

Thanks for bringing this up, Gyula.

The proposed changes make sense to me. +1 for them.

In addition to the proposed changes, I wonder if we should also add
something like timePerGc? This would help understand whether there are long
pauses, due to GC STW, that may lead to rpc unresponsiveness and heartbeat
timeouts. Ideally, we'd like to understand the max pause time per STW in a
recent time window. However, I don't see an easy way to separate the pause
time of each STW. Deriving the overall time per GC from the existing
metrics (time-increment / count-increment) seems to be a good alternative.
WDYT?

Best,

Xintong



On Wed, Sep 6, 2023 at 2:16 PM Rui Fan <1996fan...@gmail.com> wrote:

Thanks for the clarification!

By default the meterview measures for 1 minute sounds good to me!

+1 for this proposal.

Best,
Rui

On Wed, Sep 6, 2023 at 1:27 PM Gyula Fóra  wrote:

Thanks for the feedback Rui,

The rates would be computed using the MeterView class (like for any
other
rate metric), just because we report the value per second it doesn't
mean
that we measure in a second granularity.
By default the meterview measures for 1 minute and then we calculate
the
per second rates, but we can increase the timespan if necessary.

So I don't think we run into this problem in practice and we can keep
the
metric aligned with other time rate metrics like busyTimeMsPerSec etc.

Cheers,
Gyula

On Wed, Sep 6, 2023 at 4:55 AM Rui Fan <1996fan...@gmail.com> wrote:

Hi Gyula,

+1 for this proposal. The current GC metric is really unfriendly.

I have a concern with your proposed rate metric: the rate is
perSecond
instead of per minute. I'm unsure whether it's suitable for GC
metric.

There are two reasons why I suspect perSecond may not be well
compatible with GC metric:

1. GCs are usually infrequent and may only occur for a small number
of time periods within a minute.

Metrics are collected periodically, for example, reported every
minute.
If the result reported by the GC metric is 1s/perSecond, it does not
mean that the GC of the TM is serious, because there may be no GC
in the remaining 59s.

On the contrary, the GC metric reports 0s/perSecond, which does not
mean that the GC of the TM is not serious, and the GC may be very
serious in the remaining 59s.

2. Stop-the-world may cause the metric to fail(delay) to report

The TM will stop the world during GC, especially full GC. It means
the metric cannot be collected or reported during full GC.

So the collected GC metric may never be 1s/perSecond. This metric
may always be good because the metric will only be reported when
the GC is not severe.


If these concerns make sense, how about updating the GC rate
at minute level?

We can define the type to Gauge for TimeMsPerMiunte, and updating
this Gauge every second, it is:
GC Total.Time of current time - GC total time of one miunte ago.

Best,
Rui

On Tue, Sep 5, 2023 at 11:05 PM Maximilian Michels 
wrote:

Hi Gyula,

+1 The proposed changes make sense and are in line with what is
available for other metrics, e.g. number of records processed.

-Max

On Tue, Sep 5, 2023 at 2:43 PM Gyula Fóra 
wrote:

Hi Devs,

I would like to start a discussion on FLIP-361: Improve GC
Metrics
[1].

The current Flink GC metrics [2] are not very useful for
monitoring
purposes as they require post processing logic that is also
dependent
on
the current runtime environment.

Problems:
- Total time is not very relevant for long running applications,
only
the
rate of change (msPerSec)
- In most cases it's best to simply aggregate the time/count
across
the
different GabrageCollectors, however the specific collectors are
dependent
on the current Java runtime

We propose to improve the current situation by:
- Exposing rate metrics per GarbageCollector
- Exposing aggregated Total time/count/rate metrics

These new metrics are all derived from the existing ones with
minimal
overhead.

Looking forward to your feedback.

Cheers,
Gyula

[1]





https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+I

[jira] [Created] (FLINK-33042) Allow trigger flamegraph when task is initializing

[Rui Fan (Jira)]

Rui Fan created FLINK-33042:
---

 Summary: Allow trigger flamegraph when task is initializing 
 Key: FLINK-33042
 URL: https://issues.apache.org/jira/browse/FLINK-33042
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / REST, Runtime / Web Frontend
Reporter: Rui Fan
Assignee: Rui Fan
 Attachments: image-2023-09-06-15-43-37-075.png

Currently, the flamegraph can be triggered when task is running.

After FLINK-17012 and FLINK-22215, flink split the running to running and 
initializing. We should allow trigger flamegraph when task is initializing. For 
example, the initialization is very slow, we need to troubleshoot.

 

Here is a stack example, task is rebuilding the rocksdb after the parallelism 
is changed.

 

!image-2023-09-06-15-43-37-075.png!

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33043) Search is broken on the Flink website

[Danny Cranmer (Jira)]

Danny Cranmer created FLINK-33043:
-

 Summary: Search is broken on the Flink website
 Key: FLINK-33043
 URL: https://issues.apache.org/jira/browse/FLINK-33043
 Project: Flink
  Issue Type: Bug
  Components: Project Website
Reporter: Danny Cranmer


The Search box on [https://flink.apache.org/] does not work. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[liu ron]

Hi, Gyula

Thanks for driving this proposal, GC-related metrics are beneficial for us
to profile the root cause, +1 for this proposal.

Best,
Ron

Matt Wang  于2023年9月6日周三 15:24写道：

> Hi Gyula,
>
> +1 for this proposal.
>
> Do we need to add a metric to record the count of different
> collectors? Now there is only a total count. For example,
> for G1, there is no way to distinguish whether it is the
> young generation or the old generation.
>
>
>
> --
>
> Best,
> Matt Wang
>
>
>  Replied Message 
> | From | Gyula Fóra |
> | Date | 09/6/2023 15:03 |
> | To |  |
> | Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
> Thanks Xintong!
>
> Just so I understand correctly, do you suggest adding a metric for
> delta(Time) / delta(Count) since the last reporting ?
> .TimePerGc or .AverageTime would make sense.
> AverageTime may be a bit nicer :)
>
> My only concern is how useful this will be in reality. If there are only
> (or several) long pauses then the msPerSec metrics will show it already,
> and if there is a single long pause that may not be shown at all if there
> are several shorter pauses as well with this metric.
>
> Gyula
>
> On Wed, Sep 6, 2023 at 8:46 AM Xintong Song  wrote:
>
> Thanks for bringing this up, Gyula.
>
> The proposed changes make sense to me. +1 for them.
>
> In addition to the proposed changes, I wonder if we should also add
> something like timePerGc? This would help understand whether there are long
> pauses, due to GC STW, that may lead to rpc unresponsiveness and heartbeat
> timeouts. Ideally, we'd like to understand the max pause time per STW in a
> recent time window. However, I don't see an easy way to separate the pause
> time of each STW. Deriving the overall time per GC from the existing
> metrics (time-increment / count-increment) seems to be a good alternative.
> WDYT?
>
> Best,
>
> Xintong
>
>
>
> On Wed, Sep 6, 2023 at 2:16 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> Thanks for the clarification!
>
> By default the meterview measures for 1 minute sounds good to me!
>
> +1 for this proposal.
>
> Best,
> Rui
>
> On Wed, Sep 6, 2023 at 1:27 PM Gyula Fóra  wrote:
>
> Thanks for the feedback Rui,
>
> The rates would be computed using the MeterView class (like for any
> other
> rate metric), just because we report the value per second it doesn't
> mean
> that we measure in a second granularity.
> By default the meterview measures for 1 minute and then we calculate
> the
> per second rates, but we can increase the timespan if necessary.
>
> So I don't think we run into this problem in practice and we can keep
> the
> metric aligned with other time rate metrics like busyTimeMsPerSec etc.
>
> Cheers,
> Gyula
>
> On Wed, Sep 6, 2023 at 4:55 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> Hi Gyula,
>
> +1 for this proposal. The current GC metric is really unfriendly.
>
> I have a concern with your proposed rate metric: the rate is
> perSecond
> instead of per minute. I'm unsure whether it's suitable for GC
> metric.
>
> There are two reasons why I suspect perSecond may not be well
> compatible with GC metric:
>
> 1. GCs are usually infrequent and may only occur for a small number
> of time periods within a minute.
>
> Metrics are collected periodically, for example, reported every
> minute.
> If the result reported by the GC metric is 1s/perSecond, it does not
> mean that the GC of the TM is serious, because there may be no GC
> in the remaining 59s.
>
> On the contrary, the GC metric reports 0s/perSecond, which does not
> mean that the GC of the TM is not serious, and the GC may be very
> serious in the remaining 59s.
>
> 2. Stop-the-world may cause the metric to fail(delay) to report
>
> The TM will stop the world during GC, especially full GC. It means
> the metric cannot be collected or reported during full GC.
>
> So the collected GC metric may never be 1s/perSecond. This metric
> may always be good because the metric will only be reported when
> the GC is not severe.
>
>
> If these concerns make sense, how about updating the GC rate
> at minute level?
>
> We can define the type to Gauge for TimeMsPerMiunte, and updating
> this Gauge every second, it is:
> GC Total.Time of current time - GC total time of one miunte ago.
>
> Best,
> Rui
>
> On Tue, Sep 5, 2023 at 11:05 PM Maximilian Michels 
> wrote:
>
> Hi Gyula,
>
> +1 The proposed changes make sense and are in line with what is
> available for other metrics, e.g. number of records processed.
>
> -Max
>
> On Tue, Sep 5, 2023 at 2:43 PM Gyula Fóra 
> wrote:
>
> Hi Devs,
>
> I would like to start a discussion on FLIP-361: Improve GC
> Metrics
> [1].
>
> The current Flink GC metrics [2] are not very useful for
> monitoring
> purposes as they require post processing logic that is also
> dependent
> on
> the current runtime environment.
>
> Problems:
> - Total time is not very relevant for long running applications,
> only
> the
> rate of change (msPerSec)
> - In most cases it's best to simply aggregate the time/count
> acr

Re: [DISCUSS] Add config to enable job stop with savepoint on exceeding tolerable checkpoint Failures

[Hangxiang Yu]

Hi, Dongwoo.
IIUC, you mean using savepoint to store a snapshot to other storage if
checkpoints fail multiple times due to some long lasting exceptions of
external storage, right ?
I think it's better to achieve this by an external tool instead of
introducing a config like that:
1. it's not so easy to judge whether an exception occurs due to external
storage or not sometimes, and it's not so reasonable that we just trigger a
savepoint if checkpoints fail multiple times.
2. It's better to let some logic about triggering savepoint, e.g. periodic
savepoint, triggering stop-with-savepoint, done by external tools or
platform. As you could see from [1], we intend to make their scopes clear.

Maybe you could check the status and failure message by [2] periodically in
your external tool or platform and then trigger savepoint or
stop-with-savepoint by REST API or CLI.

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/checkpoints_vs_savepoints/
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints

On Wed, Sep 6, 2023 at 11:05 AM Yanfei Lei  wrote:

> Hi Dongwoo,
>
> If the checkpoint has failed
> `execution.checkpointing.tolerable-failed-checkpoints` times, then
> stopWithSavepoint is likely to fail as well.
> If stopWithSavepoint succeeds or fails, will the job just stop?  I am
> more curious about how this option works with the restart strategy?
>
> Best,
> Yanfei
>
>
> Dongwoo Kim  于2023年9月4日周一 22:17写道：
> >
> > Hi all,
> > I have a proposal that aims to enhance the flink application's
> resilience in cases of unexpected failures in checkpoint storages like S3
> or HDFS,
> >
> > [Background]
> > When using self managed S3-compatible object storage, we faced
> checkpoint async failures lasting for an extended period more than 30
> minutes,
> > leading to multiple job restarts and causing lags in our streaming
> application.
> >
> > [Current Behavior]
> > Currently, when the number of checkpoint failures exceeds a predefined
> tolerable limit, flink will either restart or fail the job based on how
> it's configured.
> > In my opinion this does not handle scenarios where the checkpoint
> storage itself may be unreliable or experiencing downtime.
> >
> > [Proposed Feature]
> > I propose a config that allows for a graceful job stop with a savepoint
> when the tolerable checkpoint failure limit is reached.
> > Instead of restarting/failing the job when tolerable checkpoint failure
> exceeds, when this new config is set to true just trigger stopWithSavepoint.
> >
> > This could offer the following benefits.
> > - Indication of Checkpoint Storage State: Exceeding tolerable checkpoint
> failures could indicate unstable checkpoint storage.
> > - Automated Fallback Strategy: When combined with a monitoring cron job,
> this feature could act as an automated fallback strategy for handling
> unstable checkpoint storage.
> >   The job would stop safely, take a savepoint, and then you could
> automatically restart with different checkpoint storage configured like
> switching from S3 to HDFS.
> >
> > For example let's say checkpoint path is configured to s3 and savepoint
> path is configured to hdfs.
> > When the new config is set to true the job stops with savepoint like
> below when tolerable checkpoint failure exceeds.
> > And we can restart the job from that savepoint while the checkpoint
> configured as hdfs.
> >
> >
> >
> > Looking forward to hearing the community's thoughts on this proposal.
> > And also want to ask how the community is handling long lasting unstable
> checkpoint storage issues.
> >
> > Thanks in advance.
> >
> > Best dongwoo,
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Gyula Fóra]

Matt Wang,

I think the currently exposed info is all that is available through
GarbageCollectorMXBean. This FLIP does not aim to introduce a new more
granular way of reporting the per collector metrics, that would require a
new mechanism and may be a breaking change.

We basically want to simply extend the current reporting here with the rate
metrics and the total metrics.

Gyula

On Wed, Sep 6, 2023 at 9:24 AM Matt Wang  wrote:

> Hi Gyula,
>
> +1 for this proposal.
>
> Do we need to add a metric to record the count of different
> collectors? Now there is only a total count. For example,
> for G1, there is no way to distinguish whether it is the
> young generation or the old generation.
>
>
>
> --
>
> Best,
> Matt Wang
>
>
>  Replied Message 
> | From | Gyula Fóra |
> | Date | 09/6/2023 15:03 |
> | To |  |
> | Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
> Thanks Xintong!
>
> Just so I understand correctly, do you suggest adding a metric for
> delta(Time) / delta(Count) since the last reporting ?
> .TimePerGc or .AverageTime would make sense.
> AverageTime may be a bit nicer :)
>
> My only concern is how useful this will be in reality. If there are only
> (or several) long pauses then the msPerSec metrics will show it already,
> and if there is a single long pause that may not be shown at all if there
> are several shorter pauses as well with this metric.
>
> Gyula
>
> On Wed, Sep 6, 2023 at 8:46 AM Xintong Song  wrote:
>
> Thanks for bringing this up, Gyula.
>
> The proposed changes make sense to me. +1 for them.
>
> In addition to the proposed changes, I wonder if we should also add
> something like timePerGc? This would help understand whether there are long
> pauses, due to GC STW, that may lead to rpc unresponsiveness and heartbeat
> timeouts. Ideally, we'd like to understand the max pause time per STW in a
> recent time window. However, I don't see an easy way to separate the pause
> time of each STW. Deriving the overall time per GC from the existing
> metrics (time-increment / count-increment) seems to be a good alternative.
> WDYT?
>
> Best,
>
> Xintong
>
>
>
> On Wed, Sep 6, 2023 at 2:16 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> Thanks for the clarification!
>
> By default the meterview measures for 1 minute sounds good to me!
>
> +1 for this proposal.
>
> Best,
> Rui
>
> On Wed, Sep 6, 2023 at 1:27 PM Gyula Fóra  wrote:
>
> Thanks for the feedback Rui,
>
> The rates would be computed using the MeterView class (like for any
> other
> rate metric), just because we report the value per second it doesn't
> mean
> that we measure in a second granularity.
> By default the meterview measures for 1 minute and then we calculate
> the
> per second rates, but we can increase the timespan if necessary.
>
> So I don't think we run into this problem in practice and we can keep
> the
> metric aligned with other time rate metrics like busyTimeMsPerSec etc.
>
> Cheers,
> Gyula
>
> On Wed, Sep 6, 2023 at 4:55 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> Hi Gyula,
>
> +1 for this proposal. The current GC metric is really unfriendly.
>
> I have a concern with your proposed rate metric: the rate is
> perSecond
> instead of per minute. I'm unsure whether it's suitable for GC
> metric.
>
> There are two reasons why I suspect perSecond may not be well
> compatible with GC metric:
>
> 1. GCs are usually infrequent and may only occur for a small number
> of time periods within a minute.
>
> Metrics are collected periodically, for example, reported every
> minute.
> If the result reported by the GC metric is 1s/perSecond, it does not
> mean that the GC of the TM is serious, because there may be no GC
> in the remaining 59s.
>
> On the contrary, the GC metric reports 0s/perSecond, which does not
> mean that the GC of the TM is not serious, and the GC may be very
> serious in the remaining 59s.
>
> 2. Stop-the-world may cause the metric to fail(delay) to report
>
> The TM will stop the world during GC, especially full GC. It means
> the metric cannot be collected or reported during full GC.
>
> So the collected GC metric may never be 1s/perSecond. This metric
> may always be good because the metric will only be reported when
> the GC is not severe.
>
>
> If these concerns make sense, how about updating the GC rate
> at minute level?
>
> We can define the type to Gauge for TimeMsPerMiunte, and updating
> this Gauge every second, it is:
> GC Total.Time of current time - GC total time of one miunte ago.
>
> Best,
> Rui
>
> On Tue, Sep 5, 2023 at 11:05 PM Maximilian Michels 
> wrote:
>
> Hi Gyula,
>
> +1 The proposed changes make sense and are in line with what is
> available for other metrics, e.g. number of records processed.
>
> -Max
>
> On Tue, Sep 5, 2023 at 2:43 PM Gyula Fóra 
> wrote:
>
> Hi Devs,
>
> I would like to start a discussion on FLIP-361: Improve GC
> Metrics
> [1].
>
> The current Flink GC metrics [2] are not very useful for
> monitoring
> purposes as they require post processing logic t

[jira] [Created] (FLINK-33044) Reduce the frequency of triggering flush for the disk tier of the tiered storage

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33044:
-

 Summary: Reduce the frequency of triggering flush for the disk 
tier of the tiered storage
 Key: FLINK-33044
 URL: https://issues.apache.org/jira/browse/FLINK-33044
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


The disk cache of tiered storage will flush at the end of each subpartition's 
segment, which is too frequent and is bad for performance. We should improve it 
with some better flushing methods, e.g. flushing buffers with batch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33045) format avro-confluent - disable auto-registering schema in registry

[Michael Lorenzi (Jira)]

Michael Lorenzi created FLINK-33045:
---

 Summary: format avro-confluent - disable auto-registering schema 
in registry
 Key: FLINK-33045
 URL: https://issues.apache.org/jira/browse/FLINK-33045
 Project: Flink
  Issue Type: Improvement
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.17.1
 Environment: Flink-Kubernetes-Operator

format avro-confluent
Reporter: Michael Lorenzi


{code:java}
'value.format' = 'avro-confluent',
'value.avro-confluent.url' = '',
'value.avro-confluent.schema' = '', -- available in 1.18.x{code}

Currently the format {{avro-confluent}} seems to always try to register the 
schema to the schema-registry, be it computed out of the table-schema or set by 
property {{schema}} (if 1.18.x is used).

In our setup we would like to prevent our flink-jobs from registering schemas. 
Flink should only lookup the schema-id by calling {{getId}} from confluents 
schema-registry-client. We control publishing of new schema-versions outside of 
flink.

The bevaviour is expected to be similar to the behvaiour of KafkaAvroSerializer 
from confluent when {{auto.register.schemas=false}} is set.

Could there be a option like
{code:java}
'value.avro-confluent.auto-register.schema' = 'false'{code}
to disable auto-registering schema to the schema-registry?

Thank you for your help!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Drop python 3.7 support in 1.19

[Gabor Somogyi]

Hi Xingbo,

> So I guess you want to run ci tests  on the m1
> environment, but the current version of miniconda cannot meet this
> requirement, so there is a pre-step that must drop python 3.7?

In short yes + local wheel build fix on the M1. In a bit more detailed
please let me elaborate.

You're right about the arm64 build creation, Flink is doing that for quite
some time already and wheels are running fine.
There are gaps however.

>From user perspective:
* we're building wheels for ARM64 but we're not executing any python tests
on ARM64 but only on X64.
I think it's a must to test platform packages on the related platform since
not all python packages are ARM compatible or must be built/used in a
different way.

>From dev perspective:
* Local MacOs M1 wheel or python source distro builds are simply failing
with error messages.
Just to give an example GRPC pip install is failing and require the
following changes:
https://github.com/apache/flink/pull/23359/files#diff-c4c4916cd6ab7f87a4b154ef9603f9ba6d3b74a00096be9adacb7df363b5efcfR28-R32
I can't really imagine how Azure is able to build MacOS ARM64 wheels...
* Python tests are not running on any kind of ARM CPUs which makes any bug
hunting/development hard or nearly impossible

In order to give ARM support with local development + test execution
possibility new miniconda is needed. New miniconda runs 3.8+.

There is already a PR which is under development:
https://github.com/apache/flink/pull/23359
This fills all the mentioned gaps: local test/development/test execution on
ARM CPUs + the previously mentioned 3+ years not actively supported python
eliminated.

Hope this helps.

G


On Wed, Sep 6, 2023 at 9:00 AM Xingbo Huang  wrote:

> Hi Gyala and Gabor,
>
> Pyflink has provided arm64 wheel packages for Apple silicon since 1.16[1].
> The use of Miniconda is only related to ci testing and packaging on linux
> platform, and building mac platform wheels are dependent on
> cibuildwheel[2]. So I guess you want to run ci tests  on the m1
> environment, but the current version of miniconda cannot meet this
> requirement, so there is a pre-step that must drop python 3.7?
>
> [1] https://pypi.org/project/apache-flink/1.16.2/#files
> [2]
>
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml#L30
>
> Best,
> Xingbo
>
> Gabor Somogyi  于2023年9月6日周三 14:36写道：
>
> > Hi Xingbo,
> >
> > *Constraint:*
> > I personally not found any miniconda version which provides arm64 support
> > together with python 3.7.
> > [image: image.png]
> >
> > At the moment I think new platform support means 3.7 drop.
> >
> > I fully to agree with Gyula, if we start now maybe we can release it in
> > half a year however *3.7 active support already ended in 27 Jun 2020*.
> > At the moment any python development/test execution on MacOS M1 is just
> > not working as-is just like any kind of python test execution on any ARM
> > CPU.
> >
> > Gains:
> > * We can release a working version in half a year hopefully and not
> > shifting support to 1+ year
> > * MacOS M1 local development would work finally which is essential for
> > user engagement
> > * It would be possible to execute python tests on ARM64 machines
> > * We can shake up the python development story because it's not the most
> > loved area
> >
> > BR,
> > G
> >
> >
> > On Wed, Sep 6, 2023 at 8:06 AM Gyula Fóra  wrote:
> >
> >> Hi Xingbo!
> >>
> >> I think we have to analyze what we gain by dropping 3.7 and upgrading
> to a
> >> miniconda version with a multiarch support.
> >>
> >> If this is what we need to get Apple silicon support then I think it's
> >> worth doing it already in 1.19. Keep in mind that 1.18 is not even
> >> released
> >> yet so if we delay this to 1.20 that is basically 1 year from now.
> >> Making this change can increase the adoption instantly if we enable new
> >> platforms.
> >>
> >> Cheers,
> >> Gyula
> >>
> >> On Wed, Sep 6, 2023 at 4:46 AM Xingbo Huang  wrote:
> >>
> >> > Hi Gabor,
> >> >
> >> > Thanks for bringing this up. In my opinion, it is a bit aggressive to
> >> > directly drop Python 3.7 in 1.19. Python 3.7 is still used a lot[1],
> >> and as
> >> > far as I know, many Pyflink users are still using python 3.7 as their
> >> > default interpreter. I prefer to deprecate Python 3.7 in 1.19 just
> like
> >> we
> >> > deprecated Python 3.6 in 1.16[2] and dropped Python 3.6 in 1.17[3].
> >> >
> >> > For the support of Python 3.11, I am very supportive of the
> >> implementation
> >> > in 1.19 (many users have this appeal, and I originally wanted to
> >> support it
> >> > in 1.18).
> >> >
> >> > Regarding the miniconda upgrade, I tend to upgrade miniconda to the
> >> latest
> >> > version that can support python 3.7 to 3.11 at the same time.
> >> >
> >> > [1] https://w3techs.com/technologies/history_details/pl-python/3
> >> > [2] https://issues.apache.org/jira/browse/FLINK-28195
> >> > [3] https://issues.apache.org/jira/browse/FLINK-27929
> >> >
> >> > Best,
> >> > Xingbo
>

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Xintong Song]

>
> Just so I understand correctly, do you suggest adding a metric for
> delta(Time) / delta(Count) since the last reporting ?
> .TimePerGc or .AverageTime would make sense.
> AverageTime may be a bit nicer :)
>

Yes, that's what I mean.

My only concern is how useful this will be in reality. If there are only
> (or several) long pauses then the msPerSec metrics will show it already,
> and if there is a single long pause that may not be shown at all if there
> are several shorter pauses as well with this metric.


Let's say we measure this for every minute and see a 900 msPerSec (which
means 54s within the minute are spent on GC). This may come from a single
GC that lasts for 54s, or 2 GCs each lasting for ~27s, or more GCs with
less time each. As the default heartbeat timeout is 50s, the former means
there's likely a heartbeat timeout due to the GC pause, while the latter
means otherwise.


Mathematically, it is possible that there's 1 long pause together with
several short pauses within the same measurement period, making the long
pause not observable with AverageTime. However, from my experience, such a
pattern is not normal in reality. My observation is that GCs happen at a
similar time usually take a similar length of time. Admittedly, this is not
a hard guarantee.


Best,

Xintong



On Wed, Sep 6, 2023 at 3:59 PM Gyula Fóra  wrote:

> Matt Wang,
>
> I think the currently exposed info is all that is available through
> GarbageCollectorMXBean. This FLIP does not aim to introduce a new more
> granular way of reporting the per collector metrics, that would require a
> new mechanism and may be a breaking change.
>
> We basically want to simply extend the current reporting here with the rate
> metrics and the total metrics.
>
> Gyula
>
> On Wed, Sep 6, 2023 at 9:24 AM Matt Wang  wrote:
>
> > Hi Gyula,
> >
> > +1 for this proposal.
> >
> > Do we need to add a metric to record the count of different
> > collectors? Now there is only a total count. For example,
> > for G1, there is no way to distinguish whether it is the
> > young generation or the old generation.
> >
> >
> >
> > --
> >
> > Best,
> > Matt Wang
> >
> >
> >  Replied Message 
> > | From | Gyula Fóra |
> > | Date | 09/6/2023 15:03 |
> > | To |  |
> > | Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
> > Thanks Xintong!
> >
> > Just so I understand correctly, do you suggest adding a metric for
> > delta(Time) / delta(Count) since the last reporting ?
> > .TimePerGc or .AverageTime would make sense.
> > AverageTime may be a bit nicer :)
> >
> > My only concern is how useful this will be in reality. If there are only
> > (or several) long pauses then the msPerSec metrics will show it already,
> > and if there is a single long pause that may not be shown at all if there
> > are several shorter pauses as well with this metric.
> >
> > Gyula
> >
> > On Wed, Sep 6, 2023 at 8:46 AM Xintong Song 
> wrote:
> >
> > Thanks for bringing this up, Gyula.
> >
> > The proposed changes make sense to me. +1 for them.
> >
> > In addition to the proposed changes, I wonder if we should also add
> > something like timePerGc? This would help understand whether there are
> long
> > pauses, due to GC STW, that may lead to rpc unresponsiveness and
> heartbeat
> > timeouts. Ideally, we'd like to understand the max pause time per STW in
> a
> > recent time window. However, I don't see an easy way to separate the
> pause
> > time of each STW. Deriving the overall time per GC from the existing
> > metrics (time-increment / count-increment) seems to be a good
> alternative.
> > WDYT?
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Sep 6, 2023 at 2:16 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Thanks for the clarification!
> >
> > By default the meterview measures for 1 minute sounds good to me!
> >
> > +1 for this proposal.
> >
> > Best,
> > Rui
> >
> > On Wed, Sep 6, 2023 at 1:27 PM Gyula Fóra  wrote:
> >
> > Thanks for the feedback Rui,
> >
> > The rates would be computed using the MeterView class (like for any
> > other
> > rate metric), just because we report the value per second it doesn't
> > mean
> > that we measure in a second granularity.
> > By default the meterview measures for 1 minute and then we calculate
> > the
> > per second rates, but we can increase the timespan if necessary.
> >
> > So I don't think we run into this problem in practice and we can keep
> > the
> > metric aligned with other time rate metrics like busyTimeMsPerSec etc.
> >
> > Cheers,
> > Gyula
> >
> > On Wed, Sep 6, 2023 at 4:55 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Hi Gyula,
> >
> > +1 for this proposal. The current GC metric is really unfriendly.
> >
> > I have a concern with your proposed rate metric: the rate is
> > perSecond
> > instead of per minute. I'm unsure whether it's suitable for GC
> > metric.
> >
> > There are two reasons why I suspect perSecond may not be well
> > compatible with GC metric:
> >
> > 1. GCs are usually infrequent and may onl

Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy

[Gyula Fóra]

@Zhanghao Chen:

I am not completely sure at this point what this will mean for 2.0 simply
because I am also not sure what that will mean for the operator as well :)
I think this will depend on the compatibility guarantees we can provide
across Flink major versions in general. We have to look into that and
tackle the question there independently.

Gyula

On Tue, Sep 5, 2023 at 6:12 PM Maximilian Michels  wrote:

> +1 Sounds good! Four releases give a decent amount of time to migrate
> to the next Flink version.
>
> On Tue, Sep 5, 2023 at 5:33 PM Őrhidi Mátyás 
> wrote:
> >
> > +1
> >
> > On Tue, Sep 5, 2023 at 8:03 AM Thomas Weise  wrote:
> >
> > > +1, thanks for the proposal
> > >
> > > On Tue, Sep 5, 2023 at 8:13 AM Gyula Fóra 
> wrote:
> > >
> > > > Hi All!
> > > >
> > > > @Maximilian Michels  has raised the question of
> Flink
> > > > version support in the operator before the last release. I would
> like to
> > > > open this discussion publicly so we can finalize this before the next
> > > > release.
> > > >
> > > > Background:
> > > > Currently the Flink Operator supports all Flink versions since Flink
> > > 1.13.
> > > > While this is great for the users, it introduces a lot of backward
> > > > compatibility related code in the operator logic and also adds
> > > considerable
> > > > time to the CI. We should strike a reasonable balance here that
> allows us
> > > > to move forward and eliminate some of this tech debt.
> > > >
> > > > In the current model it is also impossible to support all features
> for
> > > all
> > > > Flink versions which leads to some confusion over time.
> > > >
> > > > Proposal:
> > > > Since it's a key feature of the kubernetes operator to support
> several
> > > > versions at the same time, I propose to support the last 4 stable
> Flink
> > > > minor versions. Currently this would mean to support Flink 1.14-1.17
> (and
> > > > drop 1.13 support). When Flink 1.18 is released we would drop 1.14
> > > support
> > > > and so on. Given the Flink release cadence this means about 2 year
> > > support
> > > > for each Flink version.
> > > >
> > > > What do you think?
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> > >
>

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Gyula Fóra]

Makes sense Xintong, I am happy to extend the proposal with the average gc
time metric +1

Gyula

On Wed, Sep 6, 2023 at 10:09 AM Xintong Song  wrote:

> >
> > Just so I understand correctly, do you suggest adding a metric for
> > delta(Time) / delta(Count) since the last reporting ?
> > .TimePerGc or .AverageTime would make sense.
> > AverageTime may be a bit nicer :)
> >
>
> Yes, that's what I mean.
>
> My only concern is how useful this will be in reality. If there are only
> > (or several) long pauses then the msPerSec metrics will show it already,
> > and if there is a single long pause that may not be shown at all if there
> > are several shorter pauses as well with this metric.
>
>
> Let's say we measure this for every minute and see a 900 msPerSec (which
> means 54s within the minute are spent on GC). This may come from a single
> GC that lasts for 54s, or 2 GCs each lasting for ~27s, or more GCs with
> less time each. As the default heartbeat timeout is 50s, the former means
> there's likely a heartbeat timeout due to the GC pause, while the latter
> means otherwise.
>
>
> Mathematically, it is possible that there's 1 long pause together with
> several short pauses within the same measurement period, making the long
> pause not observable with AverageTime. However, from my experience, such a
> pattern is not normal in reality. My observation is that GCs happen at a
> similar time usually take a similar length of time. Admittedly, this is not
> a hard guarantee.
>
>
> Best,
>
> Xintong
>
>
>
> On Wed, Sep 6, 2023 at 3:59 PM Gyula Fóra  wrote:
>
> > Matt Wang,
> >
> > I think the currently exposed info is all that is available through
> > GarbageCollectorMXBean. This FLIP does not aim to introduce a new more
> > granular way of reporting the per collector metrics, that would require a
> > new mechanism and may be a breaking change.
> >
> > We basically want to simply extend the current reporting here with the
> rate
> > metrics and the total metrics.
> >
> > Gyula
> >
> > On Wed, Sep 6, 2023 at 9:24 AM Matt Wang  wrote:
> >
> > > Hi Gyula,
> > >
> > > +1 for this proposal.
> > >
> > > Do we need to add a metric to record the count of different
> > > collectors? Now there is only a total count. For example,
> > > for G1, there is no way to distinguish whether it is the
> > > young generation or the old generation.
> > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Matt Wang
> > >
> > >
> > >  Replied Message 
> > > | From | Gyula Fóra |
> > > | Date | 09/6/2023 15:03 |
> > > | To |  |
> > > | Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
> > > Thanks Xintong!
> > >
> > > Just so I understand correctly, do you suggest adding a metric for
> > > delta(Time) / delta(Count) since the last reporting ?
> > > .TimePerGc or .AverageTime would make sense.
> > > AverageTime may be a bit nicer :)
> > >
> > > My only concern is how useful this will be in reality. If there are
> only
> > > (or several) long pauses then the msPerSec metrics will show it
> already,
> > > and if there is a single long pause that may not be shown at all if
> there
> > > are several shorter pauses as well with this metric.
> > >
> > > Gyula
> > >
> > > On Wed, Sep 6, 2023 at 8:46 AM Xintong Song 
> > wrote:
> > >
> > > Thanks for bringing this up, Gyula.
> > >
> > > The proposed changes make sense to me. +1 for them.
> > >
> > > In addition to the proposed changes, I wonder if we should also add
> > > something like timePerGc? This would help understand whether there are
> > long
> > > pauses, due to GC STW, that may lead to rpc unresponsiveness and
> > heartbeat
> > > timeouts. Ideally, we'd like to understand the max pause time per STW
> in
> > a
> > > recent time window. However, I don't see an easy way to separate the
> > pause
> > > time of each STW. Deriving the overall time per GC from the existing
> > > metrics (time-increment / count-increment) seems to be a good
> > alternative.
> > > WDYT?
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Wed, Sep 6, 2023 at 2:16 PM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Thanks for the clarification!
> > >
> > > By default the meterview measures for 1 minute sounds good to me!
> > >
> > > +1 for this proposal.
> > >
> > > Best,
> > > Rui
> > >
> > > On Wed, Sep 6, 2023 at 1:27 PM Gyula Fóra 
> wrote:
> > >
> > > Thanks for the feedback Rui,
> > >
> > > The rates would be computed using the MeterView class (like for any
> > > other
> > > rate metric), just because we report the value per second it doesn't
> > > mean
> > > that we measure in a second granularity.
> > > By default the meterview measures for 1 minute and then we calculate
> > > the
> > > per second rates, but we can increase the timespan if necessary.
> > >
> > > So I don't think we run into this problem in practice and we can keep
> > > the
> > > metric aligned with other time rate metrics like busyTimeMsPerSec etc.
> > >
> > > Cheers,
> > > Gyula
> > >
> > > On Wed, Sep 6, 2023 at 4:5

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Xintong Song]

Thank you :)

Best,

Xintong



On Wed, Sep 6, 2023 at 4:17 PM Gyula Fóra  wrote:

> Makes sense Xintong, I am happy to extend the proposal with the average gc
> time metric +1
>
> Gyula
>
> On Wed, Sep 6, 2023 at 10:09 AM Xintong Song 
> wrote:
>
> > >
> > > Just so I understand correctly, do you suggest adding a metric for
> > > delta(Time) / delta(Count) since the last reporting ?
> > > .TimePerGc or .AverageTime would make sense.
> > > AverageTime may be a bit nicer :)
> > >
> >
> > Yes, that's what I mean.
> >
> > My only concern is how useful this will be in reality. If there are only
> > > (or several) long pauses then the msPerSec metrics will show it
> already,
> > > and if there is a single long pause that may not be shown at all if
> there
> > > are several shorter pauses as well with this metric.
> >
> >
> > Let's say we measure this for every minute and see a 900 msPerSec (which
> > means 54s within the minute are spent on GC). This may come from a single
> > GC that lasts for 54s, or 2 GCs each lasting for ~27s, or more GCs with
> > less time each. As the default heartbeat timeout is 50s, the former means
> > there's likely a heartbeat timeout due to the GC pause, while the latter
> > means otherwise.
> >
> >
> > Mathematically, it is possible that there's 1 long pause together with
> > several short pauses within the same measurement period, making the long
> > pause not observable with AverageTime. However, from my experience, such
> a
> > pattern is not normal in reality. My observation is that GCs happen at a
> > similar time usually take a similar length of time. Admittedly, this is
> not
> > a hard guarantee.
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Sep 6, 2023 at 3:59 PM Gyula Fóra  wrote:
> >
> > > Matt Wang,
> > >
> > > I think the currently exposed info is all that is available through
> > > GarbageCollectorMXBean. This FLIP does not aim to introduce a new more
> > > granular way of reporting the per collector metrics, that would
> require a
> > > new mechanism and may be a breaking change.
> > >
> > > We basically want to simply extend the current reporting here with the
> > rate
> > > metrics and the total metrics.
> > >
> > > Gyula
> > >
> > > On Wed, Sep 6, 2023 at 9:24 AM Matt Wang  wrote:
> > >
> > > > Hi Gyula,
> > > >
> > > > +1 for this proposal.
> > > >
> > > > Do we need to add a metric to record the count of different
> > > > collectors? Now there is only a total count. For example,
> > > > for G1, there is no way to distinguish whether it is the
> > > > young generation or the old generation.
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best,
> > > > Matt Wang
> > > >
> > > >
> > > >  Replied Message 
> > > > | From | Gyula Fóra |
> > > > | Date | 09/6/2023 15:03 |
> > > > | To |  |
> > > > | Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
> > > > Thanks Xintong!
> > > >
> > > > Just so I understand correctly, do you suggest adding a metric for
> > > > delta(Time) / delta(Count) since the last reporting ?
> > > > .TimePerGc or .AverageTime would make sense.
> > > > AverageTime may be a bit nicer :)
> > > >
> > > > My only concern is how useful this will be in reality. If there are
> > only
> > > > (or several) long pauses then the msPerSec metrics will show it
> > already,
> > > > and if there is a single long pause that may not be shown at all if
> > there
> > > > are several shorter pauses as well with this metric.
> > > >
> > > > Gyula
> > > >
> > > > On Wed, Sep 6, 2023 at 8:46 AM Xintong Song 
> > > wrote:
> > > >
> > > > Thanks for bringing this up, Gyula.
> > > >
> > > > The proposed changes make sense to me. +1 for them.
> > > >
> > > > In addition to the proposed changes, I wonder if we should also add
> > > > something like timePerGc? This would help understand whether there
> are
> > > long
> > > > pauses, due to GC STW, that may lead to rpc unresponsiveness and
> > > heartbeat
> > > > timeouts. Ideally, we'd like to understand the max pause time per STW
> > in
> > > a
> > > > recent time window. However, I don't see an easy way to separate the
> > > pause
> > > > time of each STW. Deriving the overall time per GC from the existing
> > > > metrics (time-increment / count-increment) seems to be a good
> > > alternative.
> > > > WDYT?
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Wed, Sep 6, 2023 at 2:16 PM Rui Fan <1996fan...@gmail.com> wrote:
> > > >
> > > > Thanks for the clarification!
> > > >
> > > > By default the meterview measures for 1 minute sounds good to me!
> > > >
> > > > +1 for this proposal.
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Wed, Sep 6, 2023 at 1:27 PM Gyula Fóra 
> > wrote:
> > > >
> > > > Thanks for the feedback Rui,
> > > >
> > > > The rates would be computed using the MeterView class (like for any
> > > > other
> > > > rate metric), just because we report the value per second it doesn't
> > > > mean
> > > > that we measure in a second granularity.
> >

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Gyula Fóra]

I added the average time metric to the FLIP document. I also included it
for the aggregate (total) across all collectors. But maybe it doesn't make
too much sense as collection times usually differ greatly depending on the
collector.

Gyula

On Wed, Sep 6, 2023 at 10:21 AM Xintong Song  wrote:

> Thank you :)
>
> Best,
>
> Xintong
>
>
>
> On Wed, Sep 6, 2023 at 4:17 PM Gyula Fóra  wrote:
>
> > Makes sense Xintong, I am happy to extend the proposal with the average
> gc
> > time metric +1
> >
> > Gyula
> >
> > On Wed, Sep 6, 2023 at 10:09 AM Xintong Song 
> > wrote:
> >
> > > >
> > > > Just so I understand correctly, do you suggest adding a metric for
> > > > delta(Time) / delta(Count) since the last reporting ?
> > > > .TimePerGc or .AverageTime would make sense.
> > > > AverageTime may be a bit nicer :)
> > > >
> > >
> > > Yes, that's what I mean.
> > >
> > > My only concern is how useful this will be in reality. If there are
> only
> > > > (or several) long pauses then the msPerSec metrics will show it
> > already,
> > > > and if there is a single long pause that may not be shown at all if
> > there
> > > > are several shorter pauses as well with this metric.
> > >
> > >
> > > Let's say we measure this for every minute and see a 900 msPerSec
> (which
> > > means 54s within the minute are spent on GC). This may come from a
> single
> > > GC that lasts for 54s, or 2 GCs each lasting for ~27s, or more GCs with
> > > less time each. As the default heartbeat timeout is 50s, the former
> means
> > > there's likely a heartbeat timeout due to the GC pause, while the
> latter
> > > means otherwise.
> > >
> > >
> > > Mathematically, it is possible that there's 1 long pause together with
> > > several short pauses within the same measurement period, making the
> long
> > > pause not observable with AverageTime. However, from my experience,
> such
> > a
> > > pattern is not normal in reality. My observation is that GCs happen at
> a
> > > similar time usually take a similar length of time. Admittedly, this is
> > not
> > > a hard guarantee.
> > >
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Wed, Sep 6, 2023 at 3:59 PM Gyula Fóra 
> wrote:
> > >
> > > > Matt Wang,
> > > >
> > > > I think the currently exposed info is all that is available through
> > > > GarbageCollectorMXBean. This FLIP does not aim to introduce a new
> more
> > > > granular way of reporting the per collector metrics, that would
> > require a
> > > > new mechanism and may be a breaking change.
> > > >
> > > > We basically want to simply extend the current reporting here with
> the
> > > rate
> > > > metrics and the total metrics.
> > > >
> > > > Gyula
> > > >
> > > > On Wed, Sep 6, 2023 at 9:24 AM Matt Wang  wrote:
> > > >
> > > > > Hi Gyula,
> > > > >
> > > > > +1 for this proposal.
> > > > >
> > > > > Do we need to add a metric to record the count of different
> > > > > collectors? Now there is only a total count. For example,
> > > > > for G1, there is no way to distinguish whether it is the
> > > > > young generation or the old generation.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Matt Wang
> > > > >
> > > > >
> > > > >  Replied Message 
> > > > > | From | Gyula Fóra |
> > > > > | Date | 09/6/2023 15:03 |
> > > > > | To |  |
> > > > > | Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
> > > > > Thanks Xintong!
> > > > >
> > > > > Just so I understand correctly, do you suggest adding a metric for
> > > > > delta(Time) / delta(Count) since the last reporting ?
> > > > > .TimePerGc or .AverageTime would make sense.
> > > > > AverageTime may be a bit nicer :)
> > > > >
> > > > > My only concern is how useful this will be in reality. If there are
> > > only
> > > > > (or several) long pauses then the msPerSec metrics will show it
> > > already,
> > > > > and if there is a single long pause that may not be shown at all if
> > > there
> > > > > are several shorter pauses as well with this metric.
> > > > >
> > > > > Gyula
> > > > >
> > > > > On Wed, Sep 6, 2023 at 8:46 AM Xintong Song  >
> > > > wrote:
> > > > >
> > > > > Thanks for bringing this up, Gyula.
> > > > >
> > > > > The proposed changes make sense to me. +1 for them.
> > > > >
> > > > > In addition to the proposed changes, I wonder if we should also add
> > > > > something like timePerGc? This would help understand whether there
> > are
> > > > long
> > > > > pauses, due to GC STW, that may lead to rpc unresponsiveness and
> > > > heartbeat
> > > > > timeouts. Ideally, we'd like to understand the max pause time per
> STW
> > > in
> > > > a
> > > > > recent time window. However, I don't see an easy way to separate
> the
> > > > pause
> > > > > time of each STW. Deriving the overall time per GC from the
> existing
> > > > > metrics (time-increment / count-increment) seems to be a good
> > > > alternative.
> > > > > WDYT?
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Se

Re: [DISCUSS] Drop python 3.7 support in 1.19

[Xingbo Huang]

Hi Gabor,

Thanks a lot for the detailed explanation. Building a test environment with
multi-platforms is indeed what we have been lacking,
not only for M1 users, but also for Windows users. At 1.16, I remember that
Azure did not provide the m1 environment,
and the github action just planned m1 in the annual plan at that
time[1][2].

+1 for drop 3.7 support in 1.19

[1] https://github.com/actions/runner-images/issues/2187
[2] https://github.com/github/roadmap/issues/528

Best,
Xingbo

Gabor Somogyi  于2023年9月6日周三 16:07写道：

> Hi Xingbo,
>
> > So I guess you want to run ci tests  on the m1
> > environment, but the current version of miniconda cannot meet this
> > requirement, so there is a pre-step that must drop python 3.7?
>
> In short yes + local wheel build fix on the M1. In a bit more detailed
> please let me elaborate.
>
> You're right about the arm64 build creation, Flink is doing that for quite
> some time already and wheels are running fine.
> There are gaps however.
>
> From user perspective:
> * we're building wheels for ARM64 but we're not executing any python tests
> on ARM64 but only on X64.
> I think it's a must to test platform packages on the related platform since
> not all python packages are ARM compatible or must be built/used in a
> different way.
>
> From dev perspective:
> * Local MacOs M1 wheel or python source distro builds are simply failing
> with error messages.
> Just to give an example GRPC pip install is failing and require the
> following changes:
>
> https://github.com/apache/flink/pull/23359/files#diff-c4c4916cd6ab7f87a4b154ef9603f9ba6d3b74a00096be9adacb7df363b5efcfR28-R32
> I can't really imagine how Azure is able to build MacOS ARM64 wheels...
> * Python tests are not running on any kind of ARM CPUs which makes any bug
> hunting/development hard or nearly impossible
>
> In order to give ARM support with local development + test execution
> possibility new miniconda is needed. New miniconda runs 3.8+.
>
> There is already a PR which is under development:
> https://github.com/apache/flink/pull/23359
> This fills all the mentioned gaps: local test/development/test execution on
> ARM CPUs + the previously mentioned 3+ years not actively supported python
> eliminated.
>
> Hope this helps.
>
> G
>
>
> On Wed, Sep 6, 2023 at 9:00 AM Xingbo Huang  wrote:
>
> > Hi Gyala and Gabor,
> >
> > Pyflink has provided arm64 wheel packages for Apple silicon since
> 1.16[1].
> > The use of Miniconda is only related to ci testing and packaging on linux
> > platform, and building mac platform wheels are dependent on
> > cibuildwheel[2]. So I guess you want to run ci tests  on the m1
> > environment, but the current version of miniconda cannot meet this
> > requirement, so there is a pre-step that must drop python 3.7?
> >
> > [1] https://pypi.org/project/apache-flink/1.16.2/#files
> > [2]
> >
> >
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml#L30
> >
> > Best,
> > Xingbo
> >
> > Gabor Somogyi  于2023年9月6日周三 14:36写道：
> >
> > > Hi Xingbo,
> > >
> > > *Constraint:*
> > > I personally not found any miniconda version which provides arm64
> support
> > > together with python 3.7.
> > > [image: image.png]
> > >
> > > At the moment I think new platform support means 3.7 drop.
> > >
> > > I fully to agree with Gyula, if we start now maybe we can release it in
> > > half a year however *3.7 active support already ended in 27 Jun 2020*.
> > > At the moment any python development/test execution on MacOS M1 is just
> > > not working as-is just like any kind of python test execution on any
> ARM
> > > CPU.
> > >
> > > Gains:
> > > * We can release a working version in half a year hopefully and not
> > > shifting support to 1+ year
> > > * MacOS M1 local development would work finally which is essential for
> > > user engagement
> > > * It would be possible to execute python tests on ARM64 machines
> > > * We can shake up the python development story because it's not the
> most
> > > loved area
> > >
> > > BR,
> > > G
> > >
> > >
> > > On Wed, Sep 6, 2023 at 8:06 AM Gyula Fóra 
> wrote:
> > >
> > >> Hi Xingbo!
> > >>
> > >> I think we have to analyze what we gain by dropping 3.7 and upgrading
> > to a
> > >> miniconda version with a multiarch support.
> > >>
> > >> If this is what we need to get Apple silicon support then I think it's
> > >> worth doing it already in 1.19. Keep in mind that 1.18 is not even
> > >> released
> > >> yet so if we delay this to 1.20 that is basically 1 year from now.
> > >> Making this change can increase the adoption instantly if we enable
> new
> > >> platforms.
> > >>
> > >> Cheers,
> > >> Gyula
> > >>
> > >> On Wed, Sep 6, 2023 at 4:46 AM Xingbo Huang 
> wrote:
> > >>
> > >> > Hi Gabor,
> > >> >
> > >> > Thanks for bringing this up. In my opinion, it is a bit aggressive
> to
> > >> > directly drop Python 3.7 in 1.19. Python 3.7 is still used a lot[1],
> > >> and as
> > >> > far as I know, many Pyflink users are still using python 3.7

Re: [DISCUSS] Drop python 3.7 support in 1.19

[Gabor Somogyi]

Hi Xingbo,

Thanks for your support!

I agree that multi-platform in general is a good idea. We start with ARM
but this opens the door for all others.

G


On Wed, Sep 6, 2023 at 10:46 AM Xingbo Huang  wrote:

> Hi Gabor,
>
> Thanks a lot for the detailed explanation. Building a test environment with
> multi-platforms is indeed what we have been lacking,
> not only for M1 users, but also for Windows users. At 1.16, I remember that
> Azure did not provide the m1 environment,
> and the github action just planned m1 in the annual plan at that
> time[1][2].
>
> +1 for drop 3.7 support in 1.19
>
> [1] https://github.com/actions/runner-images/issues/2187
> [2] https://github.com/github/roadmap/issues/528
>
> Best,
> Xingbo
>
> Gabor Somogyi  于2023年9月6日周三 16:07写道：
>
> > Hi Xingbo,
> >
> > > So I guess you want to run ci tests  on the m1
> > > environment, but the current version of miniconda cannot meet this
> > > requirement, so there is a pre-step that must drop python 3.7?
> >
> > In short yes + local wheel build fix on the M1. In a bit more detailed
> > please let me elaborate.
> >
> > You're right about the arm64 build creation, Flink is doing that for
> quite
> > some time already and wheels are running fine.
> > There are gaps however.
> >
> > From user perspective:
> > * we're building wheels for ARM64 but we're not executing any python
> tests
> > on ARM64 but only on X64.
> > I think it's a must to test platform packages on the related platform
> since
> > not all python packages are ARM compatible or must be built/used in a
> > different way.
> >
> > From dev perspective:
> > * Local MacOs M1 wheel or python source distro builds are simply failing
> > with error messages.
> > Just to give an example GRPC pip install is failing and require the
> > following changes:
> >
> >
> https://github.com/apache/flink/pull/23359/files#diff-c4c4916cd6ab7f87a4b154ef9603f9ba6d3b74a00096be9adacb7df363b5efcfR28-R32
> > I can't really imagine how Azure is able to build MacOS ARM64 wheels...
> > * Python tests are not running on any kind of ARM CPUs which makes any
> bug
> > hunting/development hard or nearly impossible
> >
> > In order to give ARM support with local development + test execution
> > possibility new miniconda is needed. New miniconda runs 3.8+.
> >
> > There is already a PR which is under development:
> > https://github.com/apache/flink/pull/23359
> > This fills all the mentioned gaps: local test/development/test execution
> on
> > ARM CPUs + the previously mentioned 3+ years not actively supported
> python
> > eliminated.
> >
> > Hope this helps.
> >
> > G
> >
> >
> > On Wed, Sep 6, 2023 at 9:00 AM Xingbo Huang  wrote:
> >
> > > Hi Gyala and Gabor,
> > >
> > > Pyflink has provided arm64 wheel packages for Apple silicon since
> > 1.16[1].
> > > The use of Miniconda is only related to ci testing and packaging on
> linux
> > > platform, and building mac platform wheels are dependent on
> > > cibuildwheel[2]. So I guess you want to run ci tests  on the m1
> > > environment, but the current version of miniconda cannot meet this
> > > requirement, so there is a pre-step that must drop python 3.7?
> > >
> > > [1] https://pypi.org/project/apache-flink/1.16.2/#files
> > > [2]
> > >
> > >
> >
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml#L30
> > >
> > > Best,
> > > Xingbo
> > >
> > > Gabor Somogyi  于2023年9月6日周三 14:36写道：
> > >
> > > > Hi Xingbo,
> > > >
> > > > *Constraint:*
> > > > I personally not found any miniconda version which provides arm64
> > support
> > > > together with python 3.7.
> > > > [image: image.png]
> > > >
> > > > At the moment I think new platform support means 3.7 drop.
> > > >
> > > > I fully to agree with Gyula, if we start now maybe we can release it
> in
> > > > half a year however *3.7 active support already ended in 27 Jun
> 2020*.
> > > > At the moment any python development/test execution on MacOS M1 is
> just
> > > > not working as-is just like any kind of python test execution on any
> > ARM
> > > > CPU.
> > > >
> > > > Gains:
> > > > * We can release a working version in half a year hopefully and not
> > > > shifting support to 1+ year
> > > > * MacOS M1 local development would work finally which is essential
> for
> > > > user engagement
> > > > * It would be possible to execute python tests on ARM64 machines
> > > > * We can shake up the python development story because it's not the
> > most
> > > > loved area
> > > >
> > > > BR,
> > > > G
> > > >
> > > >
> > > > On Wed, Sep 6, 2023 at 8:06 AM Gyula Fóra 
> > wrote:
> > > >
> > > >> Hi Xingbo!
> > > >>
> > > >> I think we have to analyze what we gain by dropping 3.7 and
> upgrading
> > > to a
> > > >> miniconda version with a multiarch support.
> > > >>
> > > >> If this is what we need to get Apple silicon support then I think
> it's
> > > >> worth doing it already in 1.19. Keep in mind that 1.18 is not even
> > > >> released
> > > >> yet so if we delay this to 1.20 that is basically 

[jira] [Created] (FLINK-33046) [FLIP-333] Redesign Apache Flink website

[Danny Cranmer (Jira)]

Danny Cranmer created FLINK-33046:
-

 Summary: [FLIP-333] Redesign Apache Flink website
 Key: FLINK-33046
 URL: https://issues.apache.org/jira/browse/FLINK-33046
 Project: Flink
  Issue Type: New Feature
  Components: Project Website
Reporter: Danny Cranmer


Apply styling as per 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-333%3A+Redesign+Apache+Flink+website



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33047) Project website refresh Dark Mode

[Danny Cranmer (Jira)]

Danny Cranmer created FLINK-33047:
-

 Summary: Project website refresh Dark Mode
 Key: FLINK-33047
 URL: https://issues.apache.org/jira/browse/FLINK-33047
 Project: Flink
  Issue Type: Sub-task
Reporter: Danny Cranmer






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33049) Project website refresh Code Samples Widget

[Danny Cranmer (Jira)]

Danny Cranmer created FLINK-33049:
-

 Summary: Project website refresh Code Samples Widget
 Key: FLINK-33049
 URL: https://issues.apache.org/jira/browse/FLINK-33049
 Project: Flink
  Issue Type: Sub-task
Reporter: Danny Cranmer






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33048) Project website refresh Light Mode

[Danny Cranmer (Jira)]

Danny Cranmer created FLINK-33048:
-

 Summary: Project website refresh Light Mode
 Key: FLINK-33048
 URL: https://issues.apache.org/jira/browse/FLINK-33048
 Project: Flink
  Issue Type: Sub-task
Reporter: Danny Cranmer






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Xintong Song]

>
> I added the average time metric to the FLIP document. I also included it
> for the aggregate (total) across all collectors. But maybe it doesn't make
> too much sense as collection times usually differ greatly depending on the
> collector.
>

LGTM


Best,

Xintong



On Wed, Sep 6, 2023 at 4:31 PM Gyula Fóra  wrote:

> I added the average time metric to the FLIP document. I also included it
> for the aggregate (total) across all collectors. But maybe it doesn't make
> too much sense as collection times usually differ greatly depending on the
> collector.
>
> Gyula
>
> On Wed, Sep 6, 2023 at 10:21 AM Xintong Song 
> wrote:
>
> > Thank you :)
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Sep 6, 2023 at 4:17 PM Gyula Fóra  wrote:
> >
> > > Makes sense Xintong, I am happy to extend the proposal with the average
> > gc
> > > time metric +1
> > >
> > > Gyula
> > >
> > > On Wed, Sep 6, 2023 at 10:09 AM Xintong Song 
> > > wrote:
> > >
> > > > >
> > > > > Just so I understand correctly, do you suggest adding a metric for
> > > > > delta(Time) / delta(Count) since the last reporting ?
> > > > > .TimePerGc or .AverageTime would make sense.
> > > > > AverageTime may be a bit nicer :)
> > > > >
> > > >
> > > > Yes, that's what I mean.
> > > >
> > > > My only concern is how useful this will be in reality. If there are
> > only
> > > > > (or several) long pauses then the msPerSec metrics will show it
> > > already,
> > > > > and if there is a single long pause that may not be shown at all if
> > > there
> > > > > are several shorter pauses as well with this metric.
> > > >
> > > >
> > > > Let's say we measure this for every minute and see a 900 msPerSec
> > (which
> > > > means 54s within the minute are spent on GC). This may come from a
> > single
> > > > GC that lasts for 54s, or 2 GCs each lasting for ~27s, or more GCs
> with
> > > > less time each. As the default heartbeat timeout is 50s, the former
> > means
> > > > there's likely a heartbeat timeout due to the GC pause, while the
> > latter
> > > > means otherwise.
> > > >
> > > >
> > > > Mathematically, it is possible that there's 1 long pause together
> with
> > > > several short pauses within the same measurement period, making the
> > long
> > > > pause not observable with AverageTime. However, from my experience,
> > such
> > > a
> > > > pattern is not normal in reality. My observation is that GCs happen
> at
> > a
> > > > similar time usually take a similar length of time. Admittedly, this
> is
> > > not
> > > > a hard guarantee.
> > > >
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Wed, Sep 6, 2023 at 3:59 PM Gyula Fóra 
> > wrote:
> > > >
> > > > > Matt Wang,
> > > > >
> > > > > I think the currently exposed info is all that is available through
> > > > > GarbageCollectorMXBean. This FLIP does not aim to introduce a new
> > more
> > > > > granular way of reporting the per collector metrics, that would
> > > require a
> > > > > new mechanism and may be a breaking change.
> > > > >
> > > > > We basically want to simply extend the current reporting here with
> > the
> > > > rate
> > > > > metrics and the total metrics.
> > > > >
> > > > > Gyula
> > > > >
> > > > > On Wed, Sep 6, 2023 at 9:24 AM Matt Wang  wrote:
> > > > >
> > > > > > Hi Gyula,
> > > > > >
> > > > > > +1 for this proposal.
> > > > > >
> > > > > > Do we need to add a metric to record the count of different
> > > > > > collectors? Now there is only a total count. For example,
> > > > > > for G1, there is no way to distinguish whether it is the
> > > > > > young generation or the old generation.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best,
> > > > > > Matt Wang
> > > > > >
> > > > > >
> > > > > >  Replied Message 
> > > > > > | From | Gyula Fóra |
> > > > > > | Date | 09/6/2023 15:03 |
> > > > > > | To |  |
> > > > > > | Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
> > > > > > Thanks Xintong!
> > > > > >
> > > > > > Just so I understand correctly, do you suggest adding a metric
> for
> > > > > > delta(Time) / delta(Count) since the last reporting ?
> > > > > > .TimePerGc or .AverageTime would make
> sense.
> > > > > > AverageTime may be a bit nicer :)
> > > > > >
> > > > > > My only concern is how useful this will be in reality. If there
> are
> > > > only
> > > > > > (or several) long pauses then the msPerSec metrics will show it
> > > > already,
> > > > > > and if there is a single long pause that may not be shown at all
> if
> > > > there
> > > > > > are several shorter pauses as well with this metric.
> > > > > >
> > > > > > Gyula
> > > > > >
> > > > > > On Wed, Sep 6, 2023 at 8:46 AM Xintong Song <
> tonysong...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Thanks for bringing this up, Gyula.
> > > > > >
> > > > > > The proposed changes make sense to me. +1 for them.
> > > > > >
> > > > > > In addition to the proposed changes, I wonder if we should also
> add
> > > > > > something 

Re: [DISCUSS] FLIP-334 : Decoupling autoscaler and kubernetes

[Rui Fan]

Hi Max,

As the FLIP mentioned, we have the plan to add the
alternative implementation.

First of all, we will develop a generic autoscaler. This generic
autoscaler will not have knowledge of specific jobs, and users
will have the flexibility to pass the JobAutoScalerContext
when utilizing the generic autoscaler. Communication with
Flink jobs can be achieved through the RestClusterClient.

   - The generic ScalingRealizer is based on the rescale API (FLIP-291).
   - The generic EventHandler is based on the logger.
   - The generic StateStore is based on the Heap. This means that the state
   information is stored in memory and can be lost if the autoscaler restarts.


Secondly, for yarn implementation, as Samrat mentioned,
There is currently no flink-yarn-operator, and we cannot
easily obtain the job list. We are not yet sure how to manage
yarn's flink jobs. In order to prevent the FLIP from being too huge,
after confirming with Gyula and Samrat before, it is decided
that the current FLIP will not implement the automated
yarn-autoscaler. And it will be a separate FLIP in the future.


After this part is finished, flink users or other flink platforms can easy
to use the autoscaler, they just pass the Context, and the autoscaler
can find the flink job using the RestClient.

The first part will be done in this FLIP. And we can discuss
whether the second part should be done in this FLIP as well.

Best,
Rui

On Wed, Sep 6, 2023 at 4:34 AM Samrat Deb  wrote:

> Hi Max,
>
> > are we planning to add an alternative implementation
> against the new interfaces?
>
> Yes, we are simultaneously working on the YARN implementation using the
> interface. During the initial interface design, we encountered some
> anomalies while implementing it in YARN.
>
> Once the interfaces are finalized, we will proceed to raise a pull request
> (PR) for YARN as well.
>
> Our initial approach was to create a decoupled interface as part of
> FLIP-334 and then implement it for YARN in the subsequent phase.
> However, if you recommend combining both phases, we can certainly consider
> that option.
>
> We look forward to hearing your thoughts on whether to have YARN
> implementation as part of FLIP-334 or seperate one ?
>
> Bests
> Samrat
>
>
>
> On Tue, Sep 5, 2023 at 8:41 PM Maximilian Michels  wrote:
>
> > Thanks Rui for the update!
> >
> > Alongside with the refactoring to decouple autoscaler logic from the
> > deployment logic, are we planning to add an alternative implementation
> > against the new interfaces? I think the best way to get the interfaces
> > right, is to have an alternative implementation in addition to
> > Kubernetes. YARN or a standalone mode implementation were already
> > mentioned. Ultimately, this is the reason we are doing the
> > refactoring. Without a new implementation, it becomes harder to
> > justify the refactoring work.
> >
> > Cheers,
> > Max
> >
> > On Tue, Sep 5, 2023 at 9:48 AM Rui Fan  wrote:
> > >
> > > After discussing this FLIP-334[1] offline with Gyula and Max,
> > > I updated the FLIP based on the latest conclusion.
> > >
> > > Big thanks to Gyula and Max for their professional advice!
> > >
> > > > Does the interface function of handlerRecommendedParallelism
> > > > in AutoScalerEventHandler conflict with
> > > > handlerScalingFailure/handlerScalingReport (one of the
> > > > handles the event of scale failure, and the other handles
> > > > the event of scale success).
> > > Hi Matt,
> > >
> > > You can take a look at the FLIP, I think the issue has been fixed.
> > > Currently, we introduced the ScalingRealizer and
> > > AutoScalerEventHandler interface.
> > >
> > > The ScalingRealizer handles scaling action.
> > >
> > > The AutoScalerEventHandler  interface handles loggable events.
> > >
> > >
> > > Looking forward to your feedback, thanks!
> > >
> > > [1] https://cwiki.apache.org/confluence/x/x4qzDw
> > >
> > > Best,
> > > Rui
> > >
> > > On Thu, Aug 24, 2023 at 10:55 AM Matt Wang  wrote:
> > >>
> > >> Sorry for the late reply, I still have a small question here:
> > >> Does the interface function of handlerRecommendedParallelism
> > >> in AutoScalerEventHandler conflict with
> > >> handlerScalingFailure/handlerScalingReport (one of the
> > >> handles the event of scale failure, and the other handles
> > >> the event of scale success).
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Best,
> > >> Matt Wang
> > >>
> > >>
> > >>  Replied Message 
> > >> | From | Rui Fan<1996fan...@gmail.com> |
> > >> | Date | 08/21/2023 17:41 |
> > >> | To |  |
> > >> | Cc | Maximilian Michels ,
> > >> Gyula Fóra ,
> > >> Matt Wang |
> > >> | Subject | Re: [DISCUSS] FLIP-334 : Decoupling autoscaler and
> > kubernetes |
> > >> Hi Max, Gyula and Matt,
> > >>
> > >> Do you have any other comments?
> > >>
> > >> The flink-kubernetes-operator 1.6 has been released recently,
> > >> it's a good time to kick off this FLIP.
> > >>
> > >> Please let me know if you have any questions or concerns,
> > >> looking forward 

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

[Xuannan Su]

Hi Jing,

Thank you for the clarification.

For the use case you mentioned, I believe we can utilize the
HybridSource, as updated in FLIP-309[1], to determine the backlog
status. For example, if the user wants to process data before time T
in batch mode and after time T in stream mode, they can set the first
source of the HybridSource to read up to time T and the last source of
the HybridSource to read from time T.

Best,
Xuannan

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog


On Mon, Sep 4, 2023 at 10:36 PM Jing Ge  wrote:
>
> Hi Xuannan,
>
> Thanks for the clarification.
>
> 3. Event time and process time are two different things. It might be rarely
> used, but conceptually, users can process data in the past within a
> specific time range in the streaming mode. All data before that range will
> be considered as backlog and needed to be processed in the batch mode,
> like, e.g. the Present Perfect Progressive tense used in English language.
>
> Best regards,
> Jing
>
> On Thu, Aug 31, 2023 at 4:45 AM Xuannan Su  wrote:
>
> > Hi Jing,
> >
> > Thanks for the reply.
> >
> > 1. You are absolutely right that the watermark lag threshold must be
> > carefully set with a thorough understanding of watermark generation. It is
> > crucial for users to take into account the WatermarkStrategy when setting
> > the watermark lag threshold.
> >
> > 2. Regarding pure processing-time based stream processing jobs,
> > alternative strategies will be implemented to determine whether the job is
> > processing backlog data. I have outlined two possible strategies below:
> >
> > - Based on the source operator's state. For example, when MySQL CDC source
> > is reading snapshot, it can claim isBacklog=true.
> > - Based on metrics. For example, when busyTimeMsPerSecond (or
> > backPressuredTimeMsPerSecond) > user_specified_threshold, then
> > isBacklog=true.
> >
> > As of the strategies proposed in this FLIP, it rely on generated
> > watermarks. Therefore, if a user intends for the job to detect backlog
> > status based on watermark, it is necessary to generate the watermark.
> >
> > 3. I'm afraid I'm not fully grasping your question. From my understanding,
> > it should work in both cases. When event times are close to the processing
> > time, resulting in watermarks close to the processing time, the job is not
> > processing backlog data. On the other hand, when event times are far from
> > processing time, causing watermarks to also be distant, if the lag
> > surpasses the defined threshold, the job is considered processing backlog
> > data.
> >
> > Best,
> > Xuannan
> >
> >
> > > On Aug 31, 2023, at 02:56, Jing Ge  wrote:
> > >
> > > Hi Xuannan,
> > >
> > > Thanks for the clarification. That is the part where I am trying to
> > > understand your thoughts. I have some follow-up questions:
> > >
> > > 1. It depends strongly on the watermarkStrategy and how customized
> > > watermark generation looks like. It mixes business logic with technical
> > > implementation and technical data processing mode. The value of the
> > > watermark lag threshold must be set very carefully. If the value is too
> > > small. any time, when the watermark generation logic is changed(business
> > > logic changes lead to the threshold getting exceeded), the same job might
> > > be running surprisingly in backlog processing mode, i.e. a butterfly
> > > effect. A comprehensive documentation is required to avoid any confusion
> > > for the users.
> > > 2. Like Jark already mentioned, use cases that do not have watermarks,
> > > like pure processing-time based stream processing[1] are not covered. It
> > is
> > > more or less a trade-off solution that does not support such use cases
> > and
> > > appropriate documentation is required. Forcing them to explicitly
> > generate
> > > watermarks that are never needed just because of this does not sound
> > like a
> > > proper solution.
> > > 3. If I am not mistaken, it only works for use cases where event times
> > are
> > > very close to the processing times, because the wall clock is used to
> > > calculate the watermark lag and the watermark is generated based on the
> > > event time.
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1]
> > >
> > https://github.com/apache/flink/blob/2c50b4e956305426f478b726d4de4a640a16b810/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkStrategy.java#L236
> > >
> > > On Wed, Aug 30, 2023 at 4:06 AM Xuannan Su 
> > wrote:
> > >
> > >> Hi Jing,
> > >>
> > >> Thank you for the suggestion.
> > >>
> > >> The definition of watermark lag is the same as the watermarkLag metric
> > in
> > >> FLIP-33[1]. More specifically, the watermark lag calculation is
> > computed at
> > >> the time when a watermark is emitted downstream in the following way:
> > >> watermarkLag = CurrentTime - Watermark. I have added this description to
> > >> the FLIP.
> > >>
> > >> I hope 

Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy

[Biao Geng]

+1 for the proposal.

Best,
Biao Geng

Gyula Fóra  于2023年9月6日周三 16:10写道：

> @Zhanghao Chen:
>
> I am not completely sure at this point what this will mean for 2.0 simply
> because I am also not sure what that will mean for the operator as well :)
> I think this will depend on the compatibility guarantees we can provide
> across Flink major versions in general. We have to look into that and
> tackle the question there independently.
>
> Gyula
>
> On Tue, Sep 5, 2023 at 6:12 PM Maximilian Michels  wrote:
>
> > +1 Sounds good! Four releases give a decent amount of time to migrate
> > to the next Flink version.
> >
> > On Tue, Sep 5, 2023 at 5:33 PM Őrhidi Mátyás 
> > wrote:
> > >
> > > +1
> > >
> > > On Tue, Sep 5, 2023 at 8:03 AM Thomas Weise  wrote:
> > >
> > > > +1, thanks for the proposal
> > > >
> > > > On Tue, Sep 5, 2023 at 8:13 AM Gyula Fóra 
> > wrote:
> > > >
> > > > > Hi All!
> > > > >
> > > > > @Maximilian Michels  has raised the question of
> > Flink
> > > > > version support in the operator before the last release. I would
> > like to
> > > > > open this discussion publicly so we can finalize this before the
> next
> > > > > release.
> > > > >
> > > > > Background:
> > > > > Currently the Flink Operator supports all Flink versions since
> Flink
> > > > 1.13.
> > > > > While this is great for the users, it introduces a lot of backward
> > > > > compatibility related code in the operator logic and also adds
> > > > considerable
> > > > > time to the CI. We should strike a reasonable balance here that
> > allows us
> > > > > to move forward and eliminate some of this tech debt.
> > > > >
> > > > > In the current model it is also impossible to support all features
> > for
> > > > all
> > > > > Flink versions which leads to some confusion over time.
> > > > >
> > > > > Proposal:
> > > > > Since it's a key feature of the kubernetes operator to support
> > several
> > > > > versions at the same time, I propose to support the last 4 stable
> > Flink
> > > > > minor versions. Currently this would mean to support Flink
> 1.14-1.17
> > (and
> > > > > drop 1.13 support). When Flink 1.18 is released we would drop 1.14
> > > > support
> > > > > and so on. Given the Flink release cadence this means about 2 year
> > > > support
> > > > > for each Flink version.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Cheers,
> > > > > Gyula
> > > > >
> > > >
> >
>

Re: [DISCUSS] FLIP-334 : Decoupling autoscaler and kubernetes

[Maximilian Michels]

Hey Rui, hey Samrat,

I want to ensure this is not just an exercise but has actual benefits
for the community. In the past, I've seen that the effort stops half
way through, the refactoring gets done with some regressions, but
actual alternative implementations based on the new design never
follow.

We need to go through these phases for the FLIP to be meaningful:

1. Decouple autoscaler from current autoscaler (generalization)
2. Ensure 100% functionality and test coverage of Kubernetes implementation
3. Interface with another backend (e.g. YARN or standalone)

If we don't follow through with this plan, I'm not sure we are better
off than with the current implementation. Apologies if I'm being a bit
strict here but the autoscaling code has become a critical
infrastructure component. We need to carefully weigh the pros and cons
here to avoid risks for our users, some of them using this code in
production and relying on it on a day to day basis.

That said, we are open to following through with the FLIP and we can
definitely help review code changes and build on the new design.

-Max


On Wed, Sep 6, 2023 at 11:26 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> Hi Max,
>
> As the FLIP mentioned, we have the plan to add the
> alternative implementation.
>
> First of all, we will develop a generic autoscaler. This generic
> autoscaler will not have knowledge of specific jobs, and users
> will have the flexibility to pass the JobAutoScalerContext
> when utilizing the generic autoscaler. Communication with
> Flink jobs can be achieved through the RestClusterClient.
>
>- The generic ScalingRealizer is based on the rescale API (FLIP-291).
>- The generic EventHandler is based on the logger.
>- The generic StateStore is based on the Heap. This means that the state
>information is stored in memory and can be lost if the autoscaler restarts.
>
>
> Secondly, for yarn implementation, as Samrat mentioned,
> There is currently no flink-yarn-operator, and we cannot
> easily obtain the job list. We are not yet sure how to manage
> yarn's flink jobs. In order to prevent the FLIP from being too huge,
> after confirming with Gyula and Samrat before, it is decided
> that the current FLIP will not implement the automated
> yarn-autoscaler. And it will be a separate FLIP in the future.
>
>
> After this part is finished, flink users or other flink platforms can easy
> to use the autoscaler, they just pass the Context, and the autoscaler
> can find the flink job using the RestClient.
>
> The first part will be done in this FLIP. And we can discuss
> whether the second part should be done in this FLIP as well.
>
> Best,
> Rui
>
> On Wed, Sep 6, 2023 at 4:34 AM Samrat Deb  wrote:
>
> > Hi Max,
> >
> > > are we planning to add an alternative implementation
> > against the new interfaces?
> >
> > Yes, we are simultaneously working on the YARN implementation using the
> > interface. During the initial interface design, we encountered some
> > anomalies while implementing it in YARN.
> >
> > Once the interfaces are finalized, we will proceed to raise a pull request
> > (PR) for YARN as well.
> >
> > Our initial approach was to create a decoupled interface as part of
> > FLIP-334 and then implement it for YARN in the subsequent phase.
> > However, if you recommend combining both phases, we can certainly consider
> > that option.
> >
> > We look forward to hearing your thoughts on whether to have YARN
> > implementation as part of FLIP-334 or seperate one ?
> >
> > Bests
> > Samrat
> >
> >
> >
> > On Tue, Sep 5, 2023 at 8:41 PM Maximilian Michels  wrote:
> >
> > > Thanks Rui for the update!
> > >
> > > Alongside with the refactoring to decouple autoscaler logic from the
> > > deployment logic, are we planning to add an alternative implementation
> > > against the new interfaces? I think the best way to get the interfaces
> > > right, is to have an alternative implementation in addition to
> > > Kubernetes. YARN or a standalone mode implementation were already
> > > mentioned. Ultimately, this is the reason we are doing the
> > > refactoring. Without a new implementation, it becomes harder to
> > > justify the refactoring work.
> > >
> > > Cheers,
> > > Max
> > >
> > > On Tue, Sep 5, 2023 at 9:48 AM Rui Fan  wrote:
> > > >
> > > > After discussing this FLIP-334[1] offline with Gyula and Max,
> > > > I updated the FLIP based on the latest conclusion.
> > > >
> > > > Big thanks to Gyula and Max for their professional advice!
> > > >
> > > > > Does the interface function of handlerRecommendedParallelism
> > > > > in AutoScalerEventHandler conflict with
> > > > > handlerScalingFailure/handlerScalingReport (one of the
> > > > > handles the event of scale failure, and the other handles
> > > > > the event of scale success).
> > > > Hi Matt,
> > > >
> > > > You can take a look at the FLIP, I think the issue has been fixed.
> > > > Currently, we introduced the ScalingRealizer and
> > > > AutoScalerEventHandler interface.
> > > >
> > >

Re: [DISCUSS] FLIP-358: flink-avro enhancement and cleanup

[吴 stephen]

Hi Becket,
I notice that a new config will introduce to Avro Format and user can input 
their own schema. Since the user can input their schema , should Avro Format 
support a validation utils that validate whether the input schema is compatible 
with table columns?

I’m modifying the Avro-Confulent Format in my team and want to make it 
serialize/deserialize by the schema exists on the schema-registry instead of 
using the schema generate by datatype. And I am thinking how to compare the 
datatype from the ddl with Avro schema. As I see the AvroSchemaConverter can 
transfer the Avro schema to datatype, can validation be simple as to judge 
whether the dataype from ddl is equal to datatype from Avro schema? If no, may 
I ask what's your opinion about the validation.

I'm interested in the flip. If there's anything I can help with, please feel 
free to reach out to me.

Best regards,
Stephen


> 2023年9月5日 下午3:15，Becket Qin  写道：
> 
> Hi Jing,
> 
> Thanks for the comments.
> 
> 1. "For the batch cases, currently the BulkFormat for DataStream is
>> missing" - true, and there is another option to leverage
>> StreamFormatAdapter[1]
>> 
> StreamFormatAdapter is internal and it requires a StreamFormat
> implementation for Avro files which does not exist either.
> 
> 2. "The following two interfaces should probably be marked as Public for
>> now and Deprecated once we deprecate the InputFormat / OutputFormat" -
>> would you like to share some background info of the deprecation of the
>> InputFormat / OutputFormat? It is for me a little bit weird to mark APIs as
>> public that are now known to be deprecated.
> 
> InputFormat and OutputFormat are legacy APIs for SourceFunction and
> SinkFunction. So when the SourceFunction and SinkFunction are deprecated,
> the InputFormat and OutputFormat should also be deprecated accordingly. As
> of now, technically speaking, we have not deprecated these two APIs. So,
> making them public for now is just to fix the stability annotation because
> they are already used publicly by the users.
> 
> 3. "Remove the PublicEvolving annotation for the following deprecated
>> classes. It does not make sense for an API to be PublicEvolving and
>> Deprecated at the same time" - this is very common in the Flink code base
>> to have PublicEvolving and Deprecated at the same time. APIs that do not
>> survive the PublicEvolving phase will be marked as deprecated in addition.
>> Removing PublicEvolving in this case will break Flink API graduation rule.
> 
> Both PublicEvolving and Deprecated are status in the API lifecycle, they
> are by definition mutually exclusive. When an API is marked as deprecated,
> either the functionality is completely going away, or another API is
> replacing the deprecated one. In either case, it does not make sense to
> evolve that API any more. Even though Flink has some APIs marked with both
> PublicEvolving and Deprecated at the same time, that does not make sense
> and needs to be fixed. If a PublicEvolving API is deprecated, it should
> only be marked as Deprecated, just like a Public API. I am not sure how
> this would violate the API graduation rule, can you explain?
> 
> By the way, there is another orthogonal abuse of the Deprecated annotation
> in the Flink code base. For private methods, we should not mark them as
> deprecated and leave the existing code base using it, while introducing a
> new method. This is a bad practice adding to technical debts. Instead, a
> proper refactor should be done immediately in the same patch to just remove
> that private method and migrate all the usage to the new method.
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> 
> 
> On Fri, Sep 1, 2023 at 12:00 AM Jing Ge  wrote:
> 
>> Hi Becket,
>> 
>> It is a very useful proposal, thanks for driving it. +1. I'd like to ask
>> some questions to make sure I understand your thoughts correctly:
>> 
>> 1. "For the batch cases, currently the BulkFormat for DataStream is
>> missing" - true, and there is another option to leverage
>> StreamFormatAdapter[1]
>> 2. "The following two interfaces should probably be marked as Public for
>> now and Deprecated once we deprecate the InputFormat / OutputFormat" -
>> would you like to share some background info of the deprecation of the
>> InputFormat / OutputFormat? It is for me a little bit weird to mark APIs as
>> public that are now known to be deprecated.
>> 3. "Remove the PublicEvolving annotation for the following deprecated
>> classes. It does not make sense for an API to be PublicEvolving and
>> Deprecated at the same time" - this is very common in the Flink code base
>> to have PublicEvolving and Deprecated at the same time. APIs that do not
>> survive the PublicEvolving phase will be marked as deprecated in addition.
>> Removing PublicEvolving in this case will break Flink API graduation rule.
>> 
>> Best regards,
>> Jing
>> 
>> 
>> 
>> [1]
>> 
>> https://github.com/apache/flink/blob/1d1247d4ae6d4313f7d952c4b2d66351314c9432/flink-connectors/f

[jira] [Created] (FLINK-33050) Prompts user to close when atomicity implementation is not supported

[tartarus (Jira)]

tartarus created FLINK-33050:


 Summary: Prompts user to close when atomicity implementation is 
not supported
 Key: FLINK-33050
 URL: https://issues.apache.org/jira/browse/FLINK-33050
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Affects Versions: 1.18.0
Reporter: tartarus


* When atomicity is enabled, an exception may occur when creating a 
DynamicTableSink, and we need to prompt the user to disable atomicity.
 * When we use InMemoryCatalog, RTAS drop table will only delete the metadata, 
not clean up the underlying data files, RTAS write data does not use overwrite 
semantics by default, so it looks like the data is duplicated, this problem 
needs to be clarified in the documentation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Add config to enable job stop with savepoint on exceeding tolerable checkpoint Failures

[Dongwoo Kim]

Hi Yanfei, Hangxiang.

Thank you for taking the time to review my suggestions.
I agree with Hangxiang that simply triggering a savepoint based on repeated
checkpoint failures isn't a reasonable approach.
Adding a specific condition, like *CHECKPOINT_ASYNC_EXCEPTION* as the
reason for the last checkpoint failure, could make it more practical,
although it's not a perfect solution.
Regarding restart policy my initial thought was to stop the job after
creating a savepoint.
I was open to further discussions about refining the restart policy,
especially if the community was interested in the idea of a primary/standby
checkpoint storage setup.
However, Hangxiang's suggestion to utilize Flink's REST API hadn't crossed
my mind, and it seems to address my needs well.

I'll try leveraging the REST API to implement a failover strategy of
checkpoint storage failure.
Thank you again for your insights, they've been extremely helpful.

Best Dongwoo,

2023년 9월 6일 (수) 오후 4:57, Hangxiang Yu 님이 작성:

> Hi, Dongwoo.
> IIUC, you mean using savepoint to store a snapshot to other storage if
> checkpoints fail multiple times due to some long lasting exceptions of
> external storage, right ?
> I think it's better to achieve this by an external tool instead of
> introducing a config like that:
> 1. it's not so easy to judge whether an exception occurs due to external
> storage or not sometimes, and it's not so reasonable that we just trigger a
> savepoint if checkpoints fail multiple times.
> 2. It's better to let some logic about triggering savepoint, e.g. periodic
> savepoint, triggering stop-with-savepoint, done by external tools or
> platform. As you could see from [1], we intend to make their scopes clear.
>
> Maybe you could check the status and failure message by [2] periodically in
> your external tool or platform and then trigger savepoint or
> stop-with-savepoint by REST API or CLI.
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/checkpoints_vs_savepoints/
> [2]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints
>
> On Wed, Sep 6, 2023 at 11:05 AM Yanfei Lei  wrote:
>
> > Hi Dongwoo,
> >
> > If the checkpoint has failed
> > `execution.checkpointing.tolerable-failed-checkpoints` times, then
> > stopWithSavepoint is likely to fail as well.
> > If stopWithSavepoint succeeds or fails, will the job just stop?  I am
> > more curious about how this option works with the restart strategy?
> >
> > Best,
> > Yanfei
> >
> >
> > Dongwoo Kim  于2023年9月4日周一 22:17写道：
> > >
> > > Hi all,
> > > I have a proposal that aims to enhance the flink application's
> > resilience in cases of unexpected failures in checkpoint storages like S3
> > or HDFS,
> > >
> > > [Background]
> > > When using self managed S3-compatible object storage, we faced
> > checkpoint async failures lasting for an extended period more than 30
> > minutes,
> > > leading to multiple job restarts and causing lags in our streaming
> > application.
> > >
> > > [Current Behavior]
> > > Currently, when the number of checkpoint failures exceeds a predefined
> > tolerable limit, flink will either restart or fail the job based on how
> > it's configured.
> > > In my opinion this does not handle scenarios where the checkpoint
> > storage itself may be unreliable or experiencing downtime.
> > >
> > > [Proposed Feature]
> > > I propose a config that allows for a graceful job stop with a savepoint
> > when the tolerable checkpoint failure limit is reached.
> > > Instead of restarting/failing the job when tolerable checkpoint failure
> > exceeds, when this new config is set to true just trigger
> stopWithSavepoint.
> > >
> > > This could offer the following benefits.
> > > - Indication of Checkpoint Storage State: Exceeding tolerable
> checkpoint
> > failures could indicate unstable checkpoint storage.
> > > - Automated Fallback Strategy: When combined with a monitoring cron
> job,
> > this feature could act as an automated fallback strategy for handling
> > unstable checkpoint storage.
> > >   The job would stop safely, take a savepoint, and then you could
> > automatically restart with different checkpoint storage configured like
> > switching from S3 to HDFS.
> > >
> > > For example let's say checkpoint path is configured to s3 and savepoint
> > path is configured to hdfs.
> > > When the new config is set to true the job stops with savepoint like
> > below when tolerable checkpoint failure exceeds.
> > > And we can restart the job from that savepoint while the checkpoint
> > configured as hdfs.
> > >
> > >
> > >
> > > Looking forward to hearing the community's thoughts on this proposal.
> > > And also want to ask how the community is handling long lasting
> unstable
> > checkpoint storage issues.
> > >
> > > Thanks in advance.
> > >
> > > Best dongwoo,
> >
>
>
> --
> Best,
> Hangxiang.
>

Re: [DISCUSS] Add config to enable job stop with savepoint on exceeding tolerable checkpoint Failures

[Martijn Visser]

Hi dongwoo,

Thanks for the proposal. I don't think this is a good idea, especially
since there's no guarantee that if a checkpoint has failed, that a
savepoint will work. I also don't think it's necessarily a good idea that
Flink needs to change because of an external factor: isn't there another
mechanism that you would normally use to determine that your storage layer
has an issue?

Best regards,

Martijn

On Wed, Sep 6, 2023 at 3:40 PM Dongwoo Kim  wrote:

> Hi Yanfei, Hangxiang.
>
> Thank you for taking the time to review my suggestions.
> I agree with Hangxiang that simply triggering a savepoint based on repeated
> checkpoint failures isn't a reasonable approach.
> Adding a specific condition, like *CHECKPOINT_ASYNC_EXCEPTION* as the
> reason for the last checkpoint failure, could make it more practical,
> although it's not a perfect solution.
> Regarding restart policy my initial thought was to stop the job after
> creating a savepoint.
> I was open to further discussions about refining the restart policy,
> especially if the community was interested in the idea of a primary/standby
> checkpoint storage setup.
> However, Hangxiang's suggestion to utilize Flink's REST API hadn't crossed
> my mind, and it seems to address my needs well.
>
> I'll try leveraging the REST API to implement a failover strategy of
> checkpoint storage failure.
> Thank you again for your insights, they've been extremely helpful.
>
> Best Dongwoo,
>
> 2023년 9월 6일 (수) 오후 4:57, Hangxiang Yu 님이 작성:
>
> > Hi, Dongwoo.
> > IIUC, you mean using savepoint to store a snapshot to other storage if
> > checkpoints fail multiple times due to some long lasting exceptions of
> > external storage, right ?
> > I think it's better to achieve this by an external tool instead of
> > introducing a config like that:
> > 1. it's not so easy to judge whether an exception occurs due to external
> > storage or not sometimes, and it's not so reasonable that we just
> trigger a
> > savepoint if checkpoints fail multiple times.
> > 2. It's better to let some logic about triggering savepoint, e.g.
> periodic
> > savepoint, triggering stop-with-savepoint, done by external tools or
> > platform. As you could see from [1], we intend to make their scopes
> clear.
> >
> > Maybe you could check the status and failure message by [2] periodically
> in
> > your external tool or platform and then trigger savepoint or
> > stop-with-savepoint by REST API or CLI.
> >
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/checkpoints_vs_savepoints/
> > [2]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints
> >
> > On Wed, Sep 6, 2023 at 11:05 AM Yanfei Lei  wrote:
> >
> > > Hi Dongwoo,
> > >
> > > If the checkpoint has failed
> > > `execution.checkpointing.tolerable-failed-checkpoints` times, then
> > > stopWithSavepoint is likely to fail as well.
> > > If stopWithSavepoint succeeds or fails, will the job just stop?  I am
> > > more curious about how this option works with the restart strategy?
> > >
> > > Best,
> > > Yanfei
> > >
> > >
> > > Dongwoo Kim  于2023年9月4日周一 22:17写道：
> > > >
> > > > Hi all,
> > > > I have a proposal that aims to enhance the flink application's
> > > resilience in cases of unexpected failures in checkpoint storages like
> S3
> > > or HDFS,
> > > >
> > > > [Background]
> > > > When using self managed S3-compatible object storage, we faced
> > > checkpoint async failures lasting for an extended period more than 30
> > > minutes,
> > > > leading to multiple job restarts and causing lags in our streaming
> > > application.
> > > >
> > > > [Current Behavior]
> > > > Currently, when the number of checkpoint failures exceeds a
> predefined
> > > tolerable limit, flink will either restart or fail the job based on how
> > > it's configured.
> > > > In my opinion this does not handle scenarios where the checkpoint
> > > storage itself may be unreliable or experiencing downtime.
> > > >
> > > > [Proposed Feature]
> > > > I propose a config that allows for a graceful job stop with a
> savepoint
> > > when the tolerable checkpoint failure limit is reached.
> > > > Instead of restarting/failing the job when tolerable checkpoint
> failure
> > > exceeds, when this new config is set to true just trigger
> > stopWithSavepoint.
> > > >
> > > > This could offer the following benefits.
> > > > - Indication of Checkpoint Storage State: Exceeding tolerable
> > checkpoint
> > > failures could indicate unstable checkpoint storage.
> > > > - Automated Fallback Strategy: When combined with a monitoring cron
> > job,
> > > this feature could act as an automated fallback strategy for handling
> > > unstable checkpoint storage.
> > > >   The job would stop safely, take a savepoint, and then you could
> > > automatically restart with different checkpoint storage configured like
> > > switching from S3 to HDFS.
> > > >
> > > > For example let's say checkpoint path is configured t

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

[Venkatakrishnan Sowrirajan]

The voting time for [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

 has passed. I'm closing the vote now.

There were 5 +1 votes which were binding and 3 +1 votes which were
non-binding:

- Jark Wu (binding)
- Martijn Visser (binding)
- Sergey Nuyanzin (binding)
- Becket Qin (binding)
- Jinsong Li (binding)

- Yuepeng Pan (non-binding)
- Conrad Jam (non-binding)
- Jiabao Sun (non-binding)

There were no -1 votes.

Thus, FLIP-356 has been accepted.

Thanks everyone for joining the discussion and giving feedback!

Regards
Venkata krishnan


On Tue, Sep 5, 2023 at 10:25 PM Jingsong Li  wrote:

> +1
>
> On Wed, Sep 6, 2023 at 1:18 PM Becket Qin  wrote:
> >
> > Thanks for pushing the FLIP through.
> >
> > +1 on the updated FLIP wiki.
> >
> > Cheers,
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Sep 6, 2023 at 1:12 PM Venkatakrishnan Sowrirajan <
> vsowr...@asu.edu>
> > wrote:
> >
> > > Based on the recent discussions in the thread [DISCUSS] FLIP-356:
> Support
> > > Nested Fields Filter Pushdown
> > > <
> https://urldefense.com/v3/__https://lists.apache.org/thread/686bhgwrrb4xmbfzlk60szwxos4z64t7__;!!IKRxdwAv5BmarQ!fsjoxJdthYZriAJalwMW9WrL898-EmmNnhula2SBLMaghtAqtI7jEmcCZ8gloPISdiYElPbFj5gmfViqMCswvQ$
> >, I made
> > > some changes to the FLIP-356
> > > <
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-356*3A*Support*Nested*Fields*Filter*Pushdown__;JSsrKysr!!IKRxdwAv5BmarQ!fsjoxJdthYZriAJalwMW9WrL898-EmmNnhula2SBLMaghtAqtI7jEmcCZ8gloPISdiYElPbFj5gmfVjJ1KdSjw$
> > > >.
> > > Unless anyone else has any concerns, we can continue with this vote to
> > > reach consensus.
> > >
> > > Regards
> > > Venkata krishnan
> > >
> > >
> > > On Tue, Sep 5, 2023 at 8:04 AM Sergey Nuyanzin 
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > On Tue, Sep 5, 2023 at 4:55 PM Jiabao Sun  > > > .invalid>
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Jiabao
> > > > >
> > > > >
> > > > > > 2023年9月5日 下午10:33，Martijn Visser  写道：
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > On Tue, Sep 5, 2023 at 4:16 PM ConradJam 
> > > wrote:
> > > > > >
> > > > > >> +1 (non-binding)
> > > > > >>
> > > > > >> Yuepeng Pan  于2023年9月1日周五 15:43写道：
> > > > > >>
> > > > > >>> +1 (non-binding)
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Yuepeng
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> At 2023-09-01 14:32:19, "Jark Wu"  wrote:
> > > > >  +1 (binding)
> > > > > 
> > > > >  Best,
> > > > >  Jark
> > > > > 
> > > > > > 2023年8月30日 02:40，Venkatakrishnan Sowrirajan <
> vsowr...@asu.edu>
> > > 写道：
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Thank you all for your feedback on FLIP-356. I'd like to
> start a
> > > > > vote.
> > > > > >
> > > > > > Discussion thread:
> > > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://lists.apache.org/thread/686bhgwrrb4xmbfzlk60szwxos4z64t7__;!!IKRxdwAv5BmarQ!eNR1R48e8jbqDCSdXqWj6bjfmP1uMn-IUIgVX3uXlgzYp_9rcf-nZOaAZ7KzFo2JwMAJPGYv8wfRxuRMAA$
> > > > > > FLIP:
> > > > > >
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-356*3A*Support*Nested*Fields*Filter*Pushdown__;JSsrKysr!!IKRxdwAv5BmarQ!eNR1R48e8jbqDCSdXqWj6bjfmP1uMn-IUIgVX3uXlgzYp_9rcf-nZOaAZ7KzFo2JwMAJPGYv8wdkI0waFw$
> > > > > >
> > > > > > Regards
> > > > > > Venkata krishnan
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Sergey
> > > >
> > >
>

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

[Bonnie Arogyam Varghese]

Hi Liu Ron,
 To answer your question,
   Security might not be the main reason for disabling this option but
other arguments brought forward by Timo. Let me know if you have any
further questions or concerns.

On Tue, Sep 5, 2023 at 9:35 PM Bonnie Arogyam Varghese <
bvargh...@confluent.io> wrote:

> It looks like it will be nice to have a config to disable hints. Any other
> thoughts/concerns before we can close this discussion?
>
> On Fri, Aug 18, 2023 at 7:43 AM Timo Walther  wrote:
>
>>  > lots of the streaming SQL syntax are extensions of SQL standard
>>
>> That is true. But hints are kind of a special case because they are not
>> even "part of Flink SQL" that's why they are written in a comment syntax.
>>
>> Anyway, I feel hints could be sometimes confusing for users because most
>> of them have no effect for streaming and long-term we could also set
>> some hints via the CompiledPlan. And if you have multiple teams,
>> non-skilled users should not play around with hints and leave the
>> decision to the system that might become smarter over time.
>>
>> Regards,
>> Timo
>>
>>
>> On 17.08.23 18:47, liu ron wrote:
>> > Hi, Bonnie
>> >
>> >> Options hints could be a security concern since users can override
>> > settings.
>> >
>> > I think this still doesn't answer my question
>> >
>> > Best,
>> > Ron
>> >
>> > Jark Wu  于2023年8月17日周四 19:51写道：
>> >
>> >> Sorry, I still don't understand why we need to disable the query hint.
>> >> It doesn't have the security problems as options hint. Bonnie said it
>> >> could affect performance, but that depends on users using it
>> explicitly.
>> >> If there is any performance problem, users can remove the hint.
>> >>
>> >> If we want to disable query hint just because it's an extension to SQL
>> >> standard.
>> >> I'm afraid we have to introduce a bunch of configuration, because lots
>> of
>> >> the streaming SQL syntax are extensions of SQL standard.
>> >>
>> >> Best,
>> >> Jark
>> >>
>> >> On Thu, 17 Aug 2023 at 15:43, Timo Walther  wrote:
>> >>
>> >>> +1 for this proposal.
>> >>>
>> >>> Not every data team would like to enable hints. Also because they are
>> an
>> >>> extension to the SQL standard. It might also be the case that custom
>> >>> rules would be overwritten otherwise. Setting hints could also be the
>> >>> exclusive task of a DevOp team.
>> >>>
>> >>> Regards,
>> >>> Timo
>> >>>
>> >>>
>> >>> On 17.08.23 09:30, Konstantin Knauf wrote:
>>  Hi Bonnie,
>> 
>>  this makes sense to me, in particular, given that we already have
>> this
>>  toggle for a different type of hints.
>> 
>>  Best,
>> 
>>  Konstantin
>> 
>>  Am Mi., 16. Aug. 2023 um 19:38 Uhr schrieb Bonnie Arogyam Varghese
>>  :
>> 
>> > Hi Liu,
>> >Options hints could be a security concern since users can
>> override
>> > settings. However, query hints specifically could affect
>> performance.
>> > Since we have a config to disable Options hint, I'm suggesting we
>> also
>> >>> have
>> > a config to disable Query hints.
>> >
>> > On Wed, Aug 16, 2023 at 9:41 AM liu ron  wrote:
>> >
>> >> Hi,
>> >>
>> >> Thanks for driving this proposal.
>> >>
>> >> Can you explain why you would need to disable query hints because
>> of
>> >> security issues? I don't really understand why query hints affects
>> >> security.
>> >>
>> >> Best,
>> >> Ron
>> >>
>> >> Bonnie Arogyam Varghese 
>> >> 于2023年8月16日周三
>> >> 23:59写道：
>> >>
>> >>> Platform providers may want to disable hints completely for
>> security
>> >>> reasons.
>> >>>
>> >>> Currently, there is a configuration to disable OPTIONS hint -
>> >>>
>> >>>
>> >>
>> >
>> >>>
>> >>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-dynamic-table-options-enabled
>> >>>
>> >>> However, there is no configuration available to disable QUERY
>> hints
>> >> -
>> >>>
>> >>>
>> >>
>> >
>> >>>
>> >>
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/hints/#query-hints
>> >>>
>> >>> The proposal is to add a new configuration:
>> >>>
>> >>> Name: table.query-options.enabled
>> >>> Description: Enable or disable the QUERY hint, if disabled, an
>> >>> exception would be thrown if any QUERY hints are specified
>> >>> Note: The default value will be set to true.
>> >>>
>> >>
>> >
>> 
>> 
>> >>>
>> >>>
>> >>
>> >
>>
>>

[jira] [Created] (FLINK-33051) GlobalFailureHandler interface should be retired in favor of LabeledGlobalFailureHandler

[Panagiotis Garefalakis (Jira)]

Panagiotis Garefalakis created FLINK-33051:
--

 Summary: GlobalFailureHandler interface should be retired in favor 
of LabeledGlobalFailureHandler
 Key: FLINK-33051
 URL: https://issues.apache.org/jira/browse/FLINK-33051
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Panagiotis Garefalakis


FLIP-304 introduced `LabeledGlobalFailureHandler` interface that is an 
extension of `GlobalFailureHandler` interface.  The later can thus be removed 
in the future to avoid the existence of interfaces with duplicate functions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

[Jing Ge]

Hi Xuannan,

I thought FLIP-328 will compete with FLIP-309 while setting the value of
the backlog. Understood. Thanks for the hint.

Best regards,
Jing

On Wed, Sep 6, 2023 at 12:12 PM Xuannan Su  wrote:

> Hi Jing,
>
> Thank you for the clarification.
>
> For the use case you mentioned, I believe we can utilize the
> HybridSource, as updated in FLIP-309[1], to determine the backlog
> status. For example, if the user wants to process data before time T
> in batch mode and after time T in stream mode, they can set the first
> source of the HybridSource to read up to time T and the last source of
> the HybridSource to read from time T.
>
> Best,
> Xuannan
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog
>
>
> On Mon, Sep 4, 2023 at 10:36 PM Jing Ge 
> wrote:
> >
> > Hi Xuannan,
> >
> > Thanks for the clarification.
> >
> > 3. Event time and process time are two different things. It might be
> rarely
> > used, but conceptually, users can process data in the past within a
> > specific time range in the streaming mode. All data before that range
> will
> > be considered as backlog and needed to be processed in the batch mode,
> > like, e.g. the Present Perfect Progressive tense used in English
> language.
> >
> > Best regards,
> > Jing
> >
> > On Thu, Aug 31, 2023 at 4:45 AM Xuannan Su 
> wrote:
> >
> > > Hi Jing,
> > >
> > > Thanks for the reply.
> > >
> > > 1. You are absolutely right that the watermark lag threshold must be
> > > carefully set with a thorough understanding of watermark generation.
> It is
> > > crucial for users to take into account the WatermarkStrategy when
> setting
> > > the watermark lag threshold.
> > >
> > > 2. Regarding pure processing-time based stream processing jobs,
> > > alternative strategies will be implemented to determine whether the
> job is
> > > processing backlog data. I have outlined two possible strategies below:
> > >
> > > - Based on the source operator's state. For example, when MySQL CDC
> source
> > > is reading snapshot, it can claim isBacklog=true.
> > > - Based on metrics. For example, when busyTimeMsPerSecond (or
> > > backPressuredTimeMsPerSecond) > user_specified_threshold, then
> > > isBacklog=true.
> > >
> > > As of the strategies proposed in this FLIP, it rely on generated
> > > watermarks. Therefore, if a user intends for the job to detect backlog
> > > status based on watermark, it is necessary to generate the watermark.
> > >
> > > 3. I'm afraid I'm not fully grasping your question. From my
> understanding,
> > > it should work in both cases. When event times are close to the
> processing
> > > time, resulting in watermarks close to the processing time, the job is
> not
> > > processing backlog data. On the other hand, when event times are far
> from
> > > processing time, causing watermarks to also be distant, if the lag
> > > surpasses the defined threshold, the job is considered processing
> backlog
> > > data.
> > >
> > > Best,
> > > Xuannan
> > >
> > >
> > > > On Aug 31, 2023, at 02:56, Jing Ge 
> wrote:
> > > >
> > > > Hi Xuannan,
> > > >
> > > > Thanks for the clarification. That is the part where I am trying to
> > > > understand your thoughts. I have some follow-up questions:
> > > >
> > > > 1. It depends strongly on the watermarkStrategy and how customized
> > > > watermark generation looks like. It mixes business logic with
> technical
> > > > implementation and technical data processing mode. The value of the
> > > > watermark lag threshold must be set very carefully. If the value is
> too
> > > > small. any time, when the watermark generation logic is
> changed(business
> > > > logic changes lead to the threshold getting exceeded), the same job
> might
> > > > be running surprisingly in backlog processing mode, i.e. a butterfly
> > > > effect. A comprehensive documentation is required to avoid any
> confusion
> > > > for the users.
> > > > 2. Like Jark already mentioned, use cases that do not have
> watermarks,
> > > > like pure processing-time based stream processing[1] are not
> covered. It
> > > is
> > > > more or less a trade-off solution that does not support such use
> cases
> > > and
> > > > appropriate documentation is required. Forcing them to explicitly
> > > generate
> > > > watermarks that are never needed just because of this does not sound
> > > like a
> > > > proper solution.
> > > > 3. If I am not mistaken, it only works for use cases where event
> times
> > > are
> > > > very close to the processing times, because the wall clock is used to
> > > > calculate the watermark lag and the watermark is generated based on
> the
> > > > event time.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > [1]
> > > >
> > >
> https://github.com/apache/flink/blob/2c50b4e956305426f478b726d4de4a640a16b810/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkStrategy.java#L236
> > > >
> > > > On Wed, Aug 30, 2023 at 4

Re: [DISCUSS] FLIP-358: flink-avro enhancement and cleanup

[Jing Ge]

Hi Becket,

Thanks for the clarification.


> StreamFormatAdapter is internal and it requires a StreamFormat
> implementation for Avro files which does not exist either.
>

I thought the cases 1-6 described in the FLIP mean there is a StreamFormat
implementation for Avro. That was my fault. I didn't understand it
correctly.


> InputFormat and OutputFormat are legacy APIs for SourceFunction and
> SinkFunction. So when the SourceFunction and SinkFunction are deprecated,
> the InputFormat and OutputFormat should also be deprecated accordingly. As
> of now, technically speaking, we have not deprecated these two APIs. So,
> making them public for now is just to fix the stability annotation because
> they are already used publicly by the users.
>

Since SourceFunction is already deprecated and we are working on
SinkFunction deprecation for 1.19, I would suggest directly
marking InputFormat and OutputFormat as deprecated. Because, once we mark
them as public in one release, users might start to use them(they are
public APIs). It will be weird for them to have freshly graduated public
APIs get deprecated just after one minor release.


> Both PublicEvolving and Deprecated are status in the API lifecycle, they
> are by definition mutually exclusive. When an API is marked as deprecated,
> either the functionality is completely going away, or another API is
> replacing the deprecated one. In either case, it does not make sense to
> evolve that API any more. Even though Flink has some APIs marked with both
> PublicEvolving and Deprecated at the same time, that does not make sense
> and needs to be fixed. If a PublicEvolving API is deprecated, it should
> only be marked as Deprecated, just like a Public API. I am not sure how
> this would violate the API graduation rule, can you explain?
>

According to the definition of PublicEvolving [1]:
"Classes and methods with this annotation are intended for public use and
have stable behavior.
 However, their interfaces and signatures are not considered to be stable
and might be changed
 across versions."

Let's think about it from users' point of view. Once APIs are marked as
PublicEvolving, it means the APIs are public, users will be using and
depending on them. If we remove @PublicEvolving between minor releases, it
means for me a regression. The APIs are downgraded from public(evolving)
back to non-public. They could even be removed in the next minor release,
since they only have @Deprecated annotation. No one knows they were
PublicEvolving if developers don't go through the git history (in most
cases on one will check and care the git history). This, for me, breaks the
contract of @PublicEvolving.

Best regards,
Jing


[1]
https://github.com/apache/flink/blob/40882d5a896830a7bb638890a4771ed7c24dac52/flink-annotations/src/main/java/org/apache/flink/annotation/PublicEvolving.java#L29

On Tue, Sep 5, 2023 at 9:16 AM Becket Qin  wrote:

> Hi Jing,
>
> Thanks for the comments.
>
> 1. "For the batch cases, currently the BulkFormat for DataStream is
> > missing" - true, and there is another option to leverage
> > StreamFormatAdapter[1]
> >
>


> StreamFormatAdapter is internal and it requires a StreamFormat
> implementation for Avro files which does not exist either.
>


>
> 2. "The following two interfaces should probably be marked as Public for
> > now and Deprecated once we deprecate the InputFormat / OutputFormat" -
> > would you like to share some background info of the deprecation of the
> > InputFormat / OutputFormat? It is for me a little bit weird to mark APIs
> as
> > public that are now known to be deprecated.
>
>

> InputFormat and OutputFormat are legacy APIs for SourceFunction and
> SinkFunction. So when the SourceFunction and SinkFunction are deprecated,
> the InputFormat and OutputFormat should also be deprecated accordingly. As
> of now, technically speaking, we have not deprecated these two APIs. So,
> making them public for now is just to fix the stability annotation because
> they are already used publicly by the users.
>


>
> 3. "Remove the PublicEvolving annotation for the following deprecated
> > classes. It does not make sense for an API to be PublicEvolving and
> > Deprecated at the same time" - this is very common in the Flink code base
> > to have PublicEvolving and Deprecated at the same time. APIs that do not
> > survive the PublicEvolving phase will be marked as deprecated in
> addition.
> > Removing PublicEvolving in this case will break Flink API graduation
> rule.
>
>

> Both PublicEvolving and Deprecated are status in the API lifecycle, they
> are by definition mutually exclusive. When an API is marked as deprecated,
> either the functionality is completely going away, or another API is
> replacing the deprecated one. In either case, it does not make sense to
> evolve that API any more. Even though Flink has some APIs marked with both
> PublicEvolving and Deprecated at the same time, that does not make sense
> and needs to be fixed. If a Public

[jira] [Created] (FLINK-33052) codespeed server is down

[Jing Ge (Jira)]

Jing Ge created FLINK-33052:
---

 Summary: codespeed server is down
 Key: FLINK-33052
 URL: https://issues.apache.org/jira/browse/FLINK-33052
 Project: Flink
  Issue Type: Bug
  Components: Test Infrastructure
Reporter: Jing Ge
Assignee: Jing Ge


No update in #flink-dev-benchmarks slack channel since 25th August.

It was a EC2 running in a legacy aws account. Currently on one know which 
account it is. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [NOTICE] Experimental Java 17 support now available on master

[Jing Ge]

Hi folks,

Sorry to come back to this thread, since I saw FLINK-32327 is already
closed and Chesnay announced on 16th June that Flink master branch (at that
time, now it should be both 1.18 and master branches) builds and runs with
Java 17 out-of-the-box. But according to the feedback from Kurt, I'd like
to double confirm with the following question:

@Chesnay @Mathias: Will the upcoming Flink 1.18.0 release support Java 17
even if there are known issues with Java records because of kryo 2.x?
Thanks!

Best regards,
Jing


On Sun, Jun 18, 2023 at 8:45 PM Kurt Ostfeld  wrote:

> I think there is some confusion:
>
> Chesnay, not me, recently checked in changes into master so that Flink
> will build + test + run with experimental support for Java 17 but with Kryo
> 2.x as-is so this will error with Java records. Chesnay created this
> particular email thread related to this work.
>
> I (Kurt), created a PR+FLIP several weeks ago for upgrading Kryo from 2.x
> to 5.x, with full backward compatibility for existing
> savepoints/checkpoints, that enables Flink to run on Java 17 with support
> for Java records. This isn't merged into master. I haven't gotten much
> feedback on this.
>
> I recently rebased the Kryo upgrade PR onto the master branch, whicch
> includes Chesnay commits. The PR branch was already running successfully on
> Java 17, Chesnay's changes enable Flink to build and run the CI test suite
> in Java 17 as well. However, without the Kryo upgrade, Flink isn't
> compatible with Java records.
>
> I'd be happy to follow the standard process and do the the FLIP vote, but
> before this is ready for a vote, this PR needs review + testing by someone
> other than me. Specifically, I'd like someone to try to create a Flink
> application that tries to break the upgrade process: either confirm that
> everything works or demonstrate an error scenario.
>
> The Kryo PR code is passing all automated CI tests, which include several
> tests covering backwards compatibility scenarios. I also created this
> simple application https://github.com/kurtostfeld/flink-kryo-upgrade-demo
> to create state with Flink 1.17 and test the upgrade process. From what I
> can see it works, but this would definitely need more testing from people
> other than just me.
>
>
>
> --- Original Message ---
> On Sunday, June 18th, 2023 at 7:41 AM, Jing Ge 
> wrote:
>
>
> >
> >
> > Hi Kurt,
> >
> > Thanks for your contribution. I am a little bit confused about the email
> > title, since your PR[1] is not merged into the master yet. I guess, with
> > "Experimental Java 17 support", you meant it is available on your branch
> > which is based on the master.
> >
> > If I am not mistaken, there is no vote thread of FLIP 317 on ML. Would
> you
> > like to follow the standard process[2] defined by the Flink community?
> > Thanks!
> >
> >
> > Best regards,
> > Jing
> >
> > [1] https://github.com/apache/flink/pull/22660
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >
> > On Sun, Jun 18, 2023 at 1:18 AM Kurt Ostfeld
> kurtostf...@proton.me.invalid
> >
> > wrote:
> >
> > > I built the Flink master branch and tried running this simple Flink app
> > > that uses a Java record:
> > >
> > >
> https://github.com/kurtostfeld/flink-kryo-upgrade-demo/blob/main/flink-record-demo/src/main/java/demo/app/Main.java
> > >
> > > It fails with the normal exception that Kryo 2.x throws when you try to
> > > serialize a Java record. The full stack trace is here:
> > > https://pastebin.com/HGhGKUWt
> > >
> > > I tried removing this line:
> > >
> > >
> https://github.com/kurtostfeld/flink-kryo-upgrade-demo/blob/main/flink-record-demo/src/main/java/demo/app/Main.java#L36
> > >
> > > and that had no impact, I got the same error.
> > >
> > > In the other thread, you said that the plan was to use PojoSerializer
> to
> > > serialize records rather than Kryo. Currently, the Flink code bases
> uses
> > > Kryo 2.x by default for generic user data types, and that will fail
> when
> > > the data type is a record or contains records. Ultimately, if Flink
> wants
> > > to fully support Java records, it seems that it has to move off of Kryo
> > > 2.x. PojoSerializer is part of what is basically a custom serialization
> > > library internal to Flink that is an alternative to Kryo. That's one
> > > option: move off of Kryo to a Flink-internal serialization library. The
> > > other two options are upgrade to the new Kryo or use a different
> > > serialization library.
> > >
> > > The Kryo 5.5.0 upgrade PR I submitted (
> > > https://github.com/apache/flink/pull/22660) with FLIP 317 (
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-317%3A+Upgrade+Kryo+from+2.24.0+to+5.5.0
> )
> > > works with records. The Flink app linked above that uses records works
> with
> > > the PR and that's what I posted to this mailing list a few weeks ago. I
> > > rebased the pull request on to the latest master branch and it's
> passing
> > > al

PyFlink logging

[Őrhidi Mátyás]

Hey folks,

I'm struggling to find in the code where we set/override the logging
configuration?

logging.basicConfig(level=logging.INFO)
logging.info("I'm printed")
env = StreamExecutionEnvironment.get_execution_environment()
logging.info("I'm not")

Can you give me some pointers?

Thanks,
Matyas

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

[Venkatakrishnan Sowrirajan]

Hi everyone,

Posted a PR (https://github.com/apache/flink/pull/23313) to add nested
fields filter pushdown. Please review. Thanks.

Regards
Venkata krishnan


On Tue, Sep 5, 2023 at 10:04 PM Venkatakrishnan Sowrirajan 
wrote:

> Based on an offline discussion with Becket Qin, I added *fieldIndices *
> back which is the field index of the nested field at every level to the 
> *NestedFieldReferenceExpression
> *in FLIP-356
> 
> *. *2 reasons to do it:
>
> 1. Agree with using *fieldIndices *as the only contract to refer to the
> column from the underlying datasource.
> 2. To keep it consistent with *FieldReferenceExpression*
>
> Having said that, I see that with *projection pushdown, *index of the
> fields are used whereas with *filter pushdown (*based on scanning few
> tablesources) *FieldReferenceExpression*'s name is used for eg: even in
> the Flink's *FileSystemTableSource, IcebergSource, JDBCDatsource*. This
> way, I feel the contract is not quite clear and explicit. Wanted to
> understand other's thoughts as well.
>
> Regards
> Venkata krishnan
>
>
> On Tue, Sep 5, 2023 at 5:34 PM Becket Qin  wrote:
>
>> Hi Venkata,
>>
>>
>> > Also I made minor changes to the *NestedFieldReferenceExpression,
>> *instead
>> > of *fieldIndexArray* we can just do away with *fieldNames *array that
>> > includes fieldName at every level for the nested field.
>>
>>
>> I don't think keeping only the field names array would work. At the end of
>> the day, the contract between Flink SQL and the connectors is based on the
>> indexes, not the names. Technically speaking, the connectors only emit a
>> bunch of RowData which is based on positions. The field names are added by
>> the SQL framework via the DDL for those RowData. In this sense, the
>> connectors may not be aware of the field names in Flink DDL at all. The
>> common language between Flink SQL and source is just positions. This is
>> also why ProjectionPushDown would work by only relying on the indexes, not
>> the field names. So I think the field index array is a must have here in
>> the NestedFieldReferenceExpression.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Fri, Sep 1, 2023 at 8:12 AM Venkatakrishnan Sowrirajan <
>> vsowr...@asu.edu>
>> wrote:
>>
>> > Gentle ping on the vote for FLIP-356: Support Nested fields filter
>> pushdown
>> > <
>> https://urldefense.com/v3/__https://www.mail-archive.com/dev@flink.apache.org/msg69289.html__;!!IKRxdwAv5BmarQ!bOW26WlafOQQcb32eWtUiXBAl0cTCK1C6iYhDI2f_z__eczudAWmTRvjDiZg6gzlXmPXrDV4KJS5cFxagFE$
>> >.
>> >
>> > Regards
>> > Venkata krishnan
>> >
>> >
>> > On Tue, Aug 29, 2023 at 9:18 PM Venkatakrishnan Sowrirajan <
>> > vsowr...@asu.edu>
>> > wrote:
>> >
>> > > Sure, will reference this discussion to resume where we started as
>> part
>> > of
>> > > the flip to refactor SupportsProjectionPushDown.
>> > >
>> > > On Tue, Aug 29, 2023, 7:22 PM Jark Wu  wrote:
>> > >
>> > >> I'm fine with this. `ReferenceExpression` and
>> > `SupportsProjectionPushDown`
>> > >> can be another FLIP. However, could you summarize the design of this
>> > part
>> > >> in the future part of the FLIP? This can be easier to get started
>> with
>> > in
>> > >> the future.
>> > >>
>> > >>
>> > >> Best,
>> > >> Jark
>> > >>
>> > >>
>> > >> On Wed, 30 Aug 2023 at 02:45, Venkatakrishnan Sowrirajan <
>> > >> vsowr...@asu.edu>
>> > >> wrote:
>> > >>
>> > >> > Thanks Jark. Sounds good.
>> > >> >
>> > >> > One more thing, earlier in my summary I mentioned,
>> > >> >
>> > >> > Introduce a new *ReferenceExpression* (or
>> *BaseReferenceExpression*)
>> > >> > > abstract class which will be extended by both
>> > >> *FieldReferenceExpression*
>> > >> > >  and *NestedFieldReferenceExpression* (to be introduced as part
>> of
>> > >> this
>> > >> > > FLIP)
>> > >> >
>> > >> > This can be punted for now and can be handled as part of
>> refactoring
>> > >> > SupportsProjectionPushDown.
>> > >> >
>> > >> > Also I made minor changes to the *NestedFieldReferenceExpression,
>> > >> *instead
>> > >> > of *fieldIndexArray* we can just do away with *fieldNames *array
>> that
>> > >> > includes fieldName at every level for the nested field.
>> > >> >
>> > >> > Updated the FLIP-357
>> > >> > <
>> > >> >
>> > >>
>> >
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-356*3A*Support*Nested*Fields*Filter*Pushdown__;JSsrKysr!!IKRxdwAv5BmarQ!YAk6kV4CYvUSPfpoUDQRs6VlbmJXVX8KOKqFxKbNDkUWKzShvwpkLRGkAV1tgV3EqClNrjGS-Ij86Q$
>> > >> > >
>> > >> > wiki as well.
>> > >> >
>> > >> > Regards
>> > >> > Venkata krishnan
>> > >> >
>> > >> >
>> > >> > On Tue, Aug 29, 2023 at 5:21 AM Jark Wu  wrote:
>> > >> >
>> > >> > > Hi Venkata,
>> > >> > >
>> > >> > > Your summary looks good to me. +1 to start a vote.
>> > >> > >
>> > >> > > I think we don't need "inputIndex" in
>> > NestedFieldReferenceExpression.
>> > >> > > Actually, I think it is also not needed in
>> Fi

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

[liu ron]

Hi, Boonie

I'm with Jark on why disable hint is needed if it won't affect security. If
users don't need to use hint, then they won't care about it and I don't
think it's going to be a nuisance. On top of that, Lookup Join Hint is very
useful for streaming jobs, and disabling the hint would result in users not
being able to use it.

Best,
Ron

Bonnie Arogyam Varghese  于2023年9月6日周三
23:52写道：

> Hi Liu Ron,
>  To answer your question,
>Security might not be the main reason for disabling this option but
> other arguments brought forward by Timo. Let me know if you have any
> further questions or concerns.
>
> On Tue, Sep 5, 2023 at 9:35 PM Bonnie Arogyam Varghese <
> bvargh...@confluent.io> wrote:
>
> > It looks like it will be nice to have a config to disable hints. Any
> other
> > thoughts/concerns before we can close this discussion?
> >
> > On Fri, Aug 18, 2023 at 7:43 AM Timo Walther  wrote:
> >
> >>  > lots of the streaming SQL syntax are extensions of SQL standard
> >>
> >> That is true. But hints are kind of a special case because they are not
> >> even "part of Flink SQL" that's why they are written in a comment
> syntax.
> >>
> >> Anyway, I feel hints could be sometimes confusing for users because most
> >> of them have no effect for streaming and long-term we could also set
> >> some hints via the CompiledPlan. And if you have multiple teams,
> >> non-skilled users should not play around with hints and leave the
> >> decision to the system that might become smarter over time.
> >>
> >> Regards,
> >> Timo
> >>
> >>
> >> On 17.08.23 18:47, liu ron wrote:
> >> > Hi, Bonnie
> >> >
> >> >> Options hints could be a security concern since users can override
> >> > settings.
> >> >
> >> > I think this still doesn't answer my question
> >> >
> >> > Best,
> >> > Ron
> >> >
> >> > Jark Wu  于2023年8月17日周四 19:51写道：
> >> >
> >> >> Sorry, I still don't understand why we need to disable the query
> hint.
> >> >> It doesn't have the security problems as options hint. Bonnie said it
> >> >> could affect performance, but that depends on users using it
> >> explicitly.
> >> >> If there is any performance problem, users can remove the hint.
> >> >>
> >> >> If we want to disable query hint just because it's an extension to
> SQL
> >> >> standard.
> >> >> I'm afraid we have to introduce a bunch of configuration, because
> lots
> >> of
> >> >> the streaming SQL syntax are extensions of SQL standard.
> >> >>
> >> >> Best,
> >> >> Jark
> >> >>
> >> >> On Thu, 17 Aug 2023 at 15:43, Timo Walther 
> wrote:
> >> >>
> >> >>> +1 for this proposal.
> >> >>>
> >> >>> Not every data team would like to enable hints. Also because they
> are
> >> an
> >> >>> extension to the SQL standard. It might also be the case that custom
> >> >>> rules would be overwritten otherwise. Setting hints could also be
> the
> >> >>> exclusive task of a DevOp team.
> >> >>>
> >> >>> Regards,
> >> >>> Timo
> >> >>>
> >> >>>
> >> >>> On 17.08.23 09:30, Konstantin Knauf wrote:
> >>  Hi Bonnie,
> >> 
> >>  this makes sense to me, in particular, given that we already have
> >> this
> >>  toggle for a different type of hints.
> >> 
> >>  Best,
> >> 
> >>  Konstantin
> >> 
> >>  Am Mi., 16. Aug. 2023 um 19:38 Uhr schrieb Bonnie Arogyam Varghese
> >>  :
> >> 
> >> > Hi Liu,
> >> >Options hints could be a security concern since users can
> >> override
> >> > settings. However, query hints specifically could affect
> >> performance.
> >> > Since we have a config to disable Options hint, I'm suggesting we
> >> also
> >> >>> have
> >> > a config to disable Query hints.
> >> >
> >> > On Wed, Aug 16, 2023 at 9:41 AM liu ron 
> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Thanks for driving this proposal.
> >> >>
> >> >> Can you explain why you would need to disable query hints because
> >> of
> >> >> security issues? I don't really understand why query hints
> affects
> >> >> security.
> >> >>
> >> >> Best,
> >> >> Ron
> >> >>
> >> >> Bonnie Arogyam Varghese 
> >> >> 于2023年8月16日周三
> >> >> 23:59写道：
> >> >>
> >> >>> Platform providers may want to disable hints completely for
> >> security
> >> >>> reasons.
> >> >>>
> >> >>> Currently, there is a configuration to disable OPTIONS hint -
> >> >>>
> >> >>>
> >> >>
> >> >
> >> >>>
> >> >>
> >>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-dynamic-table-options-enabled
> >> >>>
> >> >>> However, there is no configuration available to disable QUERY
> >> hints
> >> >> -
> >> >>>
> >> >>>
> >> >>
> >> >
> >> >>>
> >> >>
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/hints/#query-hints
> >> >>>
> >> >>> The proposal is to add a new configuration:
> >> >>>
> >> >>> Name: table.query-options.enabled
> >> >>> Description: Enable or disable 

来自Jiangchuan.Lee的邮件

[Jiangchuan.Lee]

Hi,

I want to contribute to Apache Flink. 
Would you please give me the contributor permission? 
My JIRA ID is LeeJiangchuan.

Re: Proposal for Implementing Keyed Watermarks in Apache Flink

[liu ron]

Hi Tawfik,

Fast and slow streaming in distributed scenarios leads to watermark
advancing too fast, which leads to lost data and is a headache in Flink.
Can't wait to read your research paper!

Best,
Ron

Yun Tang  于2023年9月6日周三 14:46写道：

> Hi Tawfik,
>
> Thanks for offering such a proposal, looking forward to your research
> paper!
>
> You could also ask the edit permission for Flink improvement proposals to
> create a new proposal if you want to contribute this to the community by
> yourself.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> Best
> Yun Tang
> 
> From: yuxia 
> Sent: Wednesday, September 6, 2023 12:31
> To: dev 
> Subject: Re: Proposal for Implementing Keyed Watermarks in Apache Flink
>
> Hi, Tawfik Yasser.
> Thanks for the proposal.
> It sounds exciting. I can't wait the research paper for more details.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "David Morávek" 
> 收件人: "dev" 
> 发送时间: 星期二, 2023年 9 月 05日 下午 4:36:51
> 主题: Re: Proposal for Implementing Keyed Watermarks in Apache Flink
>
> Hi Tawfik,
>
> It's exciting to see any ongoing research that tries to push Flink forward!
>
> The get the discussion started, can you please your paper with the
> community? Assessing the proposal without further context is tough.
>
> Best,
> D.
>
> On Mon, Sep 4, 2023 at 4:42 PM Tawfek Yasser Tawfek 
> wrote:
>
> > Dear Apache Flink Development Team,
> >
> > I hope this email finds you well. I am writing to propose an exciting new
> > feature for Apache Flink that has the potential to significantly enhance
> > its capabilities in handling unbounded streams of events, particularly in
> > the context of event-time windowing.
> >
> > As you may be aware, Apache Flink has been at the forefront of Big Data
> > Stream processing engines, leveraging windowing techniques to manage
> > unbounded event streams effectively. The accuracy of the results obtained
> > from these streams relies heavily on the ability to gather all relevant
> > input within a window. At the core of this process are watermarks, which
> > serve as unique timestamps marking the progression of events in time.
> >
> > However, our analysis has revealed a critical issue with the current
> > watermark generation method in Apache Flink. This method, which operates
> at
> > the input stream level, exhibits a bias towards faster sub-streams,
> > resulting in the unfortunate consequence of dropped events from slower
> > sub-streams. Our investigations showed that Apache Flink's conventional
> > watermark generation approach led to an alarming data loss of
> approximately
> > 33% when 50% of the keys around the median experienced delays. This loss
> > further escalated to over 37% when 50% of random keys were delayed.
> >
> > In response to this issue, we have authored a research paper outlining a
> > novel strategy named "keyed watermarks" to address data loss and
> > substantially enhance data processing accuracy, achieving at least 99%
> > accuracy in most scenarios.
> >
> > Moreover, we have conducted comprehensive comparative studies to evaluate
> > the effectiveness of our strategy against the conventional watermark
> > generation method, specifically in terms of event-time tracking accuracy.
> >
> > We believe that implementing keyed watermarks in Apache Flink can greatly
> > enhance its performance and reliability, making it an even more valuable
> > tool for organizations dealing with complex, high-throughput data
> > processing tasks.
> >
> > We kindly request your consideration of this proposal. We would be eager
> > to discuss further details, provide the full research paper, or
> collaborate
> > closely to facilitate the integration of this feature into Apache Flink.
> >
> > Thank you for your time and attention to this proposal. We look forward
> to
> > the opportunity to contribute to the continued success and evolution of
> > Apache Flink.
> >
> > Best Regards,
> >
> > Tawfik Yasser
> > Senior Teaching Assistant @ Nile University, Egypt
> > Email: tyas...@nu.edu.eg
> > LinkedIn: https://www.linkedin.com/in/tawfikyasser/
> >
>

Re: [DISCUSS] FLIP-358: flink-avro enhancement and cleanup

[Becket Qin]

Hi Jing,

Thanks for the explanation.

Since SourceFunction is already deprecated and we are working on
> SinkFunction deprecation for 1.19, I would suggest directly
> marking InputFormat and OutputFormat as deprecated. Because, once we mark
> them as public in one release, users might start to use them(they are
> public APIs). It will be weird for them to have freshly graduated public
> APIs get deprecated just after one minor release.

OK, then let's mark them as deprecated as well.

According to the definition of PublicEvolving [1]:
> "Classes and methods with this annotation are intended for public use and
> have stable behavior.
>  However, their interfaces and signatures are not considered to be stable
> and might be changed
>  across versions."


> Let's think about it from users' point of view. Once APIs are marked as
> PublicEvolving, it means the APIs are public, users will be using and
> depending on them. If we remove @PublicEvolving between minor releases, it
> means for me a regression. The APIs are downgraded from public(evolving)
> back to non-public. They could even be removed in the next minor release,
> since they only have @Deprecated annotation. No one knows they were
> PublicEvolving if developers don't go through the git history (in most
> cases on one will check and care the git history). This, for me, breaks the
> contract of @PublicEvolving.


A deprecated API is still a *public* API. In fact, the Deprecated
annotation should only be applied to public APIs. For internal APIs, an
immediate refactor should be done. So, I don't think it will break the API
contract.
That said, I think the part that confuses me with both PublicEvolving and
Deprecated is whether this API will still evolve or not, as Deprecated
basically means "public, no more change, to be removed", while
PublicEvolving indicates still evolving. But I guess by intuition users
will just consider the evolving part overridden by the deprecation. Maybe
it is fine to keep both. I'll update the FLIP.

Thanks,

Jiangjie (Becket) Qin


On Thu, Sep 7, 2023 at 12:35 AM Jing Ge  wrote:

> Hi Becket,
>
> Thanks for the clarification.
>
>
> > StreamFormatAdapter is internal and it requires a StreamFormat
> > implementation for Avro files which does not exist either.
> >
>
> I thought the cases 1-6 described in the FLIP mean there is a StreamFormat
> implementation for Avro. That was my fault. I didn't understand it
> correctly.
>
>
> > InputFormat and OutputFormat are legacy APIs for SourceFunction and
> > SinkFunction. So when the SourceFunction and SinkFunction are deprecated,
> > the InputFormat and OutputFormat should also be deprecated accordingly.
> As
> > of now, technically speaking, we have not deprecated these two APIs. So,
> > making them public for now is just to fix the stability annotation
> because
> > they are already used publicly by the users.
> >
>
> Since SourceFunction is already deprecated and we are working on
> SinkFunction deprecation for 1.19, I would suggest directly
> marking InputFormat and OutputFormat as deprecated. Because, once we mark
> them as public in one release, users might start to use them(they are
> public APIs). It will be weird for them to have freshly graduated public
> APIs get deprecated just after one minor release.
>
>
> > Both PublicEvolving and Deprecated are status in the API lifecycle, they
> > are by definition mutually exclusive. When an API is marked as
> deprecated,
> > either the functionality is completely going away, or another API is
> > replacing the deprecated one. In either case, it does not make sense to
> > evolve that API any more. Even though Flink has some APIs marked with
> both
> > PublicEvolving and Deprecated at the same time, that does not make sense
> > and needs to be fixed. If a PublicEvolving API is deprecated, it should
> > only be marked as Deprecated, just like a Public API. I am not sure how
> > this would violate the API graduation rule, can you explain?
> >
>
> According to the definition of PublicEvolving [1]:
> "Classes and methods with this annotation are intended for public use and
> have stable behavior.
>  However, their interfaces and signatures are not considered to be stable
> and might be changed
>  across versions."
>
> Let's think about it from users' point of view. Once APIs are marked as
> PublicEvolving, it means the APIs are public, users will be using and
> depending on them. If we remove @PublicEvolving between minor releases, it
> means for me a regression. The APIs are downgraded from public(evolving)
> back to non-public. They could even be removed in the next minor release,
> since they only have @Deprecated annotation. No one knows they were
> PublicEvolving if developers don't go through the git history (in most
> cases on one will check and care the git history). This, for me, breaks the
> contract of @PublicEvolving.
>
> Best regards,
> Jing
>
>
> [1]
>
> https://github.com/apache/flink/blob/40882d5a896830a7bb638890a4771ed7c

Re: 来自Jiangchuan.Lee的邮件

[Jane Chan]

Hi Jiangchuan,

If you already have a JIRA account, you can comment under the issue you're
interested in (typically starting with bug fixes or documentation and
outlining how you'll do it) and request the committer assign it to you.

Best,
Jane

On Thu, Sep 7, 2023 at 9:45 AM Jiangchuan.Lee 
wrote:

> Hi,
>
> I want to contribute to Apache Flink.
> Would you please give me the contributor permission?
> My JIRA ID is LeeJiangchuan.

Re: [DISCUSS] FLIP-358: flink-avro enhancement and cleanup

[Becket Qin]

Hi Stephen,

I don't think you should compare the DataType with the AvroSchema directly.
They are for different purposes and sometimes cannot be mapped in both
directions.

As of now, the following conversions are needed in Flink format:
1. Avro Schema -> Flink Table Schema (DataType). This is required when
registering the Flink table.
2. Flink Table Schema (DataType) -> Avro Schema. This is because after
projection pushdown, maybe only some of the fields need to be read from the
Avro record. So Flink Avro format needs to generate an Avro reader schema
from the projected fields represented in DataType.

The issue today is when you convert an AvroSchema_A in step 1 to get the
DataType, and try to convert that DataType back to AvroSchema_B,
AvroSchema_A and AvroSchema_B are not compatible. The idea is to use the
original AvroSchema_A as the assistance in step 2, so that AvroSchema_A and
AvroSchema_B are compatible. In your case, the Avro schema stored in the
schema registry will be that original Avro schema, i.e. AvroSchema_A.

Thanks,

Jiangjie (Becket) Qin

On Wed, Sep 6, 2023 at 8:32 PM 吴 stephen  wrote:

> Hi Becket,
> I notice that a new config will introduce to Avro Format and user can
> input their own schema. Since the user can input their schema , should Avro
> Format support a validation utils that validate whether the input schema is
> compatible with table columns?
>
> I’m modifying the Avro-Confulent Format in my team and want to make it
> serialize/deserialize by the schema exists on the schema-registry instead
> of using the schema generate by datatype. And I am thinking how to compare
> the datatype from the ddl with Avro schema. As I see the
> AvroSchemaConverter can transfer the Avro schema to datatype, can
> validation be simple as to judge whether the dataype from ddl is equal to
> datatype from Avro schema? If no, may I ask what's your opinion about the
> validation.
>
> I'm interested in the flip. If there's anything I can help with, please
> feel free to reach out to me.
>
> Best regards,
> Stephen
>
>
> > 2023年9月5日 下午3:15，Becket Qin  写道：
> >
> > Hi Jing,
> >
> > Thanks for the comments.
> >
> > 1. "For the batch cases, currently the BulkFormat for DataStream is
> >> missing" - true, and there is another option to leverage
> >> StreamFormatAdapter[1]
> >>
> > StreamFormatAdapter is internal and it requires a StreamFormat
> > implementation for Avro files which does not exist either.
> >
> > 2. "The following two interfaces should probably be marked as Public for
> >> now and Deprecated once we deprecate the InputFormat / OutputFormat" -
> >> would you like to share some background info of the deprecation of the
> >> InputFormat / OutputFormat? It is for me a little bit weird to mark
> APIs as
> >> public that are now known to be deprecated.
> >
> > InputFormat and OutputFormat are legacy APIs for SourceFunction and
> > SinkFunction. So when the SourceFunction and SinkFunction are deprecated,
> > the InputFormat and OutputFormat should also be deprecated accordingly.
> As
> > of now, technically speaking, we have not deprecated these two APIs. So,
> > making them public for now is just to fix the stability annotation
> because
> > they are already used publicly by the users.
> >
> > 3. "Remove the PublicEvolving annotation for the following deprecated
> >> classes. It does not make sense for an API to be PublicEvolving and
> >> Deprecated at the same time" - this is very common in the Flink code
> base
> >> to have PublicEvolving and Deprecated at the same time. APIs that do not
> >> survive the PublicEvolving phase will be marked as deprecated in
> addition.
> >> Removing PublicEvolving in this case will break Flink API graduation
> rule.
> >
> > Both PublicEvolving and Deprecated are status in the API lifecycle, they
> > are by definition mutually exclusive. When an API is marked as
> deprecated,
> > either the functionality is completely going away, or another API is
> > replacing the deprecated one. In either case, it does not make sense to
> > evolve that API any more. Even though Flink has some APIs marked with
> both
> > PublicEvolving and Deprecated at the same time, that does not make sense
> > and needs to be fixed. If a PublicEvolving API is deprecated, it should
> > only be marked as Deprecated, just like a Public API. I am not sure how
> > this would violate the API graduation rule, can you explain?
> >
> > By the way, there is another orthogonal abuse of the Deprecated
> annotation
> > in the Flink code base. For private methods, we should not mark them as
> > deprecated and leave the existing code base using it, while introducing a
> > new method. This is a bad practice adding to technical debts. Instead, a
> > proper refactor should be done immediately in the same patch to just
> remove
> > that private method and migrate all the usage to the new method.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Fri, Sep 1, 2023 at 12:00 AM Jing Ge 
> wrote:
> >
> 

[DISCUSS][FLINK-31788][FLINK-33015] Add back Support emitUpdateWithRetract for TableAggregateFunction

[Jane Chan]

Hi devs,

Recently, we noticed an issue regarding a feature regression related to
Table API. `org.apache.flink.table.functions.TableAggregateFunction`
provides an API `emitUpdateWithRetract` [1] to cope with updated values,
but it's not being called in the code generator. As a result, even if users
override this method, it does not work as intended.

This issue has been present since version 1.15 (when the old planner was
deprecated), but surprisingly, only two users have raised concerns about it
[2][3].

So, I would like to initiate a discussion to bring it back. Of course, if
few users use it, we can also consider deprecating it.

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#retraction-example
[2] https://lists.apache.org/thread/rnvw8k3636dqhdttpmf1c9colbpw9svp
[3] https://www.mail-archive.com/user-zh@flink.apache.org/msg15230.html

Best,
Jane

[jira] [Created] (FLINK-33053) Watcher leak in Zookeeper HA mode

[Yangze Guo (Jira)]

Yangze Guo created FLINK-33053:
--

 Summary: Watcher leak in Zookeeper HA mode
 Key: FLINK-33053
 URL: https://issues.apache.org/jira/browse/FLINK-33053
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.17.1, 1.17.0
Reporter: Yangze Guo


We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA 
mode. TM's watches on the leader of JobMaster has not been stopped after job 
finished.

Here is how we re-produce this issue:
- Start a session cluster and enable Zookeeper HA mode.
- Continuously and concurrently submit short queries, e.g. WordCount to the 
cluster.
- `echo -n wchp | nc {zk host} {zk port}` to get current watches.

We can see a lot of watches on 
~/flink/{cluster_name}/leader/{job_id}/connection_info~.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Proposal for Implementing Keyed Watermarks in Apache Flink

[Jane Chan]

Hi Tawfik,

In response to this issue, we have authored a research paper outlining a
> novel strategy named "keyed watermarks" to address data loss and
> substantially enhance data processing accuracy, achieving at least 99%
> accuracy in most scenarios.
>

Sounds like a significant improvement! Looking forward to the details of
your research.

Best,
Jane

On Thu, Sep 7, 2023 at 9:50 AM liu ron  wrote:

> Hi Tawfik,
>
> Fast and slow streaming in distributed scenarios leads to watermark
> advancing too fast, which leads to lost data and is a headache in Flink.
> Can't wait to read your research paper!
>
> Best,
> Ron
>
> Yun Tang  于2023年9月6日周三 14:46写道：
>
> > Hi Tawfik,
> >
> > Thanks for offering such a proposal, looking forward to your research
> > paper!
> >
> > You could also ask the edit permission for Flink improvement proposals to
> > create a new proposal if you want to contribute this to the community by
> > yourself.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >
> > Best
> > Yun Tang
> > 
> > From: yuxia 
> > Sent: Wednesday, September 6, 2023 12:31
> > To: dev 
> > Subject: Re: Proposal for Implementing Keyed Watermarks in Apache Flink
> >
> > Hi, Tawfik Yasser.
> > Thanks for the proposal.
> > It sounds exciting. I can't wait the research paper for more details.
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "David Morávek" 
> > 收件人: "dev" 
> > 发送时间: 星期二, 2023年 9 月 05日 下午 4:36:51
> > 主题: Re: Proposal for Implementing Keyed Watermarks in Apache Flink
> >
> > Hi Tawfik,
> >
> > It's exciting to see any ongoing research that tries to push Flink
> forward!
> >
> > The get the discussion started, can you please your paper with the
> > community? Assessing the proposal without further context is tough.
> >
> > Best,
> > D.
> >
> > On Mon, Sep 4, 2023 at 4:42 PM Tawfek Yasser Tawfek 
> > wrote:
> >
> > > Dear Apache Flink Development Team,
> > >
> > > I hope this email finds you well. I am writing to propose an exciting
> new
> > > feature for Apache Flink that has the potential to significantly
> enhance
> > > its capabilities in handling unbounded streams of events, particularly
> in
> > > the context of event-time windowing.
> > >
> > > As you may be aware, Apache Flink has been at the forefront of Big Data
> > > Stream processing engines, leveraging windowing techniques to manage
> > > unbounded event streams effectively. The accuracy of the results
> obtained
> > > from these streams relies heavily on the ability to gather all relevant
> > > input within a window. At the core of this process are watermarks,
> which
> > > serve as unique timestamps marking the progression of events in time.
> > >
> > > However, our analysis has revealed a critical issue with the current
> > > watermark generation method in Apache Flink. This method, which
> operates
> > at
> > > the input stream level, exhibits a bias towards faster sub-streams,
> > > resulting in the unfortunate consequence of dropped events from slower
> > > sub-streams. Our investigations showed that Apache Flink's conventional
> > > watermark generation approach led to an alarming data loss of
> > approximately
> > > 33% when 50% of the keys around the median experienced delays. This
> loss
> > > further escalated to over 37% when 50% of random keys were delayed.
> > >
> > > In response to this issue, we have authored a research paper outlining
> a
> > > novel strategy named "keyed watermarks" to address data loss and
> > > substantially enhance data processing accuracy, achieving at least 99%
> > > accuracy in most scenarios.
> > >
> > > Moreover, we have conducted comprehensive comparative studies to
> evaluate
> > > the effectiveness of our strategy against the conventional watermark
> > > generation method, specifically in terms of event-time tracking
> accuracy.
> > >
> > > We believe that implementing keyed watermarks in Apache Flink can
> greatly
> > > enhance its performance and reliability, making it an even more
> valuable
> > > tool for organizations dealing with complex, high-throughput data
> > > processing tasks.
> > >
> > > We kindly request your consideration of this proposal. We would be
> eager
> > > to discuss further details, provide the full research paper, or
> > collaborate
> > > closely to facilitate the integration of this feature into Apache
> Flink.
> > >
> > > Thank you for your time and attention to this proposal. We look forward
> > to
> > > the opportunity to contribute to the continued success and evolution of
> > > Apache Flink.
> > >
> > > Best Regards,
> > >
> > > Tawfik Yasser
> > > Senior Teaching Assistant @ Nile University, Egypt
> > > Email: tyas...@nu.edu.eg
> > > LinkedIn: https://www.linkedin.com/in/tawfikyasser/
> > >
> >
>

[jira] [Created] (FLINK-33054) Align the job execution result fetching timeout in CollectResultFetcher to akka timeout

[Yangze Guo (Jira)]

Yangze Guo created FLINK-33054:
--

 Summary: Align the job execution result fetching timeout in 
CollectResultFetcher to akka timeout
 Key: FLINK-33054
 URL: https://issues.apache.org/jira/browse/FLINK-33054
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Gateway
Affects Versions: 1.18.0
Reporter: Yangze Guo
Assignee: Yangze Guo
 Fix For: 1.19.0


Currently in CollectResultFetcher, the job execution result will be fetched 
after job's termination. In FLINK-17735, we arbitrarily introduce a static 
timeout for this rpc call. However, in OLAP scenario, the Dispatcher endpoint 
might be too busy to reply in time. We'd like to set it according to the akka 
ask timeout, which is commonly used as rpc timeout in Flink.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

[Xuannan Su]

Hi all,

Thank you for all the reviews and suggestions.

I believe all the comments have been addressed. If there are no
further comments, I plan to open the voting thread for this FLIP early
next week.

Best regards,
Xuannan

On Thu, Sep 7, 2023 at 12:09 AM Jing Ge  wrote:
>
> Hi Xuannan,
>
> I thought FLIP-328 will compete with FLIP-309 while setting the value of
> the backlog. Understood. Thanks for the hint.
>
> Best regards,
> Jing
>
> On Wed, Sep 6, 2023 at 12:12 PM Xuannan Su  wrote:
>
> > Hi Jing,
> >
> > Thank you for the clarification.
> >
> > For the use case you mentioned, I believe we can utilize the
> > HybridSource, as updated in FLIP-309[1], to determine the backlog
> > status. For example, if the user wants to process data before time T
> > in batch mode and after time T in stream mode, they can set the first
> > source of the HybridSource to read up to time T and the last source of
> > the HybridSource to read from time T.
> >
> > Best,
> > Xuannan
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog
> >
> >
> > On Mon, Sep 4, 2023 at 10:36 PM Jing Ge 
> > wrote:
> > >
> > > Hi Xuannan,
> > >
> > > Thanks for the clarification.
> > >
> > > 3. Event time and process time are two different things. It might be
> > rarely
> > > used, but conceptually, users can process data in the past within a
> > > specific time range in the streaming mode. All data before that range
> > will
> > > be considered as backlog and needed to be processed in the batch mode,
> > > like, e.g. the Present Perfect Progressive tense used in English
> > language.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Aug 31, 2023 at 4:45 AM Xuannan Su 
> > wrote:
> > >
> > > > Hi Jing,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 1. You are absolutely right that the watermark lag threshold must be
> > > > carefully set with a thorough understanding of watermark generation.
> > It is
> > > > crucial for users to take into account the WatermarkStrategy when
> > setting
> > > > the watermark lag threshold.
> > > >
> > > > 2. Regarding pure processing-time based stream processing jobs,
> > > > alternative strategies will be implemented to determine whether the
> > job is
> > > > processing backlog data. I have outlined two possible strategies below:
> > > >
> > > > - Based on the source operator's state. For example, when MySQL CDC
> > source
> > > > is reading snapshot, it can claim isBacklog=true.
> > > > - Based on metrics. For example, when busyTimeMsPerSecond (or
> > > > backPressuredTimeMsPerSecond) > user_specified_threshold, then
> > > > isBacklog=true.
> > > >
> > > > As of the strategies proposed in this FLIP, it rely on generated
> > > > watermarks. Therefore, if a user intends for the job to detect backlog
> > > > status based on watermark, it is necessary to generate the watermark.
> > > >
> > > > 3. I'm afraid I'm not fully grasping your question. From my
> > understanding,
> > > > it should work in both cases. When event times are close to the
> > processing
> > > > time, resulting in watermarks close to the processing time, the job is
> > not
> > > > processing backlog data. On the other hand, when event times are far
> > from
> > > > processing time, causing watermarks to also be distant, if the lag
> > > > surpasses the defined threshold, the job is considered processing
> > backlog
> > > > data.
> > > >
> > > > Best,
> > > > Xuannan
> > > >
> > > >
> > > > > On Aug 31, 2023, at 02:56, Jing Ge 
> > wrote:
> > > > >
> > > > > Hi Xuannan,
> > > > >
> > > > > Thanks for the clarification. That is the part where I am trying to
> > > > > understand your thoughts. I have some follow-up questions:
> > > > >
> > > > > 1. It depends strongly on the watermarkStrategy and how customized
> > > > > watermark generation looks like. It mixes business logic with
> > technical
> > > > > implementation and technical data processing mode. The value of the
> > > > > watermark lag threshold must be set very carefully. If the value is
> > too
> > > > > small. any time, when the watermark generation logic is
> > changed(business
> > > > > logic changes lead to the threshold getting exceeded), the same job
> > might
> > > > > be running surprisingly in backlog processing mode, i.e. a butterfly
> > > > > effect. A comprehensive documentation is required to avoid any
> > confusion
> > > > > for the users.
> > > > > 2. Like Jark already mentioned, use cases that do not have
> > watermarks,
> > > > > like pure processing-time based stream processing[1] are not
> > covered. It
> > > > is
> > > > > more or less a trade-off solution that does not support such use
> > cases
> > > > and
> > > > > appropriate documentation is required. Forcing them to explicitly
> > > > generate
> > > > > watermarks that are never needed just because of this does not sound
> > > > like a
> > > > > proper solution.
> > > > > 3. If I am not mist

Re: [DISCUSS] FLIP-334 : Decoupling autoscaler and kubernetes

[Rui Fan]

Hi Max,

Thanks for your feedback!

> We need to go through these phases for the FLIP to be meaningful:
> 1. Decouple autoscaler from current autoscaler (generalization)
> 2. Ensure 100% functionality and test coverage of Kubernetes
implementation
> 3. Interface with another backend (e.g. YARN or standalone)

These phases make sense to me.

I have updated the FLIP[1] based on our offline discussion.
I added the `5. Standalone AutoScaler` to explain the generic
autoscaler.

Please correct me if anything is wrong or missed, thanks again!

[1] https://cwiki.apache.org/confluence/x/x4qzDw

Best,
Rui

On Wed, Sep 6, 2023 at 7:00 PM Maximilian Michels  wrote:

> Hey Rui, hey Samrat,
>
> I want to ensure this is not just an exercise but has actual benefits
> for the community. In the past, I've seen that the effort stops half
> way through, the refactoring gets done with some regressions, but
> actual alternative implementations based on the new design never
> follow.
>
> We need to go through these phases for the FLIP to be meaningful:
>
> 1. Decouple autoscaler from current autoscaler (generalization)
> 2. Ensure 100% functionality and test coverage of Kubernetes implementation
> 3. Interface with another backend (e.g. YARN or standalone)
>
> If we don't follow through with this plan, I'm not sure we are better
> off than with the current implementation. Apologies if I'm being a bit
> strict here but the autoscaling code has become a critical
> infrastructure component. We need to carefully weigh the pros and cons
> here to avoid risks for our users, some of them using this code in
> production and relying on it on a day to day basis.
>
> That said, we are open to following through with the FLIP and we can
> definitely help review code changes and build on the new design.
>
> -Max
>
>
> On Wed, Sep 6, 2023 at 11:26 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Hi Max,
> >
> > As the FLIP mentioned, we have the plan to add the
> > alternative implementation.
> >
> > First of all, we will develop a generic autoscaler. This generic
> > autoscaler will not have knowledge of specific jobs, and users
> > will have the flexibility to pass the JobAutoScalerContext
> > when utilizing the generic autoscaler. Communication with
> > Flink jobs can be achieved through the RestClusterClient.
> >
> >- The generic ScalingRealizer is based on the rescale API (FLIP-291).
> >- The generic EventHandler is based on the logger.
> >- The generic StateStore is based on the Heap. This means that the
> state
> >information is stored in memory and can be lost if the autoscaler
> restarts.
> >
> >
> > Secondly, for yarn implementation, as Samrat mentioned,
> > There is currently no flink-yarn-operator, and we cannot
> > easily obtain the job list. We are not yet sure how to manage
> > yarn's flink jobs. In order to prevent the FLIP from being too huge,
> > after confirming with Gyula and Samrat before, it is decided
> > that the current FLIP will not implement the automated
> > yarn-autoscaler. And it will be a separate FLIP in the future.
> >
> >
> > After this part is finished, flink users or other flink platforms can
> easy
> > to use the autoscaler, they just pass the Context, and the autoscaler
> > can find the flink job using the RestClient.
> >
> > The first part will be done in this FLIP. And we can discuss
> > whether the second part should be done in this FLIP as well.
> >
> > Best,
> > Rui
> >
> > On Wed, Sep 6, 2023 at 4:34 AM Samrat Deb  wrote:
> >
> > > Hi Max,
> > >
> > > > are we planning to add an alternative implementation
> > > against the new interfaces?
> > >
> > > Yes, we are simultaneously working on the YARN implementation using the
> > > interface. During the initial interface design, we encountered some
> > > anomalies while implementing it in YARN.
> > >
> > > Once the interfaces are finalized, we will proceed to raise a pull
> request
> > > (PR) for YARN as well.
> > >
> > > Our initial approach was to create a decoupled interface as part of
> > > FLIP-334 and then implement it for YARN in the subsequent phase.
> > > However, if you recommend combining both phases, we can certainly
> consider
> > > that option.
> > >
> > > We look forward to hearing your thoughts on whether to have YARN
> > > implementation as part of FLIP-334 or seperate one ?
> > >
> > > Bests
> > > Samrat
> > >
> > >
> > >
> > > On Tue, Sep 5, 2023 at 8:41 PM Maximilian Michels 
> wrote:
> > >
> > > > Thanks Rui for the update!
> > > >
> > > > Alongside with the refactoring to decouple autoscaler logic from the
> > > > deployment logic, are we planning to add an alternative
> implementation
> > > > against the new interfaces? I think the best way to get the
> interfaces
> > > > right, is to have an alternative implementation in addition to
> > > > Kubernetes. YARN or a standalone mode implementation were already
> > > > mentioned. Ultimately, this is the reason we are doing the
> > > > refactoring. Without a new implementa

[jira] [Created] (FLINK-33055) Correct the error value about 'state.backend.type' in the document

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-33055:


 Summary: Correct the error value about 'state.backend.type' in the 
document
 Key: FLINK-33055
 URL: https://issues.apache.org/jira/browse/FLINK-33055
 Project: Flink
  Issue Type: Bug
  Components: Documentation, Runtime / State Backends
Reporter: Hangxiang Yu


{{}}
{code:java}
state.backend.type: The state backend to use. This defines the data structure 
mechanism for taking snapshots. Common values are filesystem or rocksdb{code}
filesystem should be replaced with hashmap after FLINK-16444.

{{}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)