Re: [VOTE] FLIP-357: Deprecate Iteration API of DataStream

[Dong Lin]

Thanks Wencong for the FLIP.

+1 (binding)

On Thu, Sep 14, 2023 at 12:36 PM Wencong Liu  wrote:

> Hi dev,
>
>
> I'd like to start a vote on FLIP-357.
>
>
> Discussion thread:
> https://lists.apache.org/thread/shf77phc0wzlbj06jsfj3nclxnm2mrv5
> FLIP:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
>
>
> Best regards,
> Wencong Liu

[VOTE] FLIP-357: Deprecate Iteration API of DataStream

[Wencong Liu]

Hi dev, 


I'd like to start a vote on FLIP-357.


Discussion thread: 
https://lists.apache.org/thread/shf77phc0wzlbj06jsfj3nclxnm2mrv5
FLIP: 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream


Best regards,
Wencong Liu

Re:[DISCUSS] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

[Wencong Liu]

Dear Dong,

I have thoroughly reviewed the proposal for FLIP-331 and believe it would be 
a valuable addition to Flink. However, I do have a few questions that I would 
like to discuss:


1. The FLIP-331 proposed the EndOfStreamWindows that is implemented by 
TimeWindow with maxTimestamp = (Long.MAX_VALUE - 1), which naturally 
supports WindowedStream and AllWindowedStream to process all records 
belonging to a key in a 'global' window under both STREAMING and BATCH 
runtime execution mode. 


However, besides coGroup and keyBy().aggregate(), other operators on 
WindowedStream and AllWindowedStream, such as join/reduce, etc, currently 
are still implemented based on WindowOperator.


In fact, these operators can also be implemented without using WindowOperator 
to prevent additional WindowAssigner#assignWindows or triggerContext#onElement 
invocation cost. Will there be plans to support these operators in the future?


2. When using EndOfStreamWindows, upstream operators no longer support 
checkpointing. This limit may be too strict, especially when dealing with 
bounded data in streaming runtime execution mode, where checkpointing 
can still be useful.

3. The proposal mentions that if a transformation has isOutputOnEOF == true, 
the 
operator as well as its upstream operators will be executed in 'batch mode' 
with 
checkpointing disabled. I would like to understand the specific implications of 
this
'batch mode' and if there are any other changes associated with it? 

Additionally, I am curious to know if this 'batch mode' conflicts with the 'mix 
mode'

described in FLIP-327. While the coGroup and keyBy().aggregate() operators on 
EndOfStreamWindows have the attribute 'isInternalSorterSupported' set to true, 
indicating support for the 'mixed mode', they also have isOutputOnEOF set to 
true, 
which suggests that the upstream operators should be executed in 'batch mode'. 
Will the 'mixed mode' be ignored when in 'batch mode'? I would appreciate any 
clarification on this matter.

Thank you for taking the time to consider my feedback. I eagerly await your 
response.

Best regards,

Wencong Liu











At 2023-09-01 11:21:47, "Dong Lin"  wrote:
>Hi all,
>
>Jinhao (cc'ed) and I are opening this thread to discuss FLIP-331: Support
>EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task
>deployment. The design doc can be found at
>https://cwiki.apache.org/confluence/display/FLINK/FLIP-331%3A+Support+EndOfStreamWindows++and+isOutputOnEOF+operator+attribute+to+optimize+task+deployment
>.
>
>This FLIP introduces isOutputOnEOF operator attribute that JobManager can
>use to optimize task deployment and resource utilization. In addition, it
>also adds EndOfStreamWindows that can be used with the DataStream APIs
>(e.g. cogroup, aggregate) to significantly increase throughput and reduce
>resource utilization.
>
>We would greatly appreciate any comment or feedback you may have on this
>proposal.
>
>Cheers,
>Dong

Re: [VOTE] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

[Yang Wang]

+1 (binding)

Best,
Yang

Becket Qin  于2023年9月14日周四 11:01写道：

> +1 (binding)
>
> Thanks for the FLIP, Archit.
>
> Cheers,
>
> Jiangjie (Becket) Qin
>
>
> On Thu, Sep 14, 2023 at 10:31 AM Dong Lin  wrote:
>
> > Thanks Archit for the FLIP.
> >
> > +1 (binding)
> >
> > Regards,
> > Dong
> >
> > On Thu, Sep 14, 2023 at 1:47 AM Archit Goyal
>  > >
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for reviewing the FLIP-355 Add parent dir of files to classpath
> > > using yarn.provided.lib.dirs :
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+355%3A+Add+parent+dir+of+files+to+classpath+using+yarn.provided.lib.dirs
> > >
> > > Following is the discussion thread :
> > > https://lists.apache.org/thread/gv0ro4jsq4o206wg5gz9z5cww15qkvb9
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours (until September 15, 12:00AM GMT) unless there is an objection or
> > an
> > > insufficient number of votes.
> > >
> > > Thanks,
> > > Archit Goyal
> > >
> >
>

Re: Future of classical remoting in Pekko

[He Pin]

Thanks, The PR[1] is ready now, would you like to help review it, thanks.

1. https://github.com/apache/incubator-pekko/pull/643

On 2023/09/13 09:20:48 Ferenc Csaky wrote:
> The target release date for 1.18 is the end of Sept [1], but I'm not sure 
> everything will come together by then. Maybe it will pushed by a couple days.
> 
> I'm happy to help out, even making the Flink related changes when we're at 
> that point.
> 
> [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> 
> --- Original Message ---
> On Tuesday, September 12th, 2023 at 17:43, He Pin  wrote:
> 
> 
> > 
> > 
> > Hi Ferenc:
> > What's the ETA of the Flink 1.18? I think we should beable to collaborate 
> > on this，and at work we are using Flink too.
> > 
> > On 2023/09/12 15:16:11 Ferenc Csaky wrote:
> > 
> > > Hi Matthew,
> > > 
> > > Thanks for bringing this up! Cca half a year ago I started to work on an 
> > > Akka Artery migration, there is a draft PR for that 1. It might be an 
> > > option to revive that work and point it against Pekko instead. Although I 
> > > would highlight FLINK-29281 2 which will replace the whole RPC 
> > > implementation in Flink to a gRPC-based one when it is done.
> > > 
> > > I am not sure about the progess on the gRPC work, it looks hanging for a 
> > > while now, so I think if there is a chance to replace Netty3 with Netty4 
> > > in Pekko in the short term it would benefit Flink and then we can decide 
> > > if it would worth to upgrade to Artery, or how fast the gRPC solution can 
> > > be done and then it will not be necessary.
> > > 
> > > All in all, in the short term I think Flink would benefit to have that 
> > > mentioned PR 3 merged, then the updated Pekko version could be included 
> > > in the first 1.18 patch probably to mitigate those pesky Netty3 CVEs that 
> > > are carried for a while ASAP.
> > > 
> > > Cheers,
> > > Ferenc
> > > 
> > > 1 https://github.com/apache/flink/pull/22271
> > > 2 https://issues.apache.org/jira/browse/FLINK-29281
> > > 3 https://github.com/apache/incubator-pekko/pull/643
> > > 
> > > --- Original Message ---
> > > On Tuesday, September 12th, 2023 at 10:29, Matthew de Detrich 
> > > matthew.dedetr...@aiven.io.INVALID wrote:
> > > 
> > > > It's come to my attention that Flink is using Pekko's classical 
> > > > remoting,
> > > > if this is the case then I would recommend making a response at
> > > > https://lists.apache.org/thread/19h2wrs2om91g5vhnftv583fo0ddfshm .
> > > > 
> > > > Quick summary of what is being discussed is what to do with Pekko's
> > > > classical remoting. Classic remoting is considered deprecated since 
> > > > 2019,
> > > > an artifact that we inherited from Akka1. Ontop of this classical
> > > > remoting happens to be using netty3 which has known CVE's2, these CVE's
> > > > were never fixed in the netty3 series.
> > > > 
> > > > The question is what should be done given this, i.e. some people in the
> > > > Pekko community are wanting to drop classical remoting as quickly as
> > > > possible (i.e. even sooner then what semver allows but this is being
> > > > discussed) and others are wanting to leave it as it is (even with the
> > > > CVE's) since we don't want to incentivize and/or create impression that 
> > > > we
> > > > are officially supporting it. There is also a currently open PR3 which
> > > > upgrades Pekko's classical remoting's from netty3 to netty4 with the
> > > > primary motivator being removing said CVE's.
> > > > 
> > > > My personal position on the matter is that Pekko shouldn't drop 
> > > > classical
> > > > remoting until 2.0.x (to satisfy semver) while also updating Pekko's
> > > > classical remoting netty dependency to netty4 so that we are not 
> > > > shipping
> > > > Pekko with known CVE's (if this gets approved such a change would likely
> > > > land in Pekko 1.1.0). As is customary, such a decision should be agreed
> > > > upon broadly in the Pekko community.
> > > > 
> > > > Note that regardless of this change, it's recommended that a plan 
> > > > should be
> > > > made at some point by Flink to move from classical remoting to artery4
> > > > although the decision that Pekko ultimately makes may influence the
> > > > timeline (hence the reason for this thread).
> > > > 
> > > > --
> > > > 
> > > > Matthew de Detrich
> > > > 
> > > > Aiven Deutschland GmbH
> > > > 
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > 
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > 
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > 
> > > > m: +491603708037
> > > > 
> > > > w: aiven.io e: matthew.dedetr...@aiven.io
> 

Re: [VOTE] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

[Becket Qin]

+1 (binding)

Thanks for the FLIP, Archit.

Cheers,

Jiangjie (Becket) Qin


On Thu, Sep 14, 2023 at 10:31 AM Dong Lin  wrote:

> Thanks Archit for the FLIP.
>
> +1 (binding)
>
> Regards,
> Dong
>
> On Thu, Sep 14, 2023 at 1:47 AM Archit Goyal  >
> wrote:
>
> > Hi everyone,
> >
> > Thanks for reviewing the FLIP-355 Add parent dir of files to classpath
> > using yarn.provided.lib.dirs :
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+355%3A+Add+parent+dir+of+files+to+classpath+using+yarn.provided.lib.dirs
> >
> > Following is the discussion thread :
> > https://lists.apache.org/thread/gv0ro4jsq4o206wg5gz9z5cww15qkvb9
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> > hours (until September 15, 12:00AM GMT) unless there is an objection or
> an
> > insufficient number of votes.
> >
> > Thanks,
> > Archit Goyal
> >
>

Re: [VOTE] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

[Dong Lin]

Thanks Archit for the FLIP.

+1 (binding)

Regards,
Dong

On Thu, Sep 14, 2023 at 1:47 AM Archit Goyal 
wrote:

> Hi everyone,
>
> Thanks for reviewing the FLIP-355 Add parent dir of files to classpath
> using yarn.provided.lib.dirs :
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+355%3A+Add+parent+dir+of+files+to+classpath+using+yarn.provided.lib.dirs
>
> Following is the discussion thread :
> https://lists.apache.org/thread/gv0ro4jsq4o206wg5gz9z5cww15qkvb9
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until September 15, 12:00AM GMT) unless there is an objection or an
> insufficient number of votes.
>
> Thanks,
> Archit Goyal
>

Re: [VOTE] FLIP-361: Improve GC Metrics

[Venkatakrishnan Sowrirajan]

+1 (non-binding)

On Wed, Sep 13, 2023, 6:55 PM Matt Wang  wrote:

> +1 (non-binding)
>
>
> Thanks for driving this FLIP
>
>
>
>
> --
>
> Best,
> Matt Wang
>
>
>  Replied Message 
> | From | Xintong Song |
> | Date | 09/14/2023 09:54 |
> | To |  |
> | Subject | Re: [VOTE] FLIP-361: Improve GC Metrics |
> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Thu, Sep 14, 2023 at 2:40 AM Samrat Deb  wrote:
>
> +1 ( non binding)
>
> These improved GC metrics will be a great addition.
>
> Bests,
> Samrat
>
> On Wed, 13 Sep 2023 at 7:58 PM, ConradJam  wrote:
>
> +1 (non-binding)
> gc metrics help with autoscale tuning features
>
> Chen Zhanghao  于2023年9月13日周三 22:16写道：
>
> +1 (unbinding). Looking forward to it
>
> Best,
> Zhanghao Chen
> 
> 发件人: Gyula Fóra 
> 发送时间: 2023年9月13日 21:16
> 收件人: dev 
> 主题: [VOTE] FLIP-361: Improve GC Metrics
>
> Hi All!
>
> Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours unless there is an objection or insufficient votes.
>
> Cheers,
> Gyula
>
> [1]
>
>
>
>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-361*3A*Improve*GC*Metrics__;JSsrKw!!IKRxdwAv5BmarQ!dpHSOqsSHlPJ5gCvZ2yxSGjcR4xA2N-mpGZ1w2jPuKb78aujNpbzENmi1e7B26d6v4UQ8bQZ7IQaUcI$
> [2]
> https://urldefense.com/v3/__https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2__;!!IKRxdwAv5BmarQ!dpHSOqsSHlPJ5gCvZ2yxSGjcR4xA2N-mpGZ1w2jPuKb78aujNpbzENmi1e7B26d6v4UQ8bQZFdEMnAg$
>
>
>
> --
> Best
>
> ConradJam
>
>
>

Re: [VOTE] FLIP-361: Improve GC Metrics

[Dong Lin]

Thanks for the FLIP!

+1(binding)

On Wed, Sep 13, 2023 at 9:16 PM Gyula Fóra  wrote:

> Hi All!
>
> Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours unless there is an objection or insufficient votes.
>
> Cheers,
> Gyula
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
> [2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2
>

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[liu ron]

+1(non-binding)

Best,
Ron

Dong Lin  于2023年9月14日周四 09:01写道：

> Thank you Rui for the proposal.
>
> +1 (binding)
>
> On Wed, Sep 13, 2023 at 10:52 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Hi all,
> >
> > Thanks for all the feedback about the FLIP-334:
> > Decoupling autoscaler and kubernetes and
> > support the Standalone Autoscaler[1].
> > This FLIP was discussed in [2].
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> > hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> > insufficient votes.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> > [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
> >
> > Best,
> > Rui
> >
>

Re: [VOTE] FLIP-361: Improve GC Metrics

[Xintong Song]

+1 (binding)

Best,

Xintong



On Thu, Sep 14, 2023 at 2:40 AM Samrat Deb  wrote:

> +1 ( non binding)
>
> These improved GC metrics will be a great addition.
>
> Bests,
> Samrat
>
> On Wed, 13 Sep 2023 at 7:58 PM, ConradJam  wrote:
>
> > +1 (non-binding)
> > gc metrics help with autoscale tuning features
> >
> > Chen Zhanghao  于2023年9月13日周三 22:16写道：
> >
> > > +1 (unbinding). Looking forward to it
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > 发件人: Gyula Fóra 
> > > 发送时间: 2023年9月13日 21:16
> > > 收件人: dev 
> > > 主题: [VOTE] FLIP-361: Improve GC Metrics
> > >
> > > Hi All!
> > >
> > > Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours unless there is an objection or insufficient votes.
> > >
> > > Cheers,
> > > Gyula
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
> > > [2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>

Re: [VOTE] FLIP-361: Improve GC Metrics

[Matt Wang]

+1 (non-binding)


Thanks for driving this FLIP




--

Best,
Matt Wang


 Replied Message 
| From | Xintong Song |
| Date | 09/14/2023 09:54 |
| To |  |
| Subject | Re: [VOTE] FLIP-361: Improve GC Metrics |
+1 (binding)

Best,

Xintong



On Thu, Sep 14, 2023 at 2:40 AM Samrat Deb  wrote:

+1 ( non binding)

These improved GC metrics will be a great addition.

Bests,
Samrat

On Wed, 13 Sep 2023 at 7:58 PM, ConradJam  wrote:

+1 (non-binding)
gc metrics help with autoscale tuning features

Chen Zhanghao  于2023年9月13日周三 22:16写道：

+1 (unbinding). Looking forward to it

Best,
Zhanghao Chen

发件人: Gyula Fóra 
发送时间: 2023年9月13日 21:16
收件人: dev 
主题: [VOTE] FLIP-361: Improve GC Metrics

Hi All!

Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]

I'd like to start a vote for it. The vote will be open for at least 72
hours unless there is an objection or insufficient votes.

Cheers,
Gyula

[1]



https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
[2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2



--
Best

ConradJam

[jira] [Created] (FLINK-33086) Protect failure enrichment against unhandled exceptions

[Panagiotis Garefalakis (Jira)]

Panagiotis Garefalakis created FLINK-33086:
--

 Summary: Protect failure enrichment against unhandled exceptions
 Key: FLINK-33086
 URL: https://issues.apache.org/jira/browse/FLINK-33086
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Panagiotis Garefalakis






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Dong Lin]

Thank you Rui for the proposal.

+1 (binding)

On Wed, Sep 13, 2023 at 10:52 AM Rui Fan <1996fan...@gmail.com> wrote:

> Hi all,
>
> Thanks for all the feedback about the FLIP-334:
> Decoupling autoscaler and kubernetes and
> support the Standalone Autoscaler[1].
> This FLIP was discussed in [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
>
> Best,
> Rui
>

Re: [DISCUSS] Flink annotation strategy/consensus

[Becket Qin]

>
> Does it make sense to clearly define that APIs without annotation are
> internal APIs and should be used outside of Flink. And deprecate @Internal?


We can do this. Although I think it is OK to keep the @Internal annotation
in case extra clarity is needed sometimes.

Thanks,

Jiangjie (Becket) Qin

On Tue, Sep 12, 2023 at 7:11 PM Jing Ge  wrote:

> Hi Becket,
>
> Thanks for your reply with details.
>
>
> > 2. I agree it would be too verbose to annotate every internal method /
> > class / interface. Currently we already treat the methods / interfaces /
> > classes without annotations as effectively @Internal.
> >
>
> Does it make sense to clearly define that APIs without annotation are
> internal APIs and should be used outside of Flink. And deprecate @Internal?
>
> Best regards,
> Jing
>
> On Mon, Sep 11, 2023 at 5:05 AM Becket Qin  wrote:
>
> > Hi Jing,
> >
> > Thanks for bringing up the discussion. My two cents:
> >
> > 1. All the public methods / classes / interfaces MUST be annotated with
> one
> > of the @Experimental / @PublicEvolving / @Public. In practice, all the
> > methods by default inherit the annotation from the containing class,
> unless
> > annotated otherwise. e.g. an @Internal method in a @Public class.
>
>
> >
> 2. I agree it would be too verbose to annotate every internal method /
> > class / interface. Currently we already treat the methods / interfaces /
> > classes without annotations as effectively @Internal.
>
>
>
> 3. Per our discussion in the other thread, @Deprecated SHOULD coexist with
> > one of the @Experimental / @PublicEvolving / @Public. In that
> > case, @Deprecated overrides the other annotation, which means that public
> > API will not evolve and will be removed according to the deprecation
> > process.
> >
> > 4. The internal methods / classes / interfaces SHOULD NOT be marked as
> > deprecated. Instead, an immediate refactor should be done to remove the
> > "deprecated" internal methods / classes / interfaces, and migrate the
> code
> > to its successor. Otherwise, technical debts will build up.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Sat, Sep 9, 2023 at 5:29 AM Jing Ge 
> wrote:
> >
> > > Hi devs,
> > >
> > > While I was joining the flink-avro enhancement and cleanup discussion
> > > driven by Becket[1], I realized that there are some issues with the
> > current
> > > Flink API annotation usage in the source code.
> > >
> > > As far as I am concerned, Flink wants to control the access/visibility
> of
> > > APIs across modules and for downstreams. Since no OSGI is used(it
> should
> > > not be used because of its complexity, IMHO), Flink decided to use a
> very
> > > lightweight but manual solution: customized annotation like @Internal,
> > > @Experimental, @PublicEvolving,
> > > etc. This is a Flink only concept on top of JDK annotation and is
> > therefore
> > > orthogonal to @Deprecated or any other annotations offered by JDK.
> After
> > > this concept has been used, APIs without one of these annotations are
> in
> > > the kind of gray area which means they have no contract in the context
> of
> > > this new concept. Without any given metadata they could be considered
> > > as @Internal or @Experimental, because changes are allowed to be
> applied
> > at
> > > any time. But there is no clear definition and therefore different
> people
> > > will understand it differently.
> > >
> > > There are two options to improve it, as far as I could figure out:
> > >
> > > option 1: All APIs must have one of those annotations. We should put
> some
> > > effort into going through all source code and add missing annotations.
> > > There were discussions[2] and activities going in this direction.
> > > option 2: the community comes to a new consensus that APIs without
> > > annotation equals one of @Internal, @Experimental, or @PublicEvolving.
> I
> > > personally will choose @Internal, because it is the safest one. And if
> > > @Internal is chosen as the default one, it could also be deprecated,
> > > because no annotation equals @Internal. If it makes sense, I can
> create a
> > > FLIP and help the community reach this consensus.
> > >
> > > Both options have their own pros and cons. I would choose option 2,
> since
> > > we will not end up with a lot of APIs marked as @Internal.
> > >
> > > Looking forward to hearing your thoughts.
> > >
> > > Best regards
> > > Jing
> > >
> > >
> > > [1] https://lists.apache.org/thread/7zsv528swbjxo5zk0bxq33hrkvd77d6f
> > > [2] https://lists.apache.org/thread/zl2rmodsjsdb49tt4hn6wv3gdwo0m31o
> > >
> >
>

Re: Flink and Flink shaded dependency

[Jing Ge]

Hi Sergey,

Thanks for doing the analysis and providing the great insight. I did my own
analysis and got the same conclusion. I just wanted to use this example to
kick off a discussion and check if there is a common guideline or concept
in the community to handle such cases, since it seems any bump-up might
have a big impact. This time, we are kind of lucky. What if CVE related
code has been used in Flink? I do see it is an issue that upgrading a lib
in the flink-shaded repo is not recommended because of the complexity.
IMHO, we don't want to put the cart before the horse.

Best regards,
Jing

On Wed, Sep 13, 2023 at 9:11 PM Sergey Nuyanzin  wrote:

> Thanks for raising this
>
> I would suggest trying to double check whether it actually impacts Flink or
> not.
>
> For instance from one side Calcite between 1.22.0..1.31.0 has a critical
> CVE-2022-39135 [1]
> from another side Flink does not use this functionality and is not impacted
> by this.
>
> Regarding guava
> After closer look at Guava's high CVE you've mentioned [2]
> Based on Github issue describing the problem (exists since 2016) [3]
> there is a security issue with class
> com.google.common.io.FileBackedOutputStream
>
> While looking at source code for
> com.google.common.io.FileBackedOutputStream usage I was not able to find
> such.
> Also I was not able to find usage of
> org.apache.flink.shaded.guava30.com.google.common.io.Files#createTempDir
> which was fixed within commit [4]
> Also I was not able to find other Guava classes which use the problem one.
> Currently I tend to think that it probably does not impact Flink as well.
>
> Please correct me if I'm wrong
>
> [1] https://nvd.nist.gov/vuln/detail/CVE-2022-39135
> [2] https://nvd.nist.gov/vuln/detail/CVE-2023-2976
> [3] https://github.com/google/guava/issues/2575
> [4]
>
> https://github.com/google/guava/commit/b3719763b9ca777b94554d7ea2d2b92e27ac6164
>
> On Wed, Sep 13, 2023 at 6:26 PM Jing Ge 
> wrote:
>
> > Hi Sergey, Hi Martijn,
> >
> > Thanks for the information and suggestions. It is rational.
> >
> > Since on one hand, we should not do the upgrade according to all your
> > thoughts, and on the other hand, Guava 30.1.1-jre has known CVEs[1] and
> the
> > next closest version that has no CVEs is 32.0.1-jre. Without the upgrade,
> > users might not be able to use Flink 1.17 in their production and Flink
> > 1.18 has not been released yet. Do we have any known concept, rule, or
> > process in the community to handle such a common CVE issue?
> >
> > Best regards,
> > Jing
> >
> >
> >
> > [1] https://mvnrepository.com/artifact/com.google.guava/guava/30.1.1-jre
> >
> > On Wed, Sep 13, 2023 at 4:31 PM Martijn Visser  >
> > wrote:
> >
> > > Hi Jing,
> > >
> > > Like Sergey said, making that change would break all connectors that
> used
> > > Flink Shaded in the released versions, given that Guava is part of the
> > > package name. I think that was the case for Kafka, Pulsar, JDBC,
> PubSub,
> > > HBase, Cassandra and maybe even more. We shouldn't do this upgrade.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Wed, Sep 13, 2023 at 2:54 PM Sergey Nuyanzin 
> > > wrote:
> > >
> > > > Hi Jing,
> > > >
> > > > If I remember correctly, bumping of guava to another major version
> > > usually
> > > > leads to package rename
> > > > since the major version number is a part of the package name...
> > > > Not 100% sure however this could be the reason for some potential
> > > breaking
> > > > changes (we also faced that with connectors while the last
> flink-shaded
> > > > update)
> > > >
> > > > On Wed, Sep 13, 2023 at 2:43 PM Jing Ge 
> > > > wrote:
> > > >
> > > > > Hi Martijn,
> > > > >
> > > > > For example, bump up guava from 30.1.1-jre to 32.1.2-jre.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Wed, Sep 13, 2023 at 2:37 PM Martijn Visser <
> > > martijnvis...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jing,
> > > > > >
> > > > > > A PR to update the readme sounds like a good plan.
> > > > > >
> > > > > > I think it depends on what are the expected updates for a
> > > flink-shaded
> > > > > > 16.2, since that version doesn't exist.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > On Wed, Sep 13, 2023 at 1:48 PM Jing Ge
>  > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Martijn,
> > > > > > >
> > > > > > > Thanks for your reply with details. Appreciate it.
> > > > > > >
> > > > > > >
> > > > > > > > Flink-Shaded is usually only updated whenever a
> > > > > > > > new Flink minor version is released and only at the beginning
> > of
> > > > the
> > > > > > > > release cycle, so that there's enough time to stabilize
> Flink.
> > > > > > >
> > > > > > >
> > > > > > > This is the information I am looking for. It will help devs
> > > > understand
> > > > > > the
> > > > > > > compatibility between different versions of flink and
> > flink-shaded,
> > > > if
> > > > > it
> > > > > > > could be 

Re: Flink and Flink shaded dependency

[Sergey Nuyanzin]

Thanks for raising this

I would suggest trying to double check whether it actually impacts Flink or
not.

For instance from one side Calcite between 1.22.0..1.31.0 has a critical
CVE-2022-39135 [1]
from another side Flink does not use this functionality and is not impacted
by this.

Regarding guava
After closer look at Guava's high CVE you've mentioned [2]
Based on Github issue describing the problem (exists since 2016) [3]
there is a security issue with class
com.google.common.io.FileBackedOutputStream

While looking at source code for
com.google.common.io.FileBackedOutputStream usage I was not able to find
such.
Also I was not able to find usage of
org.apache.flink.shaded.guava30.com.google.common.io.Files#createTempDir
which was fixed within commit [4]
Also I was not able to find other Guava classes which use the problem one.
Currently I tend to think that it probably does not impact Flink as well.

Please correct me if I'm wrong

[1] https://nvd.nist.gov/vuln/detail/CVE-2022-39135
[2] https://nvd.nist.gov/vuln/detail/CVE-2023-2976
[3] https://github.com/google/guava/issues/2575
[4]
https://github.com/google/guava/commit/b3719763b9ca777b94554d7ea2d2b92e27ac6164

On Wed, Sep 13, 2023 at 6:26 PM Jing Ge  wrote:

> Hi Sergey, Hi Martijn,
>
> Thanks for the information and suggestions. It is rational.
>
> Since on one hand, we should not do the upgrade according to all your
> thoughts, and on the other hand, Guava 30.1.1-jre has known CVEs[1] and the
> next closest version that has no CVEs is 32.0.1-jre. Without the upgrade,
> users might not be able to use Flink 1.17 in their production and Flink
> 1.18 has not been released yet. Do we have any known concept, rule, or
> process in the community to handle such a common CVE issue?
>
> Best regards,
> Jing
>
>
>
> [1] https://mvnrepository.com/artifact/com.google.guava/guava/30.1.1-jre
>
> On Wed, Sep 13, 2023 at 4:31 PM Martijn Visser 
> wrote:
>
> > Hi Jing,
> >
> > Like Sergey said, making that change would break all connectors that used
> > Flink Shaded in the released versions, given that Guava is part of the
> > package name. I think that was the case for Kafka, Pulsar, JDBC, PubSub,
> > HBase, Cassandra and maybe even more. We shouldn't do this upgrade.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Wed, Sep 13, 2023 at 2:54 PM Sergey Nuyanzin 
> > wrote:
> >
> > > Hi Jing,
> > >
> > > If I remember correctly, bumping of guava to another major version
> > usually
> > > leads to package rename
> > > since the major version number is a part of the package name...
> > > Not 100% sure however this could be the reason for some potential
> > breaking
> > > changes (we also faced that with connectors while the last flink-shaded
> > > update)
> > >
> > > On Wed, Sep 13, 2023 at 2:43 PM Jing Ge 
> > > wrote:
> > >
> > > > Hi Martijn,
> > > >
> > > > For example, bump up guava from 30.1.1-jre to 32.1.2-jre.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Wed, Sep 13, 2023 at 2:37 PM Martijn Visser <
> > martijnvis...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi Jing,
> > > > >
> > > > > A PR to update the readme sounds like a good plan.
> > > > >
> > > > > I think it depends on what are the expected updates for a
> > flink-shaded
> > > > > 16.2, since that version doesn't exist.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Wed, Sep 13, 2023 at 1:48 PM Jing Ge  >
> > > > > wrote:
> > > > >
> > > > > > Hi Martijn,
> > > > > >
> > > > > > Thanks for your reply with details. Appreciate it.
> > > > > >
> > > > > >
> > > > > > > Flink-Shaded is usually only updated whenever a
> > > > > > > new Flink minor version is released and only at the beginning
> of
> > > the
> > > > > > > release cycle, so that there's enough time to stabilize Flink.
> > > > > >
> > > > > >
> > > > > > This is the information I am looking for. It will help devs
> > > understand
> > > > > the
> > > > > > compatibility between different versions of flink and
> flink-shaded,
> > > if
> > > > it
> > > > > > could be described in the readme. If you don't mind, I can
> create a
> > > pr
> > > > > and
> > > > > > update it.
> > > > > >
> > > > > > Speaking of this rule, I have a follow-up question: do you have
> any
> > > > > concern
> > > > > > if flink-shaded 16.2 will be released and upgraded(from 16.1 to
> > 16.2)
> > > > in
> > > > > > Flink 1.17? Is there anything we should pay attention to while
> > > > releasing
> > > > > a
> > > > > > new minor flink-shaded version?
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Wed, Sep 13, 2023 at 9:01 AM Martijn Visser <
> > > > martijnvis...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jing,
> > > > > > >
> > > > > > > Flink Shaded exists so that Flinks internal usage of commonly
> > used
> > > > > > packages
> > > > > > > such as Guava, Jackson inside of Flink don't clash with
> different
> > > > > > versions
> > > > > > > that 

Re: [VOTE] FLIP-361: Improve GC Metrics

[Samrat Deb]

+1 ( non binding)

These improved GC metrics will be a great addition.

Bests,
Samrat

On Wed, 13 Sep 2023 at 7:58 PM, ConradJam  wrote:

> +1 (non-binding)
> gc metrics help with autoscale tuning features
>
> Chen Zhanghao  于2023年9月13日周三 22:16写道：
>
> > +1 (unbinding). Looking forward to it
> >
> > Best,
> > Zhanghao Chen
> > 
> > 发件人: Gyula Fóra 
> > 发送时间: 2023年9月13日 21:16
> > 收件人: dev 
> > 主题: [VOTE] FLIP-361: Improve GC Metrics
> >
> > Hi All!
> >
> > Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> > hours unless there is an objection or insufficient votes.
> >
> > Cheers,
> > Gyula
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
> > [2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2
> >
>
>
> --
> Best
>
> ConradJam
>

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Gyula Fóra]

Hi Venkata,

Unfortunately the GC CPU Usage , percentage or otherwise, is not something
that the JVM GC beans expose. So I am afraid we cannot easily add that.

Regards
Gyula

On Wed, Sep 13, 2023 at 7:59 PM Venkatakrishnan Sowrirajan 
wrote:

> Hi Gyula,
>
> Thanks for driving this FLIP.
>
> The proposal looks good to me. Only one minor suggestion I have is, can we
> also include the % GC time spent wrt the overall CPU time especially useful
> in the cases of TM which helps in easily identifying issues related to GC.
> Thoughts?
>
> Regards
> Venkata krishnan
>
>
> On Wed, Sep 13, 2023 at 6:13 AM Gyula Fóra  wrote:
>
> > Thanks for all the feedback, I will start the vote on this.
> >
> > Gyula
> >
> > On Wed, Sep 6, 2023 at 11:11 AM Xintong Song 
> > wrote:
> >
> > > >
> > > > I added the average time metric to the FLIP document. I also included
> > it
> > > > for the aggregate (total) across all collectors. But maybe it doesn't
> > > make
> > > > too much sense as collection times usually differ greatly depending
> on
> > > the
> > > > collector.
> > > >
> > >
> > > LGTM
> > >
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Wed, Sep 6, 2023 at 4:31 PM Gyula Fóra 
> wrote:
> > >
> > > > I added the average time metric to the FLIP document. I also included
> > it
> > > > for the aggregate (total) across all collectors. But maybe it doesn't
> > > make
> > > > too much sense as collection times usually differ greatly depending
> on
> > > the
> > > > collector.
> > > >
> > > > Gyula
> > > >
> > > > On Wed, Sep 6, 2023 at 10:21 AM Xintong Song 
> > > > wrote:
> > > >
> > > > > Thank you :)
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Sep 6, 2023 at 4:17 PM Gyula Fóra 
> > > wrote:
> > > > >
> > > > > > Makes sense Xintong, I am happy to extend the proposal with the
> > > average
> > > > > gc
> > > > > > time metric +1
> > > > > >
> > > > > > Gyula
> > > > > >
> > > > > > On Wed, Sep 6, 2023 at 10:09 AM Xintong Song <
> > tonysong...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > >
> > > > > > > > Just so I understand correctly, do you suggest adding a
> metric
> > > for
> > > > > > > > delta(Time) / delta(Count) since the last reporting ?
> > > > > > > > .TimePerGc or .AverageTime would make
> > > sense.
> > > > > > > > AverageTime may be a bit nicer :)
> > > > > > > >
> > > > > > >
> > > > > > > Yes, that's what I mean.
> > > > > > >
> > > > > > > My only concern is how useful this will be in reality. If there
> > are
> > > > > only
> > > > > > > > (or several) long pauses then the msPerSec metrics will show
> it
> > > > > > already,
> > > > > > > > and if there is a single long pause that may not be shown at
> > all
> > > if
> > > > > > there
> > > > > > > > are several shorter pauses as well with this metric.
> > > > > > >
> > > > > > >
> > > > > > > Let's say we measure this for every minute and see a 900
> msPerSec
> > > > > (which
> > > > > > > means 54s within the minute are spent on GC). This may come
> from
> > a
> > > > > single
> > > > > > > GC that lasts for 54s, or 2 GCs each lasting for ~27s, or more
> > GCs
> > > > with
> > > > > > > less time each. As the default heartbeat timeout is 50s, the
> > former
> > > > > means
> > > > > > > there's likely a heartbeat timeout due to the GC pause, while
> the
> > > > > latter
> > > > > > > means otherwise.
> > > > > > >
> > > > > > >
> > > > > > > Mathematically, it is possible that there's 1 long pause
> together
> > > > with
> > > > > > > several short pauses within the same measurement period, making
> > the
> > > > > long
> > > > > > > pause not observable with AverageTime. However, from my
> > experience,
> > > > > such
> > > > > > a
> > > > > > > pattern is not normal in reality. My observation is that GCs
> > happen
> > > > at
> > > > > a
> > > > > > > similar time usually take a similar length of time. Admittedly,
> > > this
> > > > is
> > > > > > not
> > > > > > > a hard guarantee.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Xintong
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Sep 6, 2023 at 3:59 PM Gyula Fóra <
> gyula.f...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Matt Wang,
> > > > > > > >
> > > > > > > > I think the currently exposed info is all that is available
> > > through
> > > > > > > > GarbageCollectorMXBean. This FLIP does not aim to introduce a
> > new
> > > > > more
> > > > > > > > granular way of reporting the per collector metrics, that
> would
> > > > > > require a
> > > > > > > > new mechanism and may be a breaking change.
> > > > > > > >
> > > > > > > > We basically want to simply extend the current reporting here
> > > with
> > > > > the
> > > > > > > rate
> > > > > > > > metrics and the total metrics.
> > > > > > > >
> > > > > > > > Gyula
> > > > > > > >
> > > > > > > > On Wed, Sep 6, 2023 at 9:24 AM Matt Wang 
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi Gyula,
> > > > > 

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Venkatakrishnan Sowrirajan]

Hi Gyula,

Thanks for driving this FLIP.

The proposal looks good to me. Only one minor suggestion I have is, can we
also include the % GC time spent wrt the overall CPU time especially useful
in the cases of TM which helps in easily identifying issues related to GC.
Thoughts?

Regards
Venkata krishnan


On Wed, Sep 13, 2023 at 6:13 AM Gyula Fóra  wrote:

> Thanks for all the feedback, I will start the vote on this.
>
> Gyula
>
> On Wed, Sep 6, 2023 at 11:11 AM Xintong Song 
> wrote:
>
> > >
> > > I added the average time metric to the FLIP document. I also included
> it
> > > for the aggregate (total) across all collectors. But maybe it doesn't
> > make
> > > too much sense as collection times usually differ greatly depending on
> > the
> > > collector.
> > >
> >
> > LGTM
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Sep 6, 2023 at 4:31 PM Gyula Fóra  wrote:
> >
> > > I added the average time metric to the FLIP document. I also included
> it
> > > for the aggregate (total) across all collectors. But maybe it doesn't
> > make
> > > too much sense as collection times usually differ greatly depending on
> > the
> > > collector.
> > >
> > > Gyula
> > >
> > > On Wed, Sep 6, 2023 at 10:21 AM Xintong Song 
> > > wrote:
> > >
> > > > Thank you :)
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Wed, Sep 6, 2023 at 4:17 PM Gyula Fóra 
> > wrote:
> > > >
> > > > > Makes sense Xintong, I am happy to extend the proposal with the
> > average
> > > > gc
> > > > > time metric +1
> > > > >
> > > > > Gyula
> > > > >
> > > > > On Wed, Sep 6, 2023 at 10:09 AM Xintong Song <
> tonysong...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > >
> > > > > > > Just so I understand correctly, do you suggest adding a metric
> > for
> > > > > > > delta(Time) / delta(Count) since the last reporting ?
> > > > > > > .TimePerGc or .AverageTime would make
> > sense.
> > > > > > > AverageTime may be a bit nicer :)
> > > > > > >
> > > > > >
> > > > > > Yes, that's what I mean.
> > > > > >
> > > > > > My only concern is how useful this will be in reality. If there
> are
> > > > only
> > > > > > > (or several) long pauses then the msPerSec metrics will show it
> > > > > already,
> > > > > > > and if there is a single long pause that may not be shown at
> all
> > if
> > > > > there
> > > > > > > are several shorter pauses as well with this metric.
> > > > > >
> > > > > >
> > > > > > Let's say we measure this for every minute and see a 900 msPerSec
> > > > (which
> > > > > > means 54s within the minute are spent on GC). This may come from
> a
> > > > single
> > > > > > GC that lasts for 54s, or 2 GCs each lasting for ~27s, or more
> GCs
> > > with
> > > > > > less time each. As the default heartbeat timeout is 50s, the
> former
> > > > means
> > > > > > there's likely a heartbeat timeout due to the GC pause, while the
> > > > latter
> > > > > > means otherwise.
> > > > > >
> > > > > >
> > > > > > Mathematically, it is possible that there's 1 long pause together
> > > with
> > > > > > several short pauses within the same measurement period, making
> the
> > > > long
> > > > > > pause not observable with AverageTime. However, from my
> experience,
> > > > such
> > > > > a
> > > > > > pattern is not normal in reality. My observation is that GCs
> happen
> > > at
> > > > a
> > > > > > similar time usually take a similar length of time. Admittedly,
> > this
> > > is
> > > > > not
> > > > > > a hard guarantee.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Sep 6, 2023 at 3:59 PM Gyula Fóra 
> > > > wrote:
> > > > > >
> > > > > > > Matt Wang,
> > > > > > >
> > > > > > > I think the currently exposed info is all that is available
> > through
> > > > > > > GarbageCollectorMXBean. This FLIP does not aim to introduce a
> new
> > > > more
> > > > > > > granular way of reporting the per collector metrics, that would
> > > > > require a
> > > > > > > new mechanism and may be a breaking change.
> > > > > > >
> > > > > > > We basically want to simply extend the current reporting here
> > with
> > > > the
> > > > > > rate
> > > > > > > metrics and the total metrics.
> > > > > > >
> > > > > > > Gyula
> > > > > > >
> > > > > > > On Wed, Sep 6, 2023 at 9:24 AM Matt Wang 
> > wrote:
> > > > > > >
> > > > > > > > Hi Gyula,
> > > > > > > >
> > > > > > > > +1 for this proposal.
> > > > > > > >
> > > > > > > > Do we need to add a metric to record the count of different
> > > > > > > > collectors? Now there is only a total count. For example,
> > > > > > > > for G1, there is no way to distinguish whether it is the
> > > > > > > > young generation or the old generation.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Matt Wang
> > > > > > > >
> > > > > > > >
> > > > > > > >  Replied Message 
> > > > > > > > | From | Gyula Fóra |
> > > > > > > > | Date | 09/6/2023 

Re: [VOTE] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

[Venkatakrishnan Sowrirajan]

Thanks for driving this FLIP, Archit. +1 (non-binding)

Regards
Venkata krishnan


On Wed, Sep 13, 2023 at 10:47 AM Archit Goyal 
wrote:

> Hi everyone,
>
> Thanks for reviewing the FLIP-355 Add parent dir of files to classpath
> using yarn.provided.lib.dirs :
>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP*355*3A*Add*parent*dir*of*files*to*classpath*using*yarn.provided.lib.dirs__;KyUrKysrKysrKys!!IKRxdwAv5BmarQ!YSiXPEWg6eTjX8YpOOMp_5Qt56yawUg5mjBsdMS4XgrzA9st0aIeaYNZF9DTuNHI-ThSZE7F49ECm52VIR10wMur$
>
> Following is the discussion thread :
>
> https://urldefense.com/v3/__https://lists.apache.org/thread/gv0ro4jsq4o206wg5gz9z5cww15qkvb9__;!!IKRxdwAv5BmarQ!YSiXPEWg6eTjX8YpOOMp_5Qt56yawUg5mjBsdMS4XgrzA9st0aIeaYNZF9DTuNHI-ThSZE7F49ECm52VIZr7rK1T$
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until September 15, 12:00AM GMT) unless there is an objection or an
> insufficient number of votes.
>
> Thanks,
> Archit Goyal
>

[VOTE] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

[Archit Goyal]

Hi everyone,

Thanks for reviewing the FLIP-355 Add parent dir of files to classpath using 
yarn.provided.lib.dirs :
https://cwiki.apache.org/confluence/display/FLINK/FLIP+355%3A+Add+parent+dir+of+files+to+classpath+using+yarn.provided.lib.dirs

Following is the discussion thread :
https://lists.apache.org/thread/gv0ro4jsq4o206wg5gz9z5cww15qkvb9

I'd like to start a vote for it. The vote will be open for at least 72 hours 
(until September 15, 12:00AM GMT) unless there is an objection or an 
insufficient number of votes.

Thanks,
Archit Goyal

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Samrat Deb]

+1 (non binding )

Best,
Samrat

On Wed, 13 Sep 2023 at 7:08 PM, Feng Jin  wrote:

> Thanks for driving this, looking forward to this feature.
>
>
> +1 (non-binding)
>
> Best,
> Feng
>
> On Wed, Sep 13, 2023 at 9:11 PM Chen Zhanghao 
> wrote:
>
> > Thanks for driving this. +1 (non-binding)
> >
> > Best,
> > Zhanghao Chen
> > 
> > 发件人: Rui Fan <1996fan...@gmail.com>
> > 发送时间: 2023年9月13日 10:52
> > 收件人: dev 
> > 主题: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the
> > Standalone Autoscaler
> >
> > Hi all,
> >
> > Thanks for all the feedback about the FLIP-334:
> > Decoupling autoscaler and kubernetes and
> > support the Standalone Autoscaler[1].
> > This FLIP was discussed in [2].
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> > hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> > insufficient votes.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> > [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
> >
> > Best,
> > Rui
> >
>

Re: Flink and Flink shaded dependency

[Jing Ge]

Hi Sergey, Hi Martijn,

Thanks for the information and suggestions. It is rational.

Since on one hand, we should not do the upgrade according to all your
thoughts, and on the other hand, Guava 30.1.1-jre has known CVEs[1] and the
next closest version that has no CVEs is 32.0.1-jre. Without the upgrade,
users might not be able to use Flink 1.17 in their production and Flink
1.18 has not been released yet. Do we have any known concept, rule, or
process in the community to handle such a common CVE issue?

Best regards,
Jing



[1] https://mvnrepository.com/artifact/com.google.guava/guava/30.1.1-jre

On Wed, Sep 13, 2023 at 4:31 PM Martijn Visser 
wrote:

> Hi Jing,
>
> Like Sergey said, making that change would break all connectors that used
> Flink Shaded in the released versions, given that Guava is part of the
> package name. I think that was the case for Kafka, Pulsar, JDBC, PubSub,
> HBase, Cassandra and maybe even more. We shouldn't do this upgrade.
>
> Best regards,
>
> Martijn
>
> On Wed, Sep 13, 2023 at 2:54 PM Sergey Nuyanzin 
> wrote:
>
> > Hi Jing,
> >
> > If I remember correctly, bumping of guava to another major version
> usually
> > leads to package rename
> > since the major version number is a part of the package name...
> > Not 100% sure however this could be the reason for some potential
> breaking
> > changes (we also faced that with connectors while the last flink-shaded
> > update)
> >
> > On Wed, Sep 13, 2023 at 2:43 PM Jing Ge 
> > wrote:
> >
> > > Hi Martijn,
> > >
> > > For example, bump up guava from 30.1.1-jre to 32.1.2-jre.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Wed, Sep 13, 2023 at 2:37 PM Martijn Visser <
> martijnvis...@apache.org
> > >
> > > wrote:
> > >
> > > > Hi Jing,
> > > >
> > > > A PR to update the readme sounds like a good plan.
> > > >
> > > > I think it depends on what are the expected updates for a
> flink-shaded
> > > > 16.2, since that version doesn't exist.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > On Wed, Sep 13, 2023 at 1:48 PM Jing Ge 
> > > > wrote:
> > > >
> > > > > Hi Martijn,
> > > > >
> > > > > Thanks for your reply with details. Appreciate it.
> > > > >
> > > > >
> > > > > > Flink-Shaded is usually only updated whenever a
> > > > > > new Flink minor version is released and only at the beginning of
> > the
> > > > > > release cycle, so that there's enough time to stabilize Flink.
> > > > >
> > > > >
> > > > > This is the information I am looking for. It will help devs
> > understand
> > > > the
> > > > > compatibility between different versions of flink and flink-shaded,
> > if
> > > it
> > > > > could be described in the readme. If you don't mind, I can create a
> > pr
> > > > and
> > > > > update it.
> > > > >
> > > > > Speaking of this rule, I have a follow-up question: do you have any
> > > > concern
> > > > > if flink-shaded 16.2 will be released and upgraded(from 16.1 to
> 16.2)
> > > in
> > > > > Flink 1.17? Is there anything we should pay attention to while
> > > releasing
> > > > a
> > > > > new minor flink-shaded version?
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Wed, Sep 13, 2023 at 9:01 AM Martijn Visser <
> > > martijnvis...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jing,
> > > > > >
> > > > > > Flink Shaded exists so that Flinks internal usage of commonly
> used
> > > > > packages
> > > > > > such as Guava, Jackson inside of Flink don't clash with different
> > > > > versions
> > > > > > that users might use when creating a Flink application. When I
> did
> > > the
> > > > > > upgrade of Flink Shaded, we already ran into a bunch of problems
> > > > because
> > > > > a
> > > > > > lot of the externalized connectors relied on Flink Shaded, which
> > made
> > > > it
> > > > > > problematic to get the connector to work on both Flink 1.17 and
> > Flink
> > > > > 1.18.
> > > > > > There's been quite a lot of effort put into making sure that
> > > > externalized
> > > > > > connectors don't rely on Flink Shaded at all anymore, by either
> > using
> > > > > their
> > > > > > own versions of shaded artifacts (which was the case with the
> > Pulsar
> > > > > > connector) or just removing the dependency on Flink Shaded all
> > > > together,
> > > > > by
> > > > > > using regular Java.
> > > > > >
> > > > > > If you would upgrade flink-shaded from 16.1 to 17.0 in Flink
> 1.17,
> > > you
> > > > > > would break all externalized connectors that rely on Flink
> Shaded's
> > > > Guava
> > > > > > version, plus you potentially would impact the runtime given that
> > > > > there's a
> > > > > > newer Netty version etc. Flink-Shaded is usually only updated
> > > whenever
> > > > a
> > > > >
> > > > > new Flink minor version is released and only at the beginning of
> the
> > > > > > release cycle, so that there's enough time to stabilize Flink.
> > > > >
> > > > > All in all, we shouldn't upgrade flink-shaded in Flink 1.17.
> > > > > >
> > > > > > Best regards,
> > > > 

Re: Flink and Flink shaded dependency

[Martijn Visser]

Hi Jing,

Like Sergey said, making that change would break all connectors that used
Flink Shaded in the released versions, given that Guava is part of the
package name. I think that was the case for Kafka, Pulsar, JDBC, PubSub,
HBase, Cassandra and maybe even more. We shouldn't do this upgrade.

Best regards,

Martijn

On Wed, Sep 13, 2023 at 2:54 PM Sergey Nuyanzin  wrote:

> Hi Jing,
>
> If I remember correctly, bumping of guava to another major version usually
> leads to package rename
> since the major version number is a part of the package name...
> Not 100% sure however this could be the reason for some potential breaking
> changes (we also faced that with connectors while the last flink-shaded
> update)
>
> On Wed, Sep 13, 2023 at 2:43 PM Jing Ge 
> wrote:
>
> > Hi Martijn,
> >
> > For example, bump up guava from 30.1.1-jre to 32.1.2-jre.
> >
> > Best regards,
> > Jing
> >
> > On Wed, Sep 13, 2023 at 2:37 PM Martijn Visser  >
> > wrote:
> >
> > > Hi Jing,
> > >
> > > A PR to update the readme sounds like a good plan.
> > >
> > > I think it depends on what are the expected updates for a flink-shaded
> > > 16.2, since that version doesn't exist.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Wed, Sep 13, 2023 at 1:48 PM Jing Ge 
> > > wrote:
> > >
> > > > Hi Martijn,
> > > >
> > > > Thanks for your reply with details. Appreciate it.
> > > >
> > > >
> > > > > Flink-Shaded is usually only updated whenever a
> > > > > new Flink minor version is released and only at the beginning of
> the
> > > > > release cycle, so that there's enough time to stabilize Flink.
> > > >
> > > >
> > > > This is the information I am looking for. It will help devs
> understand
> > > the
> > > > compatibility between different versions of flink and flink-shaded,
> if
> > it
> > > > could be described in the readme. If you don't mind, I can create a
> pr
> > > and
> > > > update it.
> > > >
> > > > Speaking of this rule, I have a follow-up question: do you have any
> > > concern
> > > > if flink-shaded 16.2 will be released and upgraded(from 16.1 to 16.2)
> > in
> > > > Flink 1.17? Is there anything we should pay attention to while
> > releasing
> > > a
> > > > new minor flink-shaded version?
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Wed, Sep 13, 2023 at 9:01 AM Martijn Visser <
> > martijnvis...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi Jing,
> > > > >
> > > > > Flink Shaded exists so that Flinks internal usage of commonly used
> > > > packages
> > > > > such as Guava, Jackson inside of Flink don't clash with different
> > > > versions
> > > > > that users might use when creating a Flink application. When I did
> > the
> > > > > upgrade of Flink Shaded, we already ran into a bunch of problems
> > > because
> > > > a
> > > > > lot of the externalized connectors relied on Flink Shaded, which
> made
> > > it
> > > > > problematic to get the connector to work on both Flink 1.17 and
> Flink
> > > > 1.18.
> > > > > There's been quite a lot of effort put into making sure that
> > > externalized
> > > > > connectors don't rely on Flink Shaded at all anymore, by either
> using
> > > > their
> > > > > own versions of shaded artifacts (which was the case with the
> Pulsar
> > > > > connector) or just removing the dependency on Flink Shaded all
> > > together,
> > > > by
> > > > > using regular Java.
> > > > >
> > > > > If you would upgrade flink-shaded from 16.1 to 17.0 in Flink 1.17,
> > you
> > > > > would break all externalized connectors that rely on Flink Shaded's
> > > Guava
> > > > > version, plus you potentially would impact the runtime given that
> > > > there's a
> > > > > newer Netty version etc. Flink-Shaded is usually only updated
> > whenever
> > > a
> > > >
> > > > new Flink minor version is released and only at the beginning of the
> > > > > release cycle, so that there's enough time to stabilize Flink.
> > > >
> > > > All in all, we shouldn't upgrade flink-shaded in Flink 1.17.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Tue, Sep 12, 2023 at 7:26 PM Jing Ge  >
> > > > > wrote:
> > > > >
> > > > > > Hi Dev,
> > > > > >
> > > > > > Currently Flink 1.17 is using flink-shaded 16.1 and Flink 1.18 is
> > > using
> > > > > > flink-shaded 17.0. Do we need to consider any compatibility rules
> > > > between
> > > > > > them? E.g. is there any concern to upgrade flink-shaded from 16.1
> > to
> > > > 17.x
> > > > > > for Flink 1.17? Or there are some implicit dependency rules
> between
> > > > > > them. Looking
> > > > > > forward to hearing from you.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: [VOTE] FLIP-361: Improve GC Metrics

[ConradJam]

+1 (non-binding)
gc metrics help with autoscale tuning features

Chen Zhanghao  于2023年9月13日周三 22:16写道：

> +1 (unbinding). Looking forward to it
>
> Best,
> Zhanghao Chen
> 
> 发件人: Gyula Fóra 
> 发送时间: 2023年9月13日 21:16
> 收件人: dev 
> 主题: [VOTE] FLIP-361: Improve GC Metrics
>
> Hi All!
>
> Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours unless there is an objection or insufficient votes.
>
> Cheers,
> Gyula
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
> [2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2
>


-- 
Best

ConradJam

回复: [VOTE] FLIP-361: Improve GC Metrics

[Chen Zhanghao]

+1 (unbinding). Looking forward to it

Best,
Zhanghao Chen

发件人: Gyula Fóra 
发送时间: 2023年9月13日 21:16
收件人: dev 
主题: [VOTE] FLIP-361: Improve GC Metrics

Hi All!

Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]

I'd like to start a vote for it. The vote will be open for at least 72
hours unless there is an objection or insufficient votes.

Cheers,
Gyula

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
[2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2

Re: [VOTE] FLIP-361: Improve GC Metrics

[Ahmed Hamdy]

+1 (non-binding)
Best Regards
Ahmed Hamdy


On Wed, 13 Sept 2023 at 14:22, Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui
>
> On Wed, Sep 13, 2023 at 9:16 PM Gyula Fóra  wrote:
>
> > Hi All!
> >
> > Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> > hours unless there is an objection or insufficient votes.
> >
> > Cheers,
> > Gyula
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
> > [2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2
> >
>

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Feng Jin]

Thanks for driving this, looking forward to this feature.


+1 (non-binding)

Best,
Feng

On Wed, Sep 13, 2023 at 9:11 PM Chen Zhanghao 
wrote:

> Thanks for driving this. +1 (non-binding)
>
> Best,
> Zhanghao Chen
> 
> 发件人: Rui Fan <1996fan...@gmail.com>
> 发送时间: 2023年9月13日 10:52
> 收件人: dev 
> 主题: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the
> Standalone Autoscaler
>
> Hi all,
>
> Thanks for all the feedback about the FLIP-334:
> Decoupling autoscaler and kubernetes and
> support the Standalone Autoscaler[1].
> This FLIP was discussed in [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
>
> Best,
> Rui
>

Re: [VOTE] FLIP-361: Improve GC Metrics

[Rui Fan]

+1(binding)

Best,
Rui

On Wed, Sep 13, 2023 at 9:16 PM Gyula Fóra  wrote:

> Hi All!
>
> Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours unless there is an objection or insufficient votes.
>
> Cheers,
> Gyula
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
> [2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2
>

[VOTE] FLIP-361: Improve GC Metrics

[Gyula Fóra]

Hi All!

Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]

I'd like to start a vote for it. The vote will be open for at least 72
hours unless there is an objection or insufficient votes.

Cheers,
Gyula

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-361%3A+Improve+GC+Metrics
[2] https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2

Re: [DISCUSS] FLIP-361: Improve GC Metrics

[Gyula Fóra]

Thanks for all the feedback, I will start the vote on this.

Gyula

On Wed, Sep 6, 2023 at 11:11 AM Xintong Song  wrote:

> >
> > I added the average time metric to the FLIP document. I also included it
> > for the aggregate (total) across all collectors. But maybe it doesn't
> make
> > too much sense as collection times usually differ greatly depending on
> the
> > collector.
> >
>
> LGTM
>
>
> Best,
>
> Xintong
>
>
>
> On Wed, Sep 6, 2023 at 4:31 PM Gyula Fóra  wrote:
>
> > I added the average time metric to the FLIP document. I also included it
> > for the aggregate (total) across all collectors. But maybe it doesn't
> make
> > too much sense as collection times usually differ greatly depending on
> the
> > collector.
> >
> > Gyula
> >
> > On Wed, Sep 6, 2023 at 10:21 AM Xintong Song 
> > wrote:
> >
> > > Thank you :)
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Wed, Sep 6, 2023 at 4:17 PM Gyula Fóra 
> wrote:
> > >
> > > > Makes sense Xintong, I am happy to extend the proposal with the
> average
> > > gc
> > > > time metric +1
> > > >
> > > > Gyula
> > > >
> > > > On Wed, Sep 6, 2023 at 10:09 AM Xintong Song 
> > > > wrote:
> > > >
> > > > > >
> > > > > > Just so I understand correctly, do you suggest adding a metric
> for
> > > > > > delta(Time) / delta(Count) since the last reporting ?
> > > > > > .TimePerGc or .AverageTime would make
> sense.
> > > > > > AverageTime may be a bit nicer :)
> > > > > >
> > > > >
> > > > > Yes, that's what I mean.
> > > > >
> > > > > My only concern is how useful this will be in reality. If there are
> > > only
> > > > > > (or several) long pauses then the msPerSec metrics will show it
> > > > already,
> > > > > > and if there is a single long pause that may not be shown at all
> if
> > > > there
> > > > > > are several shorter pauses as well with this metric.
> > > > >
> > > > >
> > > > > Let's say we measure this for every minute and see a 900 msPerSec
> > > (which
> > > > > means 54s within the minute are spent on GC). This may come from a
> > > single
> > > > > GC that lasts for 54s, or 2 GCs each lasting for ~27s, or more GCs
> > with
> > > > > less time each. As the default heartbeat timeout is 50s, the former
> > > means
> > > > > there's likely a heartbeat timeout due to the GC pause, while the
> > > latter
> > > > > means otherwise.
> > > > >
> > > > >
> > > > > Mathematically, it is possible that there's 1 long pause together
> > with
> > > > > several short pauses within the same measurement period, making the
> > > long
> > > > > pause not observable with AverageTime. However, from my experience,
> > > such
> > > > a
> > > > > pattern is not normal in reality. My observation is that GCs happen
> > at
> > > a
> > > > > similar time usually take a similar length of time. Admittedly,
> this
> > is
> > > > not
> > > > > a hard guarantee.
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Sep 6, 2023 at 3:59 PM Gyula Fóra 
> > > wrote:
> > > > >
> > > > > > Matt Wang,
> > > > > >
> > > > > > I think the currently exposed info is all that is available
> through
> > > > > > GarbageCollectorMXBean. This FLIP does not aim to introduce a new
> > > more
> > > > > > granular way of reporting the per collector metrics, that would
> > > > require a
> > > > > > new mechanism and may be a breaking change.
> > > > > >
> > > > > > We basically want to simply extend the current reporting here
> with
> > > the
> > > > > rate
> > > > > > metrics and the total metrics.
> > > > > >
> > > > > > Gyula
> > > > > >
> > > > > > On Wed, Sep 6, 2023 at 9:24 AM Matt Wang 
> wrote:
> > > > > >
> > > > > > > Hi Gyula,
> > > > > > >
> > > > > > > +1 for this proposal.
> > > > > > >
> > > > > > > Do we need to add a metric to record the count of different
> > > > > > > collectors? Now there is only a total count. For example,
> > > > > > > for G1, there is no way to distinguish whether it is the
> > > > > > > young generation or the old generation.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best,
> > > > > > > Matt Wang
> > > > > > >
> > > > > > >
> > > > > > >  Replied Message 
> > > > > > > | From | Gyula Fóra |
> > > > > > > | Date | 09/6/2023 15:03 |
> > > > > > > | To |  |
> > > > > > > | Subject | Re: [DISCUSS] FLIP-361: Improve GC Metrics |
> > > > > > > Thanks Xintong!
> > > > > > >
> > > > > > > Just so I understand correctly, do you suggest adding a metric
> > for
> > > > > > > delta(Time) / delta(Count) since the last reporting ?
> > > > > > > .TimePerGc or .AverageTime would make
> > sense.
> > > > > > > AverageTime may be a bit nicer :)
> > > > > > >
> > > > > > > My only concern is how useful this will be in reality. If there
> > are
> > > > > only
> > > > > > > (or several) long pauses then the msPerSec metrics will show it
> > > > > already,
> > > > > > > and if there is a single long pause that may not be shown at
> all
> > if
> > > > > there
> 

回复: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Chen Zhanghao]

Thanks for driving this. +1 (non-binding)

Best,
Zhanghao Chen

发件人: Rui Fan <1996fan...@gmail.com>
发送时间: 2023年9月13日 10:52
收件人: dev 
主题: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the 
Standalone Autoscaler

Hi all,

Thanks for all the feedback about the FLIP-334:
Decoupling autoscaler and kubernetes and
support the Standalone Autoscaler[1].
This FLIP was discussed in [2].

I'd like to start a vote for it. The vote will be open for at least 72
hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
insufficient votes.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
[2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495

Best,
Rui

Re: Flink and Flink shaded dependency

[Sergey Nuyanzin]

Hi Jing,

If I remember correctly, bumping of guava to another major version usually
leads to package rename
since the major version number is a part of the package name...
Not 100% sure however this could be the reason for some potential breaking
changes (we also faced that with connectors while the last flink-shaded
update)

On Wed, Sep 13, 2023 at 2:43 PM Jing Ge  wrote:

> Hi Martijn,
>
> For example, bump up guava from 30.1.1-jre to 32.1.2-jre.
>
> Best regards,
> Jing
>
> On Wed, Sep 13, 2023 at 2:37 PM Martijn Visser 
> wrote:
>
> > Hi Jing,
> >
> > A PR to update the readme sounds like a good plan.
> >
> > I think it depends on what are the expected updates for a flink-shaded
> > 16.2, since that version doesn't exist.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Wed, Sep 13, 2023 at 1:48 PM Jing Ge 
> > wrote:
> >
> > > Hi Martijn,
> > >
> > > Thanks for your reply with details. Appreciate it.
> > >
> > >
> > > > Flink-Shaded is usually only updated whenever a
> > > > new Flink minor version is released and only at the beginning of the
> > > > release cycle, so that there's enough time to stabilize Flink.
> > >
> > >
> > > This is the information I am looking for. It will help devs understand
> > the
> > > compatibility between different versions of flink and flink-shaded, if
> it
> > > could be described in the readme. If you don't mind, I can create a pr
> > and
> > > update it.
> > >
> > > Speaking of this rule, I have a follow-up question: do you have any
> > concern
> > > if flink-shaded 16.2 will be released and upgraded(from 16.1 to 16.2)
> in
> > > Flink 1.17? Is there anything we should pay attention to while
> releasing
> > a
> > > new minor flink-shaded version?
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Wed, Sep 13, 2023 at 9:01 AM Martijn Visser <
> martijnvis...@apache.org
> > >
> > > wrote:
> > >
> > > > Hi Jing,
> > > >
> > > > Flink Shaded exists so that Flinks internal usage of commonly used
> > > packages
> > > > such as Guava, Jackson inside of Flink don't clash with different
> > > versions
> > > > that users might use when creating a Flink application. When I did
> the
> > > > upgrade of Flink Shaded, we already ran into a bunch of problems
> > because
> > > a
> > > > lot of the externalized connectors relied on Flink Shaded, which made
> > it
> > > > problematic to get the connector to work on both Flink 1.17 and Flink
> > > 1.18.
> > > > There's been quite a lot of effort put into making sure that
> > externalized
> > > > connectors don't rely on Flink Shaded at all anymore, by either using
> > > their
> > > > own versions of shaded artifacts (which was the case with the Pulsar
> > > > connector) or just removing the dependency on Flink Shaded all
> > together,
> > > by
> > > > using regular Java.
> > > >
> > > > If you would upgrade flink-shaded from 16.1 to 17.0 in Flink 1.17,
> you
> > > > would break all externalized connectors that rely on Flink Shaded's
> > Guava
> > > > version, plus you potentially would impact the runtime given that
> > > there's a
> > > > newer Netty version etc. Flink-Shaded is usually only updated
> whenever
> > a
> > >
> > > new Flink minor version is released and only at the beginning of the
> > > > release cycle, so that there's enough time to stabilize Flink.
> > >
> > > All in all, we shouldn't upgrade flink-shaded in Flink 1.17.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > On Tue, Sep 12, 2023 at 7:26 PM Jing Ge 
> > > > wrote:
> > > >
> > > > > Hi Dev,
> > > > >
> > > > > Currently Flink 1.17 is using flink-shaded 16.1 and Flink 1.18 is
> > using
> > > > > flink-shaded 17.0. Do we need to consider any compatibility rules
> > > between
> > > > > them? E.g. is there any concern to upgrade flink-shaded from 16.1
> to
> > > 17.x
> > > > > for Flink 1.17? Or there are some implicit dependency rules between
> > > > > them. Looking
> > > > > forward to hearing from you.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > >
> > >
> >
>


-- 
Best regards,
Sergey

回复: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

[Chen Zhanghao]

Hi Jing,

Thanks for the suggestion. Endpoint is indeed a more professional word in the 
networking world but I think location is more suited here for two reasons:

  1.  The term here is for uniquely identifying the TaskManager where the task 
is deployed while providing the host machine info as well to help identify 
taskmanager- and host-aggregative problems. So strictly speaking, it is not 
used in a pure networking context.
  2.  The term "location" is already used widely in the codebase, e.g. 
TaskManagerLocation and JobExceptions-related classes.

WDYT?

Best,
Zhanghao Chen

发件人: Jing Ge 
发送时间: 2023年9月13日 4:52
收件人: dev@flink.apache.org 
主题: Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in 
REST API and Web UI

Hi Zhanghao,

Thanks for bringing this to our attention. It is a good proposal to improve
data consistency.

Speaking of naming conventions of choosing location over host, how about
"endpoint" with the following thoughts:

1. endpoint is a more professional word than location in the network
context.
2. I know commonly endpoints mean the URLs of services. Using Hostname:port
as the endpoint follows exactly the same rule, because TaskManager is the
top level service that aligns with the top level endpoint.

WDYT?

Best regards,
Jing


On Mon, Sep 11, 2023 at 6:01 AM Weihua Hu  wrote:

> Hi, Zhanghao
>
> Since the meaning of "host" is not aligned, it seems good for me to remove
> it in the future release.
>
> Best,
> Weihua
>
>
> On Mon, Sep 11, 2023 at 11:48 AM Chen Zhanghao 
> wrote:
>
> > Hi Fan Rui,
> >
> > Thanks for clarifying the definition of "public interfaces", that helps a
> > lot!
> >
> > Best,
> > Zhanghao Chen
> > 
> > 发件人: Rui Fan <1996fan...@gmail.com>
> > 发送时间: 2023年9月11日 11:18
> > 收件人: dev@flink.apache.org 
> > 主题: Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager
> > Location in REST API and Web UI
> >
> > Thanks Zhanghao driving this FLIP, adding the port in Web UI
> > seems good to me.
> >
> > Hi Shammon and Zhanghao,
> >
> > I would like to clarify the difference between Public Interfaces
> > in FLIP and @Public in code.
> >
> > As I understand, the `Public Interfaces in FLIP` means these
> > changes will be used in user side, such as: @Public class,
> > Configuration settings, User-facing scripts/command-line tools,
> > and rest api, etc.
> >
> > You can refer to  "What are the "public interfaces" of the project?"
> > part in Flink Improvement Proposals doc[1].
> >
> > @Public class means the user will use this class directly, and
> > these rest classes won't be depended on directly. So I think
> > these classes related to rest don't need to be marked @Public.
> >
> > Please correct me if anything is wrong, thanks~
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >
> > Best,
> > Rui
> >
> > On Mon, Sep 11, 2023 at 11:09 AM Weihua Hu 
> wrote:
> >
> > > Hi, Zhanghao
> > >
> > > Thanks for bringing this proposal.
> > >
> > > I have a concern:
> > >
> > > I prefer to keep the "host" field and add a "location" field in future
> > > versions.
> > > Consider a scenario where a machine (host) with multiple TaskManagers
> has
> > > poor processing performance due to some problems.
> > > By using a host field aggregation, I can identify the problems with
> this
> > > machine and take it offline.
> > >
> > > Best,
> > > Weihua
> > >
> > >
> > > On Mon, Sep 11, 2023 at 10:34 AM Chen Zhanghao <
> > zhanghao.c...@outlook.com>
> > > wrote:
> > >
> > > > Hi Shammon,
> > > >
> > > > I think all REST API response messages (e.g.
> > > > SubtaskExecutionAttemptDetailsInfo) should be considered as part of
> the
> > > > public APIs and therefore be marked as @Public. It is true though
> none
> > of
> > > > them are marked as @public yet. Maybe we should do that. ccing
> > > > @chesnay for confirmation.
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > 
> > > > 发件人: Shammon FY 
> > > > 发送时间: 2023年9月11日 10:22
> > > > 收件人: dev@flink.apache.org 
> > > > 主题: Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager
> > > > Location in REST API and Web UI
> > > >
> > > > Thanks Zhanghao for initialing this discussion, I have just one
> > comment:
> > > >
> > > > I checked the classes `SubtasksAllAccumulatorsHandler`,
> > > > `SubtasksTimesHandler`, `SubtaskCurrentAttemptDetailsHandler`,
> > > > `JobVertexTaskManagersHandler` and `JobExceptionsHandler` you
> mentioned
> > > in
> > > > `Public Interfaces` and they are not annotated as `Public`. So do you
> > > want
> > > > to annotate them as `Plublic`? If not, I think you may need to move
> > them
> > > > from `Public Interfaces` to `Proposed Changes`.
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > > On Sat, Sep 9, 2023 at 12:11 PM Chen Zhanghao <
> > zhanghao.c...@outlook.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi 

Re: Flink and Flink shaded dependency

[Jing Ge]

Hi Martijn,

For example, bump up guava from 30.1.1-jre to 32.1.2-jre.

Best regards,
Jing

On Wed, Sep 13, 2023 at 2:37 PM Martijn Visser 
wrote:

> Hi Jing,
>
> A PR to update the readme sounds like a good plan.
>
> I think it depends on what are the expected updates for a flink-shaded
> 16.2, since that version doesn't exist.
>
> Best regards,
>
> Martijn
>
> On Wed, Sep 13, 2023 at 1:48 PM Jing Ge 
> wrote:
>
> > Hi Martijn,
> >
> > Thanks for your reply with details. Appreciate it.
> >
> >
> > > Flink-Shaded is usually only updated whenever a
> > > new Flink minor version is released and only at the beginning of the
> > > release cycle, so that there's enough time to stabilize Flink.
> >
> >
> > This is the information I am looking for. It will help devs understand
> the
> > compatibility between different versions of flink and flink-shaded, if it
> > could be described in the readme. If you don't mind, I can create a pr
> and
> > update it.
> >
> > Speaking of this rule, I have a follow-up question: do you have any
> concern
> > if flink-shaded 16.2 will be released and upgraded(from 16.1 to 16.2) in
> > Flink 1.17? Is there anything we should pay attention to while releasing
> a
> > new minor flink-shaded version?
> >
> > Best regards,
> > Jing
> >
> > On Wed, Sep 13, 2023 at 9:01 AM Martijn Visser  >
> > wrote:
> >
> > > Hi Jing,
> > >
> > > Flink Shaded exists so that Flinks internal usage of commonly used
> > packages
> > > such as Guava, Jackson inside of Flink don't clash with different
> > versions
> > > that users might use when creating a Flink application. When I did the
> > > upgrade of Flink Shaded, we already ran into a bunch of problems
> because
> > a
> > > lot of the externalized connectors relied on Flink Shaded, which made
> it
> > > problematic to get the connector to work on both Flink 1.17 and Flink
> > 1.18.
> > > There's been quite a lot of effort put into making sure that
> externalized
> > > connectors don't rely on Flink Shaded at all anymore, by either using
> > their
> > > own versions of shaded artifacts (which was the case with the Pulsar
> > > connector) or just removing the dependency on Flink Shaded all
> together,
> > by
> > > using regular Java.
> > >
> > > If you would upgrade flink-shaded from 16.1 to 17.0 in Flink 1.17, you
> > > would break all externalized connectors that rely on Flink Shaded's
> Guava
> > > version, plus you potentially would impact the runtime given that
> > there's a
> > > newer Netty version etc. Flink-Shaded is usually only updated whenever
> a
> >
> > new Flink minor version is released and only at the beginning of the
> > > release cycle, so that there's enough time to stabilize Flink.
> >
> > All in all, we shouldn't upgrade flink-shaded in Flink 1.17.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Tue, Sep 12, 2023 at 7:26 PM Jing Ge 
> > > wrote:
> > >
> > > > Hi Dev,
> > > >
> > > > Currently Flink 1.17 is using flink-shaded 16.1 and Flink 1.18 is
> using
> > > > flink-shaded 17.0. Do we need to consider any compatibility rules
> > between
> > > > them? E.g. is there any concern to upgrade flink-shaded from 16.1 to
> > 17.x
> > > > for Flink 1.17? Or there are some implicit dependency rules between
> > > > them. Looking
> > > > forward to hearing from you.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > >
> >
>

Re: Flink and Flink shaded dependency

[Martijn Visser]

Hi Jing,

A PR to update the readme sounds like a good plan.

I think it depends on what are the expected updates for a flink-shaded
16.2, since that version doesn't exist.

Best regards,

Martijn

On Wed, Sep 13, 2023 at 1:48 PM Jing Ge  wrote:

> Hi Martijn,
>
> Thanks for your reply with details. Appreciate it.
>
>
> > Flink-Shaded is usually only updated whenever a
> > new Flink minor version is released and only at the beginning of the
> > release cycle, so that there's enough time to stabilize Flink.
>
>
> This is the information I am looking for. It will help devs understand the
> compatibility between different versions of flink and flink-shaded, if it
> could be described in the readme. If you don't mind, I can create a pr and
> update it.
>
> Speaking of this rule, I have a follow-up question: do you have any concern
> if flink-shaded 16.2 will be released and upgraded(from 16.1 to 16.2) in
> Flink 1.17? Is there anything we should pay attention to while releasing a
> new minor flink-shaded version?
>
> Best regards,
> Jing
>
> On Wed, Sep 13, 2023 at 9:01 AM Martijn Visser 
> wrote:
>
> > Hi Jing,
> >
> > Flink Shaded exists so that Flinks internal usage of commonly used
> packages
> > such as Guava, Jackson inside of Flink don't clash with different
> versions
> > that users might use when creating a Flink application. When I did the
> > upgrade of Flink Shaded, we already ran into a bunch of problems because
> a
> > lot of the externalized connectors relied on Flink Shaded, which made it
> > problematic to get the connector to work on both Flink 1.17 and Flink
> 1.18.
> > There's been quite a lot of effort put into making sure that externalized
> > connectors don't rely on Flink Shaded at all anymore, by either using
> their
> > own versions of shaded artifacts (which was the case with the Pulsar
> > connector) or just removing the dependency on Flink Shaded all together,
> by
> > using regular Java.
> >
> > If you would upgrade flink-shaded from 16.1 to 17.0 in Flink 1.17, you
> > would break all externalized connectors that rely on Flink Shaded's Guava
> > version, plus you potentially would impact the runtime given that
> there's a
> > newer Netty version etc. Flink-Shaded is usually only updated whenever a
>
> new Flink minor version is released and only at the beginning of the
> > release cycle, so that there's enough time to stabilize Flink.
>
> All in all, we shouldn't upgrade flink-shaded in Flink 1.17.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Tue, Sep 12, 2023 at 7:26 PM Jing Ge 
> > wrote:
> >
> > > Hi Dev,
> > >
> > > Currently Flink 1.17 is using flink-shaded 16.1 and Flink 1.18 is using
> > > flink-shaded 17.0. Do we need to consider any compatibility rules
> between
> > > them? E.g. is there any concern to upgrade flink-shaded from 16.1 to
> 17.x
> > > for Flink 1.17? Or there are some implicit dependency rules between
> > > them. Looking
> > > forward to hearing from you.
> > >
> > > Best regards,
> > > Jing
> > >
> >
>

[jira] [Created] (FLINK-33085) Improve the error message when the invalidate lookupTableSource without primary key is used as temporal join table

[Yunhong Zheng (Jira)]

Yunhong Zheng created FLINK-33085:
-

 Summary: Improve the error message when the invalidate 
lookupTableSource without primary key is used as temporal join table
 Key: FLINK-33085
 URL: https://issues.apache.org/jira/browse/FLINK-33085
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.19.0
Reporter: Yunhong Zheng
 Fix For: 1.19.0


Improve the error message when the invalidate lookupTableSource without primary 
key is used as temporal join table.  This pr can check the legality of 
temporary table join syntax in sqlToRel phase and make the thrown error clearer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Flink and Flink shaded dependency

[Jing Ge]

Hi Martijn,

Thanks for your reply with details. Appreciate it.


> Flink-Shaded is usually only updated whenever a
> new Flink minor version is released and only at the beginning of the
> release cycle, so that there's enough time to stabilize Flink.


This is the information I am looking for. It will help devs understand the
compatibility between different versions of flink and flink-shaded, if it
could be described in the readme. If you don't mind, I can create a pr and
update it.

Speaking of this rule, I have a follow-up question: do you have any concern
if flink-shaded 16.2 will be released and upgraded(from 16.1 to 16.2) in
Flink 1.17? Is there anything we should pay attention to while releasing a
new minor flink-shaded version?

Best regards,
Jing

On Wed, Sep 13, 2023 at 9:01 AM Martijn Visser 
wrote:

> Hi Jing,
>
> Flink Shaded exists so that Flinks internal usage of commonly used packages
> such as Guava, Jackson inside of Flink don't clash with different versions
> that users might use when creating a Flink application. When I did the
> upgrade of Flink Shaded, we already ran into a bunch of problems because a
> lot of the externalized connectors relied on Flink Shaded, which made it
> problematic to get the connector to work on both Flink 1.17 and Flink 1.18.
> There's been quite a lot of effort put into making sure that externalized
> connectors don't rely on Flink Shaded at all anymore, by either using their
> own versions of shaded artifacts (which was the case with the Pulsar
> connector) or just removing the dependency on Flink Shaded all together, by
> using regular Java.
>
> If you would upgrade flink-shaded from 16.1 to 17.0 in Flink 1.17, you
> would break all externalized connectors that rely on Flink Shaded's Guava
> version, plus you potentially would impact the runtime given that there's a
> newer Netty version etc. Flink-Shaded is usually only updated whenever a

new Flink minor version is released and only at the beginning of the
> release cycle, so that there's enough time to stabilize Flink.

All in all, we shouldn't upgrade flink-shaded in Flink 1.17.
>
> Best regards,
>
> Martijn
>
> On Tue, Sep 12, 2023 at 7:26 PM Jing Ge 
> wrote:
>
> > Hi Dev,
> >
> > Currently Flink 1.17 is using flink-shaded 16.1 and Flink 1.18 is using
> > flink-shaded 17.0. Do we need to consider any compatibility rules between
> > them? E.g. is there any concern to upgrade flink-shaded from 16.1 to 17.x
> > for Flink 1.17? Or there are some implicit dependency rules between
> > them. Looking
> > forward to hearing from you.
> >
> > Best regards,
> > Jing
> >
>

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Ferenc Csaky]

Looking forward to this!

+1 (non-binding)

Cheers,
Ferenc


--- Original Message ---
On Wednesday, September 13th, 2023 at 12:33, Maximilian Michels 
 wrote:


> 
> 
> +1 (binding)
> 
> On Wed, Sep 13, 2023 at 12:28 PM Gyula Fóra gyula.f...@gmail.com wrote:
> 
> > +1 (binding)
> > 
> > Gyula
> > 
> > On Wed, 13 Sep 2023 at 09:33, Matt Wang wang...@163.com wrote:
> > 
> > > Thank you for driving this FLIP,
> > > 
> > > +1 (non-binding)
> > > 
> > > --
> > > 
> > > Best,
> > > Matt Wang
> > > 
> > >  Replied Message 
> > > | From | conradjamjam.gz...@gmail.com |
> > > | Date | 09/13/2023 15:28 |
> > > | To | dev@flink.apache.org |
> > > | Subject | Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and
> > > support the Standalone Autoscaler |
> > > best idea
> > > +1 (non-binding)
> > > 
> > > Ahmed Hamdy hamdy10...@gmail.com 于2023年9月13日周三 15:23写道：
> > > 
> > > Hi Rui,
> > > I have gone through the thread.
> > > +1 (non-binding)
> > > 
> > > Best Regards
> > > Ahmed Hamdy
> > > 
> > > On Wed, 13 Sept 2023 at 03:53, Rui Fan 1996fan...@gmail.com wrote:
> > > 
> > > Hi all,
> > > 
> > > Thanks for all the feedback about the FLIP-334:
> > > Decoupling autoscaler and kubernetes and
> > > support the Standalone Autoscaler[1].
> > > This FLIP was discussed in [2].
> > > 
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> > > insufficient votes.
> > > 
> > > [1]
> > > 
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> > > [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
> > > 
> > > Best,
> > > Rui
> > > 
> > > --
> > > Best
> > > 
> > > ConradJam

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Maximilian Michels]

+1 (binding)

On Wed, Sep 13, 2023 at 12:28 PM Gyula Fóra  wrote:
>
> +1 (binding)
>
> Gyula
>
> On Wed, 13 Sep 2023 at 09:33, Matt Wang  wrote:
>
> > Thank you for driving this FLIP,
> >
> > +1 (non-binding)
> >
> >
> > --
> >
> > Best,
> > Matt Wang
> >
> >
> >  Replied Message 
> > | From | ConradJam |
> > | Date | 09/13/2023 15:28 |
> > | To |  |
> > | Subject | Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and
> > support the Standalone Autoscaler |
> > best idea
> > +1 (non-binding)
> >
> > Ahmed Hamdy  于2023年9月13日周三 15:23写道：
> >
> > Hi Rui,
> > I have gone through the thread.
> > +1 (non-binding)
> >
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Wed, 13 Sept 2023 at 03:53, Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Hi all,
> >
> > Thanks for all the feedback about the FLIP-334:
> > Decoupling autoscaler and kubernetes and
> > support the Standalone Autoscaler[1].
> > This FLIP was discussed in [2].
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> > hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> > insufficient votes.
> >
> > [1]
> >
> >
> >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> > [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
> >
> > Best,
> > Rui
> >
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Gyula Fóra]

+1 (binding)

Gyula

On Wed, 13 Sep 2023 at 09:33, Matt Wang  wrote:

> Thank you for driving this FLIP,
>
> +1 (non-binding)
>
>
> --
>
> Best,
> Matt Wang
>
>
>  Replied Message 
> | From | ConradJam |
> | Date | 09/13/2023 15:28 |
> | To |  |
> | Subject | Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and
> support the Standalone Autoscaler |
> best idea
> +1 (non-binding)
>
> Ahmed Hamdy  于2023年9月13日周三 15:23写道：
>
> Hi Rui,
> I have gone through the thread.
> +1 (non-binding)
>
> Best Regards
> Ahmed Hamdy
>
>
> On Wed, 13 Sept 2023 at 03:53, Rui Fan <1996fan...@gmail.com> wrote:
>
> Hi all,
>
> Thanks for all the feedback about the FLIP-334:
> Decoupling autoscaler and kubernetes and
> support the Standalone Autoscaler[1].
> This FLIP was discussed in [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> insufficient votes.
>
> [1]
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
>
> Best,
> Rui
>
>
>
>
> --
> Best
>
> ConradJam
>

[jira] [Created] (FLINK-33084) Migrate globalJobParameter in ExecutionConfig to configuration instance

[Junrui Li (Jira)]

Junrui Li created FLINK-33084:
-

 Summary: Migrate globalJobParameter in ExecutionConfig to 
configuration instance
 Key: FLINK-33084
 URL: https://issues.apache.org/jira/browse/FLINK-33084
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Configuration
Reporter: Junrui Li
 Fix For: 1.19.0


Currently, the globalJobParameter field in ExecutionConfig has not been 
migrated to the Configuration. Considering the goal of unifying configuration 
options, it is necessary to migrate it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33083) SupportsReadingMetadata is not applied when loading a CompiledPlan

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-33083:


 Summary: SupportsReadingMetadata is not applied when loading a 
CompiledPlan
 Key: FLINK-33083
 URL: https://issues.apache.org/jira/browse/FLINK-33083
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.17.1, 1.16.2
Reporter: Dawid Wysakowicz


If a few conditions are met, we can not apply ReadingMetadata interface:
# source overwrites:
 {code}
@Override
public boolean supportsMetadataProjection() {
return false;
}
 {code}
# source does not implement {{SupportsProjectionPushDown}}
# table has metadata columns e.g.
{code}
CREATE TABLE src (
  physical_name STRING,
  physical_sum INT,
  timestamp TIMESTAMP_LTZ(3) NOT NULL METADATA VIRTUAL
)
{code}
# we query the table {{SELECT * FROM src}}

It fails with:
{code}
Caused by: java.lang.IllegalArgumentException: Row arity: 1, but serializer 
arity: 2
at 
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:124)
{code}

The reason is {{SupportsReadingMetadataSpec}} is created only in the 
{{PushProjectIntoTableSourceScanRule}}, but the rule is not applied when 1 & 2



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33082) Azure Pipelines 4 is not responding on AZP

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-33082:
---

 Summary: Azure Pipelines 4 is not responding on AZP
 Key: FLINK-33082
 URL: https://issues.apache.org/jira/browse/FLINK-33082
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI
Reporter: Sergey Nuyanzin


it impacts this build for 1.17 
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53163=logs=9fca669f-5c5f-59c7-4118-e31c641064f0=6e8542d7-de38-5a33-4aca-458d6c87066d]

{noformat}
##[error]We stopped hearing from agent Azure Pipelines 4. Verify the agent 
machine is running and has a healthy network connection. Anything that 
terminates an agent process, starves it for CPU, or blocks its network access 
can cause this error. For more information, see: 
https://go.microsoft.com/fwlink/?linkid=846610
Agent: Azure Pipelines 4
Started: Today at 4:59 AM
Duration: 6h 24m 38s
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Future of classical remoting in Pekko

[Ferenc Csaky]

The target release date for 1.18 is the end of Sept [1], but I'm not sure 
everything will come together by then. Maybe it will pushed by a couple days.

I'm happy to help out, even making the Flink related changes when we're at that 
point.

[1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release

--- Original Message ---
On Tuesday, September 12th, 2023 at 17:43, He Pin  wrote:


> 
> 
> Hi Ferenc:
> What's the ETA of the Flink 1.18? I think we should beable to collaborate on 
> this，and at work we are using Flink too.
> 
> On 2023/09/12 15:16:11 Ferenc Csaky wrote:
> 
> > Hi Matthew,
> > 
> > Thanks for bringing this up! Cca half a year ago I started to work on an 
> > Akka Artery migration, there is a draft PR for that 1. It might be an 
> > option to revive that work and point it against Pekko instead. Although I 
> > would highlight FLINK-29281 2 which will replace the whole RPC 
> > implementation in Flink to a gRPC-based one when it is done.
> > 
> > I am not sure about the progess on the gRPC work, it looks hanging for a 
> > while now, so I think if there is a chance to replace Netty3 with Netty4 in 
> > Pekko in the short term it would benefit Flink and then we can decide if it 
> > would worth to upgrade to Artery, or how fast the gRPC solution can be done 
> > and then it will not be necessary.
> > 
> > All in all, in the short term I think Flink would benefit to have that 
> > mentioned PR 3 merged, then the updated Pekko version could be included in 
> > the first 1.18 patch probably to mitigate those pesky Netty3 CVEs that are 
> > carried for a while ASAP.
> > 
> > Cheers,
> > Ferenc
> > 
> > 1 https://github.com/apache/flink/pull/22271
> > 2 https://issues.apache.org/jira/browse/FLINK-29281
> > 3 https://github.com/apache/incubator-pekko/pull/643
> > 
> > --- Original Message ---
> > On Tuesday, September 12th, 2023 at 10:29, Matthew de Detrich 
> > matthew.dedetr...@aiven.io.INVALID wrote:
> > 
> > > It's come to my attention that Flink is using Pekko's classical remoting,
> > > if this is the case then I would recommend making a response at
> > > https://lists.apache.org/thread/19h2wrs2om91g5vhnftv583fo0ddfshm .
> > > 
> > > Quick summary of what is being discussed is what to do with Pekko's
> > > classical remoting. Classic remoting is considered deprecated since 2019,
> > > an artifact that we inherited from Akka1. Ontop of this classical
> > > remoting happens to be using netty3 which has known CVE's2, these CVE's
> > > were never fixed in the netty3 series.
> > > 
> > > The question is what should be done given this, i.e. some people in the
> > > Pekko community are wanting to drop classical remoting as quickly as
> > > possible (i.e. even sooner then what semver allows but this is being
> > > discussed) and others are wanting to leave it as it is (even with the
> > > CVE's) since we don't want to incentivize and/or create impression that we
> > > are officially supporting it. There is also a currently open PR3 which
> > > upgrades Pekko's classical remoting's from netty3 to netty4 with the
> > > primary motivator being removing said CVE's.
> > > 
> > > My personal position on the matter is that Pekko shouldn't drop classical
> > > remoting until 2.0.x (to satisfy semver) while also updating Pekko's
> > > classical remoting netty dependency to netty4 so that we are not shipping
> > > Pekko with known CVE's (if this gets approved such a change would likely
> > > land in Pekko 1.1.0). As is customary, such a decision should be agreed
> > > upon broadly in the Pekko community.
> > > 
> > > Note that regardless of this change, it's recommended that a plan should 
> > > be
> > > made at some point by Flink to move from classical remoting to artery4
> > > although the decision that Pekko ultimately makes may influence the
> > > timeline (hence the reason for this thread).
> > > 
> > > --
> > > 
> > > Matthew de Detrich
> > > 
> > > Aiven Deutschland GmbH
> > > 
> > > Immanuelkirchstraße 26, 10405 Berlin
> > > 
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > > 
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > 
> > > m: +491603708037
> > > 
> > > w: aiven.io e: matthew.dedetr...@aiven.io

[jira] [Created] (FLINK-33081) Move parallelism override logic into scale method

[Gyula Fora (Jira)]

Gyula Fora created FLINK-33081:
--

 Summary: Move parallelism override logic into scale method
 Key: FLINK-33081
 URL: https://issues.apache.org/jira/browse/FLINK-33081
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora
Assignee: Gyula Fora
 Fix For: kubernetes-operator-1.7.0


After FLINK-32589  the parallelism overrides are applied separately from the 
scale call of the autoscaler implementation. We should simplify this by a small 
refactoring



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

[Lorenzo Nicora]

Hello
I was working with Karthi and took over development. I have a working
version already.
I also updated the FLIP (now FLIP-312, it was renumbered due to a clash)

Please, let me know how to proceed with this. Happy to answer any questions
about the FLIP

Regards
Lorenzo

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector

On 2023/08/21 12:01:51 Ahmed Hamdy wrote:
> Hello Karthi
> Is this FLIP still in progress? I see the FLIP not updated & couldn't find
> open JIRAs.
> I am happy to take over if you are no longer working on this.
> Best Regards
> Ahmed Hamdy
>
>
> On Mon, 22 May 2023 at 14:49, Martijn Visser 
> wrote:
>
> > Hi all,
> >
> > > For example, a user might want to read in logs, perform some
aggregations
> > and publish it into a metrics store for visualisation. This might be a
> > great use-case for reducing the cardinality of metrics!
> >
> > I can see that. What I would like to see in the FLIP is a proposal on
the
> > boundaries of the metrics reporter vs the Prometheus sink. I think it's
> > important that we make clear when to use a metric reporter and when
not. I
> > can imagine that there will be Flink users who think that they can get
data
> > from the metric reporter, make aggregrations in Flink and then store it
> > using the Prometheus sink.
> >
> > Overall, I think more context must be added to the FLIP, especially on
the
> > motivation.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan 
> > wrote:
> >
> > > Hi Lijie
> > >
> > > Thank you for pointing this out. I've corrected it [1]. Also, this
page
> > > [2] still shows 178 and 229 as available, which is why I picked it up.
> > >
> > > Thanks
> > > Karthi
> > >
> > > [1]
> > >
> >
https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector
> > > [2]
> > >
> >
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >
> > > On May 15, 2023, at 9:37 PM, Lijie Wang 
> > wrote:
> > >
> > >
> > > Hi Karthi,
> > >
> > > I think you are using a wrong FLIP id, the FLIP-229 has already be
> > used[1].
> > >
> > > [1]
> > >
> > >
> >
https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job
> > >
> > > Best,
> > > Lijie
> > >
> > > Martijn Visser  于2023年5月16日周二 04:44写道：
> > >
> > > Hi Karthi,
> > >
> > > Thanks for the FLIP and opening up the discussion. My main question
is:
> > why
> > > should we create a separate connector and not use and/or improve the
> > > existing integrations with Prometheus? I would like to understand
more so
> > > that it can be added to the motivation of the FLIP.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan <
> > kar...@karthitect.com>
> > > wrote:
> > >
> > > > Hello all,
> > > >
> > > > We would like to start a discussion thread on FLIP-229: Prometheus
Sink
> > > > Connector [1] where we propose to provide a sink connector for
> > Prometheus
> > > > [2] based on the Async Sink [3]. Looking forward to comments and
> > > feedback.
> > > > Thank you.
> > > >
> > > > [1]
> > > >
> > >
> > >
> >
https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector
> > > > [2] https://prometheus.io/
> > > > [3]
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> > > >
> > >
> > >
> > >
> >
>

Re: [DISCUSS] FLINK-25927: Make flink-connector-base dependency usage consistent across all connectors

[Hang Ruan]

Hi, all.

I would like to help to do some work about this issue. Because some classes
in flink-connector-base are supposed to be used inside the user jar
directly, FLINK-25927[1] has been reverted by FLINK-26701[2].

And the final solution is as follows.
- package flink-connector-base into flink-dist
- the external connectors will not bundle the connector-base module, which
is written in the Externalized Connector development docs[3]

But the most external connectors still bundle the connector-base module
now. I will check this problem and stop bundling connector-base module in
every externalized connector[4].

Best,
Hang

[1] https://issues.apache.org/jira/browse/FLINK-25927
[2] https://issues.apache.org/jira/browse/FLINK-26701
[3]
https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development
[4] https://issues.apache.org/jira/browse/FLINK-30400

Alexander Fedulov  于2022年2月14日周一 21:34写道：

> Hi,
>
> Thomas, Chesnay, thank you for your input. Below I will try to capture two
> actionable alternatives together with their benefits and downsides:
>
> Alternative #1: Package flink-connector-base into flink-dist
>
> Downsides:
> - breaks existing CI/IDE setup that previously neither relied on flink-dist
> nor added flink-connector-base as a dependency
> - could break existing connectors due to conflicts between
> flink-connector-base of different version (if they did not relocate it)
> - more work: flink-dist needs publishing to maven central to provide a
> solution for CI/IDE setups (this is currently not done)
> - flink-dist is heavy: currently about 118MB, which could be potentially
> reduced to ~70MB by removing parts that are not directly related to
> interfaces, like flink-kubernetes, but this needs more work
>
> Benefits:
> - consistency: flink-connector-base does not get "special treatment" when
> compared to other Flink APIs
> - makes it easier for connector base to use utilities of Flink (evolve
> together)
> - makes it easier to evolve dependency on core, table-commons (only source
> compatibility required, not binary)
>
>
> Alternative #2: shade and relocate flink-connector-base in every connector
>
> Downsides:
> - will break connectors that were previously transitively pulling it in via
> flink-connector-files/flink-table uber jar
> - treats this API differently than the other Flink APIs
> - increased API compatibility surface: everything that flink-connector-base
> relies on (flink-core, flink-table-commons) has to be binary compatible
> between the versions, not just the flink-connector-base itself
>
> Benefits:
> - less work from the implementation perspective - flink-dist does not need
> to be published
> - does not break existing CI/IDE setups
> - also no need to pull in the sizeable flink-dist dependency for running in
> IDEs and CI
>
>
> All in all, the issue seems to boil down to the question of API
> compatibility guarantees, as has already been rightly pointed out in this
> thread. The main difference between the approaches is were the
> compatibility guarantee emphasis is put:
>
> 1: connector -> *COMPATIBLE* -> connector-base -> [core, table-common]
> 2: connector -> connector-base -> *COMPATIBLE* -> [core, table-common]
>
> As you see, both approaches are not ideal and have their downsides. A
> better solution could be the one where users rely on a single lightweight
> module that encapsulates all public APIs. This module could then evolve in
> sync and with strict @Public compatibility guarantees. Such an approach is
> a significant effort and, as Thomas mentioned, is only hinted at in
> FLIP-196 as the eventual goal. To move forwards while minimizing the
> potential to break existing connectors and setups, we could try to reap the
> benefits and to mitigate the downsides by combining Alternative #1 and
> Alternative #2, i.e.:
>
>  - shade and relocate all dependencies to flink-connector-base for the
> connectors maintained within Flink
>  - add a documentation notice which asks external connector developers to
> also shade and relocate flink-connector-base in their implementations
>  - package flink-connector-base into flink-dist
>
> This would allow both not to break the existing CI/IDE setups
> (flink-connector-base remains included into connectors) while also not
> break the connectors that were previously pulling in flink-connector-base
> via flink-connector-files/flink-table.
>
> The mixed solution is not meant to be a permanent one, and we should
> revisit the API compatibility topic in 1.16.
>
> Let me know what you think.
>
> Thanks,
> Alexander Fedulov
>
> On Mon, Feb 14, 2022 at 10:01 AM Chesnay Schepler 
> wrote:
>
> > Letting connectors bundle it doesn't necessarily make it harder to
> > achieve; that all depends on how we approach it;
> > e.g., everything that connector-base uses from the core Flink could be
> > required to also be annotated with Public(Evolving).
> > (i.e., treat it as if it were externalized)
> >
> > On 13/02/2022 02:12, Thomas 

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Matt Wang]

Thank you for driving this FLIP,

+1 (non-binding)


--

Best,
Matt Wang


 Replied Message 
| From | ConradJam |
| Date | 09/13/2023 15:28 |
| To |  |
| Subject | Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and 
support the Standalone Autoscaler |
best idea
+1 (non-binding)

Ahmed Hamdy  于2023年9月13日周三 15:23写道：

Hi Rui,
I have gone through the thread.
+1 (non-binding)

Best Regards
Ahmed Hamdy


On Wed, 13 Sept 2023 at 03:53, Rui Fan <1996fan...@gmail.com> wrote:

Hi all,

Thanks for all the feedback about the FLIP-334:
Decoupling autoscaler and kubernetes and
support the Standalone Autoscaler[1].
This FLIP was discussed in [2].

I'd like to start a vote for it. The vote will be open for at least 72
hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
insufficient votes.

[1]


https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
[2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495

Best,
Rui




--
Best

ConradJam

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[ConradJam]

best idea
+1 (non-binding)

Ahmed Hamdy  于2023年9月13日周三 15:23写道：

> Hi Rui,
> I have gone through the thread.
> +1 (non-binding)
>
> Best Regards
> Ahmed Hamdy
>
>
> On Wed, 13 Sept 2023 at 03:53, Rui Fan <1996fan...@gmail.com> wrote:
>
> > Hi all,
> >
> > Thanks for all the feedback about the FLIP-334:
> > Decoupling autoscaler and kubernetes and
> > support the Standalone Autoscaler[1].
> > This FLIP was discussed in [2].
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> > hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> > insufficient votes.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> > [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
> >
> > Best,
> > Rui
> >
>


-- 
Best

ConradJam

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Ahmed Hamdy]

Hi Rui,
I have gone through the thread.
+1 (non-binding)

Best Regards
Ahmed Hamdy


On Wed, 13 Sept 2023 at 03:53, Rui Fan <1996fan...@gmail.com> wrote:

> Hi all,
>
> Thanks for all the feedback about the FLIP-334:
> Decoupling autoscaler and kubernetes and
> support the Standalone Autoscaler[1].
> This FLIP was discussed in [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
>
> Best,
> Rui
>

[jira] [Created] (FLINK-33080) The checkpoint storage configured in the job level by config option will not take effect

[Junrui Li (Jira)]

Junrui Li created FLINK-33080:
-

 Summary: The checkpoint storage configured in the job level by 
config option will not take effect
 Key: FLINK-33080
 URL: https://issues.apache.org/jira/browse/FLINK-33080
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Junrui Li
 Fix For: 1.19.0


When we configure the checkpoint storage at the job level, it can only be done 
through the following method:
{code:java}
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
env.getCheckpointConfig().setCheckpointStorage(xxx); {code}
However, configure the checkpoint storage by the job-side configuration like 
the following will not take effect:
{code:java}
Configuration configuration = new Configuration();
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment(configuration);
configuration.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
configuration.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, checkpointDir); 
{code}
This behavior is unexpected, we should allow this way will take effect.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Flink and Flink shaded dependency

[Martijn Visser]

Hi Jing,

Flink Shaded exists so that Flinks internal usage of commonly used packages
such as Guava, Jackson inside of Flink don't clash with different versions
that users might use when creating a Flink application. When I did the
upgrade of Flink Shaded, we already ran into a bunch of problems because a
lot of the externalized connectors relied on Flink Shaded, which made it
problematic to get the connector to work on both Flink 1.17 and Flink 1.18.
There's been quite a lot of effort put into making sure that externalized
connectors don't rely on Flink Shaded at all anymore, by either using their
own versions of shaded artifacts (which was the case with the Pulsar
connector) or just removing the dependency on Flink Shaded all together, by
using regular Java.

If you would upgrade flink-shaded from 16.1 to 17.0 in Flink 1.17, you
would break all externalized connectors that rely on Flink Shaded's Guava
version, plus you potentially would impact the runtime given that there's a
newer Netty version etc. Flink-Shaded is usually only updated whenever a
new Flink minor version is released and only at the beginning of the
release cycle, so that there's enough time to stabilize Flink. All in all,
we shouldn't upgrade flink-shaded in Flink 1.17.

Best regards,

Martijn

On Tue, Sep 12, 2023 at 7:26 PM Jing Ge  wrote:

> Hi Dev,
>
> Currently Flink 1.17 is using flink-shaded 16.1 and Flink 1.18 is using
> flink-shaded 17.0. Do we need to consider any compatibility rules between
> them? E.g. is there any concern to upgrade flink-shaded from 16.1 to 17.x
> for Flink 1.17? Or there are some implicit dependency rules between
> them. Looking
> forward to hearing from you.
>
> Best regards,
> Jing
>

[jira] [Created] (FLINK-33079) The gap between the checkpoint timeout and the interval settings is too large

[Fangliang Liu (Jira)]

Fangliang Liu created FLINK-33079:
-

 Summary: The gap between the checkpoint timeout and the interval 
settings is too large
 Key: FLINK-33079
 URL: https://issues.apache.org/jira/browse/FLINK-33079
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.19.0
Reporter: Fangliang Liu


The gap between the checkpoint timeout and the interval settings is too large

Some users will think that the documentation is the optimal solution and refer 
to this demo setting, 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [Discuss] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

[Venkatakrishnan Sowrirajan]

In this case, we required hive-site.xml which is available only on the
client side and not on all the TM nodes. But the workaround you mentioned
for HADOOP_CONF_DIR should work.

On Tue, Sep 12, 2023, 7:35 PM Biao Geng  wrote:

> +1 for the FLIP.
> Another side thought is that in my experience, when users want to make
> Flink TM use a specific hadoop/hive configuration, an easier way is to ship
> the corresponding conf dir and set the env
> variable via containerized.taskmanager.env.HADOOP_CONF_DIR.
>
> Best,
> Biao Geng
>
> Archit Goyal  于2023年9月13日周三 08:00写道：
>
> > Hi All,
> >
> > If there are no further concerns about this FLIP, I will start a vote
> > thread tomorrow.
> >
> > Thanks,
> > Archit Goyal
> >
> >
> > From: Venkatakrishnan Sowrirajan 
> > Date: Thursday, August 24, 2023 at 10:21 PM
> > To: dev@flink.apache.org 
> > Subject: Re: [Discuss] FLIP-355: Add parent dir of files to classpath
> > using yarn.provided.lib.dirs
> > Thanks Yang Wang.
> >
> > In that case, whenever you get a chance could you please review the PR?
> >
> >
> > On Wed, Aug 23, 2023, 8:15 PM Yang Wang  wrote:
> >
> > > +1 for this FLIP
> > >
> > > Maybe a FLIP is an overkill for this enhancement.
> > >
> > >
> > > Best,
> > > Yang
> > >
> > > Venkatakrishnan Sowrirajan  于2023年8月23日周三 01:44写道：
> > >
> > > > Thanks for the FLIP, Archit.
> > > >
> > > > This is definitely quite a useful addition to the
> > > *yarn.provided.lib.dirs*
> > > > . +1.
> > > >
> > > > IMO, except for the fact that *yarn.provided.lib.dirs* (platform
> > specific
> > > > jars can be cached) takes only directories whereas *yarn.ship-files*
> > > (user
> > > > files) takes both files and dirs, the overall logic in terms of
> > > > constructing the classpath in both the cases should be roughly the
> > same.
> > > >
> > > > Referencing the PR (
> > >
> >
> https://urldefense.com/v3/__https://nam06.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fgithub.com*2Fapache*2Fflink*2Fpull*2F23164__*3B!!IKRxdwAv5BmarQ!cgTpodngoQAdd-qu3CvbQeAwENiu1nf0eahTH-v1NhUsSn4Y7M7sVGQYSnBjB2XgaOlyzGe7XEiU3-cAOoy84Kw*24=05*7C01*7Cargoyal*40linkedin.com*7C1e626c31ecaf408575b008dba52b1c6a*7C72f988bf86f141af91ab2d7cd011db47*7C0*7C0*7C638285376817262437*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*7C*7C*7C=*2F8QiBnfVJlLZ9atF1WamCsMbAaK0TzEgcCvmd85uSnk*3D=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!IKRxdwAv5BmarQ!anzL1056sJNn-x90OFKxrO_kLtu8STP0j1s8uxyruNAXzhrzTf9sIRbb28d34GQlnza8cVlB8jWs1vgPXw$
> > <
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/pull/23164__;!!IKRxdwAv5BmarQ!cgTpodngoQAdd-qu3CvbQeAwENiu1nf0eahTH-v1NhUsSn4Y7M7sVGQYSnBjB2XgaOlyzGe7XEiU3-cAOoy84Kw$
> > >
> > > ) with the
> > > > initial implementation you created as well here.
> > > >
> > > > Regards
> > > > Venkata krishnan
> > > >
> > > >
> > > > On Tue, Aug 22, 2023 at 10:09 AM Archit Goyal
> > >  > > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Gentle ping if I can get a review on the FLIP.
> > > > >
> > > > > Thanks,
> > > > > Archit Goyal
> > > > >
> > > > > From: Archit Goyal 
> > > > > Date: Thursday, August 17, 2023 at 5:51 PM
> > > > > To: dev@flink.apache.org 
> > > > > Subject: [Discuss] FLIP-355: Add parent dir of files to classpath
> > using
> > > > > yarn.provided.lib.dirs
> > > > > Hi All,
> > > > >
> > > > > I am opening this thread to discuss the proposal to add parent
> > > > directories
> > > > > of files to classpath when using yarn.provided.lib.dirs. This is
> > > > documented
> > > > > in FLIP-355 <
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://nam06.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fcwiki.apache.org*2Fconfluence*2Fdisplay*2FFLINK*2FFLIP*355*3A*Add*parent*dir*of*files*to*classpath*using*yarn.provided.lib.dirs__*3BKyUrKysrKysrKys!!IKRxdwAv5BmarQ!fFlyBKWuWwYcWfOcpLflTTi36tyHPiENIUry9J0ygaZY0VURnIs0glu5yafV0w0tfSsnOb9ZxDD9Cjw2TApTekVU*24=05*7C01*7Cargoyal*40linkedin.com*7C1e626c31ecaf408575b008dba52b1c6a*7C72f988bf86f141af91ab2d7cd011db47*7C0*7C0*7C638285376817262437*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*7C*7C*7C=FfyKaJDglL1caECVWX0nX8SwhHMnejfFMLjtpo5pizo*3D=0__;JSUlJSUlJSUlJSUlKioqKioqKioqKiolJSUlJSUlJSUlJSUlJSUlJSU!!IKRxdwAv5BmarQ!anzL1056sJNn-x90OFKxrO_kLtu8STP0j1s8uxyruNAXzhrzTf9sIRbb28d34GQlnza8cVlB8jVjx2XcPQ$
> > <
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP*355*3A*Add*parent*dir*of*files*to*classpath*using*yarn.provided.lib.dirs__;KyUrKysrKysrKys!!IKRxdwAv5BmarQ!fFlyBKWuWwYcWfOcpLflTTi36tyHPiENIUry9J0ygaZY0VURnIs0glu5yafV0w0tfSsnOb9ZxDD9Cjw2TApTekVU$
> > >
> > > > > >.
> > > > >
> > > > > This FLIP mentions about enhancing YARN's classpath configuration
> to
> > > > > include parent directories of files in yarn.provided.lib.dirs.
> > > > >
> > > > > Please feel free to reply to