Hi Peter & Diljeet!
My general feedback is that we should try to introduce extension plugins
instead of plugins that completely replace key parts of the autoscaler code.
Let me give you a concrete example through FLIP-514 and FLIP-543 using the
MetricsEvaluator pluggability.
The MetricsEvaluator
Jaehyun Kim created FLINK-38284:
---
Summary: Prepare to upgrade hadoop version to 3.4.2 across Flink's
Hadoop-based FS connectors for OpenSSL 3 and Java 17 compatibility
Key: FLINK-38284
URL: https://issues.apache.org
Hi Diljeet,
Yes, I think we have similar requirements to make autoscaler even more
powerful to handle some customized requirements.
The quick PoC makes sense to me. Let's get some more feedback from the
community.
Best Regards
Peter Huang
On Mon, Aug 25, 2025 at 2:37 PM Peter Huang
wrote:
Just try to combine the discussion into one thread.
@Diljeet Singh
Posted a quick PoC for the proposal
https://github.com/apache/flink-kubernetes-operator/pull/1020.
On Mon, Aug 25, 2025 at 7:52 AM Peter Huang
wrote:
> Hi Community,
>
> Our org has been heavily using the Flink autoscaling al
this draft PR should cover part of your requirements
https://github.com/apache/flink-kubernetes-operator/pull/1020
this draft PR should cover part of your requirements [Draft] support overriding
autoscaling scaling realizer via plugins by ctrlaltdilj · Pull Request #1020 ·
apache/flink-kubernetes-operator github.com
Here is a sample in-progress PR:
https://github.com/apache/flink-kubernetes-operator/pull/1020/files Is this in
the direction you were thinking Gyula?
Zhenqiu Huang created FLINK-38283:
-
Summary: Support customized autoscaling algorithm
Key: FLINK-38283
URL: https://issues.apache.org/jira/browse/FLINK-38283
Project: Flink
Issue Type: New Fe
raphaelauv created FLINK-38282:
--
Summary: K8S operator : S3 jarURI
Key: FLINK-38282
URL: https://issues.apache.org/jira/browse/FLINK-38282
Project: Flink
Issue Type: Bug
Affects Versions: 2
Hi Community,
Our org has been heavily using the Flink autoscaling algorithm. It greatly
reduced our operation overhead and improved cost efficiency
as users always over provision resources when onboard. Recently, we have
had some requirements to customize the auto scaling algorithm
for different
Hi,
I'm not sure that this is a good solution. What happens when failOnError is
set to FALSE, but the deliveryGuarantee is set to EXACTLY_ONCE?
Best regards,
Martijn
On Sun, Aug 24, 2025 at 5:57 PM Ejaskhan S wrote:
> Hi All,
>
> Any feedback on this ?
>
> Best regards.
> Ejaskhan
>
>
>
>
> O
Going to try the plugin approach similar to
https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler
I'll draft up a PR
Hey Poorvank,
Thanks for implementing the disk-based storage solution to address
the root cause.
> Just to clarify, the goal is to support runtime updates (e.g., triggering
sampling via REST/UI without restarts).
Or introducing a new config option to indicate which config options
can be updated
Hi, Matthias.
Thank you very much for the comments.
> I was thinking about the case where a rescale was initiated by
> calling updateJobResourcesRequirements but it's not executed, yet, before
> the error occurs. In that case, a restart will be triggered that implicitly
> executes the rescale ba
that makes sense, for the scalingrealizer plugin/hook, does it make sense to add an
autoscaler option something like `job.autoscaler.scalingrealizer.class` maybe? we can have
it default to the existing implementation On Aug 25, 2025, at 3:42 AM, Gyula Fóra
wrote: Hi! In general I am not a big
Hi!
In general I am not a big fan of these blanket overrides of entire
implementations of the autoscaler logic for instance. The main problem I
see is that it does not promote a nice modular and pluggable design but
instead just makes it easier to create effective forks of the codebase
without hav
Hi Zender,
Great proposal!
+1 (non-binding)
Cheers,
Ramin
On Wed, Aug 20, 2025 at 7:12 PM Zander Matheson
wrote:
> Hi Everyone,
>
> The discussion for FLIP-541, "Making PyFlink more Pythonic (Phase-1)" has
> concluded, and I would like to call a vote to accept the proposal.
>
> You can find
Hi Yuepeng,
thanks for the update. Please see my responses below:
If, during failure handling and job recovery (i.e., when there is an
> ongoing rescale), a new resource requirement change or new available
> resource change occurs (provided that such a change can trigger a new
> rescale), the ongo
18 matches
Mail list logo
