Hi David,
Yes We have multiple disjoint DAGs in one job. We want better CPU
utilization. Open Source Flink has a scheduling issue with those types of
jobs. I made a fix on 1.13 with AS. Now we are scheduling evenly for all
DAGs. However somehow when we get an exception on a DAG we dont want to
a
> Our DAG has multiple sources which are not connected to each other.
To double-check, are you saying the job consists of multiple disjoint DAGs?
> Do you think somehow the adaptive scheduler supports region pipelines
for streaming jobs ?
It's doable but might not be straightforward since the A
Thanks David and others.
Our DAG has multiple sources which are not connected to each other. If one
of them fails, I believe Flink can restart a single region for
defaultscheduler. but it is not the same case for adaptive scheduler. Do
you think somehow the adaptive scheduler supports region pipel
Hi Talat,
For most streaming pipelines, we have to restart the whole pipeline no
matter the scheduler used because they're a single pipelined region. One
limitation of AdaptiveScheduler is the lack of support for local recovery.
This will be addressed in Flink 1.18 [1].
[1] https://issues.apache.
Hi,
AFAIK, the reactive mode always restarts the whole pipeline now.
Best,
Weihua
On Tue, Apr 11, 2023 at 8:38 AM Talat Uyarer via user
wrote:
> Hi All,
>
> We use Flink 1.13 with reactive mode for our streaming jobs. When we have
> an issue/exception on our pipeline. Flink rescheduled all ta
Hi All,
We use Flink 1.13 with reactive mode for our streaming jobs. When we have
an issue/exception on our pipeline. Flink rescheduled all tasks. Is there
any way to reschedule only task that had exceptions ?
Thanks
