Hi, Arun.
Here are some answers to your questions.
First, the fetch failure is irrelevant to the Executor Rolling feature
because the plugin itself only asked the Spark scheduler to decommission
it, not terminate it. More specifically, it's independent from the
underlying Decommissioning
Hi,
The crux of the matter here as I understand is " how should I be using
Executor Rolling, without triggering stage failures?"
The object of executor rolling is to replace decommissioning executors with
new ones while minimizing the impact on running tasks and stages. in k8s.
As mentioned
unsubscribe
Hi Qian,
How in practice have you implemented image caching for the driver and
executor pods respectively?
Thanks
On Thu, 24 Aug 2023 at 02:44, Qian Sun wrote:
> Hi Mich
>
> I agree with your opinion that the startup time of the Spark on Kubernetes
> cluster needs to be improved.
>
>
Hi Team,
I am running Apache Spark 3.4.1 Application on K8s with the below
configuration related to executor rolling and Ignore Decommission Fetch
Failure.
spark.plugins: "org.apache.spark.scheduler.cluster.k8s.ExecutorRollPlugin"
spark.kubernetes.executor.rollInterval: "1800s"
Hi, All.
Java 21 will be released in a month and Apache Spark master branch
(4.0.0-SNAPSHOT) achieved the first milestone (SPARK-43831: Build and Run
Spark on Java 21) Today.
1. JDK 21: https://openjdk.org/projects/jdk/21/
- 2023/08/24 Final Release Candidate
- 2023/09/19 General
