: Artemis User
Sent: Thursday, November 3, 2022 8:35 PM
To: user@spark.apache.org
Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of
executors when using GPUs
ATTENTION: This email originated from outside of GM.
Now I see what you want to do. If you have access to the cl
November 6, 2022 4:19 PM
To: Shay Elbaz
Cc: Artemis User ; Tom Graves ;
Tom Graves ; user@spark.apache.org
Subject: [EXTERNAL] Re: Re: Re: Re: Re: Stage level scheduling - lower the
number of executors when using GPUs
ATTENTION: This email originated from outside of GM.
May I ask why the ETL
0 GPUs.
>
> I hope that was more clear.
> Thank you very much for helping.
>
> Shay
>
> --
> *From:* Tom Graves
> *Sent:* Friday, November 4, 2022 4:19 PM
> *To:* Tom Graves ; Artemis User <
> arte...@dtechspace.com>; user@spark.apache.org ;
>
ves ; Artemis User
; user@spark.apache.org ; Shay
Elbaz
Subject: [EXTERNAL] Re: Re: Re: Re: Stage level scheduling - lower the number
of executors when using GPUs
ATTENTION: This email originated from outside of GM.
So I'm not sure I completely follow. Are you asking for a way to change
utes, if not much more. Any failure during this long
time is pretty expensive.
ShayFrom: Tom Graves
Sent: Thursday, November 3, 2022 7:56 PM
To: Artemis User ; user@spark.apache.org
; Shay Elbaz
Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of
executors when using
rsday, November 3, 2022 7:56 PM
To: Artemis User ; user@spark.apache.org
; Shay Elbaz
Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of
executors when using GPUs
ATTENTION: This email originated from outside of GM.
Stage level scheduling does not allow you to change co
2 1:16 AM
*To:* user@spark.apache.org <mailto:user@spark.apache.org>
<mailto:user@spark.apache.org>
*Subject:* [EXTERNAL] Re: Stage level scheduling - lower the number
of executors when using GPUs
*ATTENTION:*This email originated from outside of GM.
Are you using Rapids for GPU s
he profile with 1 GPU per executor. So,
the question is how do I limit the stage resources to 20 GPUs total?
Thanks again,Shay
From: Artemis User
Sent: Thursday, November 3, 2022 5:23 PM
To: user@spark.apache.org
Subject: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of
executors
;> Shay
>>
>> --
>> *From:* Artemis User
>> *Sent:* Thursday, November 3, 2022 5:23 PM
>> *To:* user@spark.apache.org
>> *Subject:* [EXTERNAL] Re: Re: Stage level scheduling - lower the number
>> of executors when using GPUs
&g
ces to 20 GPUs total?
>
> Thanks again,
> Shay
>
> --
> *From:* Artemis User
> *Sent:* Thursday, November 3, 2022 5:23 PM
> *To:* user@spark.apache.org
> *Subject:* [EXTERNAL] Re: Re: Stage level scheduling - lower the number
> of executors when using GPUs
&
to 20 GPUs total?
Thanks again,
Shay
From: Artemis User
Sent: Thursday, November 3, 2022 5:23 PM
To: user@spark.apache.org
Subject: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of
executors when using GPUs
ATTENTION: This email originated from out
*From:* Artemis User
*Sent:* Thursday, November 3, 2022 1:16 AM
*To:* user@spark.apache.org
*Subject:* [EXTERNAL] Re: Stage level scheduling - lower the number of
executors when using GPUs
*ATTENTION:*This email originated from outside of GM.
Are you using Rapids for GPU support in Spark? Cou
ResourceProfile, directly (API) or indirectly (some advanced workaround).
Thanks,
Shay
From: Artemis User
Sent: Thursday, November 3, 2022 1:16 AM
To: user@spark.apache.org
Subject: [EXTERNAL] Re: Stage level scheduling - lower the number of executors
when using GPUs
Are you using Rapids for GPU support in Spark? Couple of options you
may want to try:
1. In addition to dynamic allocation turned on, you may also need to
turn on external shuffling service.
2. Sounds like you are using Kubernetes. In that case, you may also
need to turn on shuffle track
Hi,
Our typical applications need less executors for a GPU stage than for a CPU
stage. We are using dynamic allocation with stage level scheduling, and Spark
tries to maximize the number of executors also during the GPU stage, causing a
bit of resources chaos in the cluster. This forces us to u
15 matches
Mail list logo
