domain. One lesson we had was, it is hard to have a
>>>>>>> generic algorithm that worked for all cases.
>>>>>>>
>>>>>>> Regards
>>>>>>> kalyan.
>>>>>>>
>>>>>>>
>>>>&
gt;>>>>> 4. Stitch resource profile choices into query execution.
>>>>>>>> 5. Allow different DEA algo to be chosen for different queries
>>>>>>>> within the same spark application.
>>>>>>>> 6. Fall
gt;>>> I am aware of a few experiments carried out in this area by
>>>>>>> my friends in this domain. One lesson we had was, it is hard to have a
>>>>>>> generic algorithm that worked for all cases.
>>>>
>>>>>>> Thanks for pointing out this feature to me. I will have a look when
>>>>>>> I get there.
>>>>>>>
>>>>>>> Mich Talebzadeh,
>>>>>>> Solutions Architect/Engineering Lead
>>>>>
...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for pointing out this feature to me. I will have a look when
>>>>>>> I get there.
>>>>>>>
>>>>>>> Mich Talebz
gt;>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>> for any loss, damage or destruction
t your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any m
>>>>
>>>> On Tue, 8 Aug 2023 at 11:44, roryqi(齐赫) wrote:
>>>>
>>>>> Spark 3.5 have added an method `supportsReliableStorage` in the `
>>>>> ShuffleDriverComponents` which indicate whether writing shuffle data
>>>>> to
t; https://github.com/apache/incubator-uniffle). It can enhance the
>>>> experience of Spark on K8S. After Spark 3.5 is released, Uniffle will
>>>> support the `ShuffleDriverComponents`. you can see [1].
>>>>
>>>> If you have interest about more d
m/apache/incubator-uniffle). It can enhance the
>>> experience of Spark on K8S. After Spark 3.5 is released, Uniffle will
>>> support the `ShuffleDriverComponents`. you can see [1].
>>>
>>> If you have interest about more details of Uniffle, you can see [2]
ffle%20-%20New%20chapter%20for%20the%20shuffle%20in%20the%20cloud%20native%20era
>>
>>
>>
>> *发件人**: *Mich Talebzadeh
>> *日期**: *2023年8月8日 星期二 06:53
>> *抄送**: *dev
>> *主题**: *[Internet]Re: Improving Dynamic Allocation Logic for Spark 4+
>>
>&
> > - Advisory user input (e.g. a way to say after X is done I know I need Y
> > where Y might be a bunch of GPU machines)
You are thinking of something more advanced than the Stage Level
Scheduling? Or perhaps configured differently or prestarting things
you know you will need?
Tom
On Mon,
>
> *发件人**: *Mich Talebzadeh
> *日期**: *2023年8月8日 星期二 06:53
> *抄送**: *dev
> *主题**: *[Internet]Re: Improving Dynamic Allocation Logic for Spark 4+
>
>
>
> On the subject of dynamic allocation, is the following message a cause for
> concern when running Spark on k8s?
]
https://uniffle.apache.org/blog/2023/07/21/Uniffle%20-%20New%20chapter%20for%20the%20shuffle%20in%20the%20cloud%20native%20era
发件人: Mich Talebzadeh
日期: 2023年8月8日 星期二 06:53
抄送: dev
主题: [Internet]Re: Improving Dynamic Allocation Logic for Spark 4+
On the subject of dynamic allocation
On the subject of dynamic allocation, is the following message a cause for
concern when running Spark on k8s?
INFO ExecutorAllocationManager: Dynamic allocation is enabled without a
shuffle service.
Mich Talebzadeh,
Solutions Architect/Engineering Lead
London
United Kingdom
view my Linkedin
Hi,
>From what I have seen spark on a serverless cluster has hard up getting the
driver going in a timely manner
Annotations: autopilot.gke.io/resource-adjustment:
Oh great point
On Mon, Aug 7, 2023 at 2:23 PM bo yang wrote:
> Thanks Holden for bringing this up!
>
> Maybe another thing to think about is how to make dynamic allocation more
> friendly with Kubernetes and disaggregated shuffle storage?
>
>
>
> On Mon, Aug 7, 2023 at 1:27 PM Holden Karau
Thanks Holden for bringing this up!
Maybe another thing to think about is how to make dynamic allocation more
friendly with Kubernetes and disaggregated shuffle storage?
On Mon, Aug 7, 2023 at 1:27 PM Holden Karau wrote:
> So I wondering if there is interesting in revisiting some of how
So I wondering if there is interesting in revisiting some of how Spark is
doing it's dynamica allocation for Spark 4+?
Some things that I've been thinking about:
- Advisory user input (e.g. a way to say after X is done I know I need Y
where Y might be a bunch of GPU machines)
- Configurable
19 matches
Mail list logo
