Re: External Spark shuffle service for k8s

2024-04-08 Thread Vakaris Baškirov
>> >> HTH >> >> Mich Talebzadeh, >> >> Technologist | Solutions Architect | Data Engineer | Generative AI >> London >> United Kingdom >> >> >>view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh

Re: External Spark shuffle service for k8s

2024-04-07 Thread Vakaris Baškirov
There is an IBM shuffle service plugin that supports S3 https://github.com/IBM/spark-s3-shuffle Though I would think a feature like this could be a part of the main Spark repo. Trino already has out-of-box support for s3 exchange (shuffle) and it's very useful. Vakaris On Sun, Apr 7, 2024 at

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

2024-03-20 Thread Vakaris Baškirov
Hi! Just wanted to inquire about the status of the official operator. We are looking forward to contributing and later on switching to a Spark Operator and we would prefer it to be the official one. Thanks, Vakaris On Thu, Nov 30, 2023 at 7:09 AM Shiqi Sun wrote: > Hi Zhou, > > Thanks for the

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

2023-11-14 Thread Vakaris Baškirov
+1 (non-binding) On Tue, Nov 14, 2023 at 8:03 PM Chao Sun wrote: > +1 > > On Tue, Nov 14, 2023 at 9:52 AM L. C. Hsieh wrote: > > > > +1 > > > > On Tue, Nov 14, 2023 at 9:46 AM Ye Zhou wrote: > > > > > > +1(Non-binding) > > > > > > On Tue, Nov 14, 2023 at 9:42 AM L. C. Hsieh wrote: > > >> > >