How to use Fair Scheduler Pools

I have two questions to ask: I wrote a demo referring to the official website(https://spark.apache.org/docs/latest/job-scheduling.html), but it didn't meet my expectations. I don't know if there was a problem with my writing.I hope that when I use the following fairscheduler.xml, pool1

Re: Online classes for spark topics

I can prepare some topics and present as well , if we have a prioritised list of topics already . On Thu, 9 Mar 2023 at 11:42 AM, Denny Lee wrote: > We used to run Spark webinars on the Apache Spark LinkedIn group > but >

Re: Online classes for spark topics

We used to run Spark webinars on the Apache Spark LinkedIn group but honestly the turnout was pretty low. We had dove into various features. If there are particular topics that. you would like to discuss during a live session,

Re: Online classes for spark topics

+1 On Wed, Mar 8, 2023 at 10:40 PM Winston Lai wrote: > +1, any webinar on Spark related topic is appreciated  > > Thank You & Best Regards > Winston Lai > -- > *From:* asma zgolli > *Sent:* Thursday, March 9, 2023 5:43:06 AM > *To:* karan alang > *Cc:* Mich

Spark Thrift Server - Autoscaling on K8

Hi All, We are in the process of moving our workloads to K8 and looking for some guidance to run Spark Thrift Server on K8. We need the executor pods to autoscale based on the workload vs running it with a static number of executors. If any one has done it and can share the details, it will be

How to use Fair Scheduler Pools

I have two questions to ask: I wrote a demo referring to the official website(https://spark.apache.org/docs/latest/job-scheduling.html), but it didn't meet my expectations. I don't know if there was a problem with my writing.I hope that when I use the following fairscheduler.xml, pool1

[Spark] How to find which type of key is illegal during from_json() function

*Component*: Spark *Level*: Advanced *Scenario*: How-to - *Problems Description* I have nested json string value in someone field of spark dataframe, and I would like to use from_json() to parse json object. Especially, if one of key type is not match with our defined

Re: Online classes for spark topics

+1, any webinar on Spark related topic is appreciated  Thank You & Best Regards Winston Lai From: asma zgolli Sent: Thursday, March 9, 2023 5:43:06 AM To: karan alang Cc: Mich Talebzadeh ; ashok34...@yahoo.com ; User Subject: Re: Online classes for spark

spark-submit: No "driver-" id printed in standalone mode

Hello, I'm trying to get Airflow to work with spark in cluster mode. I can successfully submit jobs via spark-submit and see them complete successfully. However, 'spark-submit' doesn't seem to print any driver- ID to the console. Clearly the drivers have an ID, as they are listed with one in the

Re: Online classes for spark topics

+1 Le mer. 8 mars 2023 à 21:32, karan alang a écrit : > +1 .. I'm happy to be part of these discussions as well ! > > > > > On Wed, Mar 8, 2023 at 12:27 PM Mich Talebzadeh > wrote: > >> Hi, >> >> I guess I can schedule this work over a course of time. I for myself can >> contribute plus learn

Re: Online classes for spark topics

+1 .. I'm happy to be part of these discussions as well ! On Wed, Mar 8, 2023 at 12:27 PM Mich Talebzadeh wrote: > Hi, > > I guess I can schedule this work over a course of time. I for myself can > contribute plus learn from others. > > So +1 for me. > > Let us see if anyone else is

Re: Online classes for spark topics

Hi, I guess I can schedule this work over a course of time. I for myself can contribute plus learn from others. So +1 for me. Let us see if anyone else is interested. HTH view my Linkedin profile

Re: Online classes for spark topics

Hello Mich. Greetings. Would you be able to arrange for Spark Structured Streaming learning webinar.? This is something I haven been struggling with recently. it will be very helpful. Thanks and Regard AKOn Tuesday, 7 March 2023 at 20:24:36 GMT, Mich Talebzadeh wrote: Hi, This might