Extending spark connectors versus providing utility libraries

Hi team, Had a design question around wether it’s a good idea to write wrappers over all existing spark connectors for adding some functionality/improving usability in terms of options passed to the connector. In contrast to providing utility libraries that takes parameters and calls the

Extending Spark Connector versus Providing utility library

Hi team, Had a design question around wether it’s a good idea to write wrappers over all existing spark connectors for adding some functionality/improving usability in terms of options passed to the connector. In contrast to providing utility libraries that takes parameters and calls the

Re: Topics for Spark online classes & webinars

Well that needs to be created first for this purpose. The appropriate name etc. to be decided. Maybe @Denny Lee can facilitate this as he offered his help. cheers view my Linkedin profile

Re: Topics for Spark online classes & webinars

Hello Mich, Can you please provide the link for the confluence page? Many thanks Asma Ph.D. in Big Data - Applied Machine Learning Le lun. 13 mars 2023 à 17:21, Mich Talebzadeh a écrit : > Apologies I missed the list. > > To move forward I selected these topics from the thread "Online classes

Re: Topics for Spark online classes & webinars

Apologies I missed the list. To move forward I selected these topics from the thread "Online classes for spark topics". To take this further I propose a confluence page to be seup. 1. Spark UI 2. Dynamic allocation 3. Tuning of jobs 4. Collecting spark metrics for monitoring and

Topics for Spark online classes & webinars

Hi guys To move forward I selected these topics from the thread "Online classes for spark topics". To take this further I propose a confluence page to be seup. Opinions and how to is welcome Cheers view my Linkedin profile

Re: org.apache.spark.shuffle.FetchFailedException in dataproc

Hi Gary Thanks for the update. So this serverless dataproc. on 3.3.1. Maybe an autoscaling policy could be an option. What is y-axis? Is that the capacity? Can you break down the join into multiple parts and save the intermediate result set? HTH view my Linkedin profile

Re: org.apache.spark.shuffle.FetchFailedException in dataproc

Hi Mich, I used the serverless spark session, not the local mode in the notebook. So machine type does not matter in this case. Below is the chart for serverless spark session execution. I also tried to increase executor memory and core, but the issue did got get resolved. I will try shutting down

unsubscribe

unsubscribe - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

unsubscribe