Re: Spike on number of tasks - dynamic allocation

Hi, What is the spark version and what type of cluster is it, spark on dataproc or other? HTH view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all

Spike on number of tasks - dynamic allocation

On an auto-scaling cluster using YARN as resource manager, we observed that when we decrease the number of worker nodes after upscaling instance types, the number of tasks for the same spark job spikes. (the total cpu/memory capacity of the cluster remains identical) the same spark job, with the

Re: [New Project] sparksql-ml : Distributed Machine Learning using SparkSQL.

I think it is awesome. Brilliant interface that is missing from Spark. Would you integrate with something like MLFlow? Thanks, Russell Jurney @rjurney russell.jur...@gmail.com LI FB datasyndrome.com

Re: Spike on number of tasks - dynamic allocation

Hey Mich, This cluster is running spark 2.4.6 on EMR On Mon, Feb 27, 2023 at 12:20 PM Mich Talebzadeh wrote: > Hi, > > What is the spark version and what type of cluster is it, spark on > dataproc or other? > > HTH > > > >view my Linkedin profile >

Fwd: [New Project] sparksql-ml : Distributed Machine Learning using SparkSQL.

Hi All, I worked on this idea a few years back as a pet project to bridge *SparkSQL* and *SparkML* and empower anyone to implement production grade, distributed machine learning over Apache Spark as long as they have SQL skills. In principle the idea works exactly like Google's BigQueryML but at

Re: Spike on number of tasks - dynamic allocation

Hi Murat, I have dealt with EMR but have used Spark cluster on Google Dataproc with 3.1.1 with autoscaling policy. My understanding is that autoscaling policy will decide on how to scale if needed without manual intervention. Is this the case with yours? HTH view my Linkedin profile