Re: Spark with GPU

For Apache Spark a stand-alone worker can manage all the resources of the box, including all GPUs. So a spark worker could be set up to manage N gpus in the box via *spark.worker.resource.gpu.amount,* and then *spark.executor.resource.gpu.amount, *as provided by on app submit, assigns GPU

Re: Spark with GPU

As far as I understand you will need a GPU for each worker node or you will need to partition the GPU processing somehow to each node which I think would defeat the purpose. In Databricks for example when you select GPU workers there is a GPU allocated to each worker. I assume this is the

Re: Spark with GPU

if you have several nodes with only one node having GPUs, you still have to wait for the result set to complete. In other words it will be as fast as the lowest denominator .. my postulation HTH view my Linkedin profile

Spark with GPU

Hello, has anyone used spark with GPUs? I wonder if every worker node in a cluster needs one GPU or if you can have several worker nodes of which only one has a GPU. Thank you! - To unsubscribe e-mail:

Error - using Spark with GPU

spark-submit /home/mwadmin/Documents/test.py 22/11/30 14:59:32 WARN Utils: Your hostname, mwadmin-HP-Z440-Workstation resolves to a loopback address: 127.0.1.1; using ***.***.**.** instead (on interface eno1) 22/11/30 14:59:32 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

Re: Spark with GPU

One of the best things that could have happened to SPARK (now mostly an overhyped ETL tool with small incremental optimisation changes and no large scale innovation) is the release by NVIDIA for GPU processing. You need some time to get your head around it, but it is supported quite easily in AWS

Re: Spark with GPU

This thread may be better suited as a discussion in our Spark plug-in’s repo: https://github.com/NVIDIA/spark-rapids/discussions. Just to answer the questions that were asked so far: I would recommend checking our documentation for what is supported as of our latest release (22.06):

Re: Spark with GPU

This isn't a Spark question, but rather a question about whatever Spark application you are talking about. RAPIDS? On Sat, Aug 13, 2022 at 10:35 AM rajat kumar wrote: > Thanks Sean. > > Also, I observed that lots of things are not supported in GPU by NVIDIA. > E.g. nested types/decimal

Re: Spark with GPU

Thanks Sean. Also, I observed that lots of things are not supported in GPU by NVIDIA. E.g. nested types/decimal type/Udfs etc. So, will it use CPU automatically for running those tasks which require nested types or will it run on GPU and fail. Thanks Rajat On Sat, Aug 13, 2022, 18:54 Sean Owen

Re: Spark with GPU

Spark does not use GPUs itself, but tasks you run on Spark can. The only 'support' there is is for requesting GPUs as resources for tasks, so it's just a question of resource management. That's in OSS. On Sat, Aug 13, 2022 at 8:16 AM rajat kumar wrote: > Hello, > > I have been hearing about GPU

Spark with GPU

Hello, I have been hearing about GPU in spark3. For batch jobs , will it help to improve GPU performance. Also is GPU support available only on Databricks or on cloud based Spark clusters ? I am new , if anyone can share insight , it will help Thanks Rajat