For Apache Spark a stand-alone worker can manage all the resources of the box, including all GPUs. So a spark worker could be set up to manage N gpus in the box via *spark.worker.resource.gpu.amount,* and then *spark.executor.resource.gpu.amount, *as provided by on app submit, assigns GPU resources to executors as they come up. Here is a getting started guide for spark-rapids but I am not sure if that's what you are looking to use. Either way, it may help with the resource setup: https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-on-prem.html#spark-standalone-cluster .
Not every node in the cluster needs to have GPUs. You could request 0 GPUs for an app (default value of spark.executor.resource.gpu.amount), and the executors will not require this resource. If you are using a yarn/k8s cluster there are other configs to pay attention to. If you need help with those let us know. On Sun, Feb 5, 2023 at 1:50 PM Jack Goodson <jackagood...@gmail.com> wrote: > As far as I understand you will need a GPU for each worker node or you > will need to partition the GPU processing somehow to each node which I > think would defeat the purpose. In Databricks for example when you select > GPU workers there is a GPU allocated to each worker. I assume this is the > “correct” approach to this problem > > On Mon, 6 Feb 2023 at 8:17 AM, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> if you have several nodes with only one node having GPUs, you still have >> to wait for the result set to complete. In other words it will be as fast >> as the lowest denominator .. >> >> my postulation >> >> HTH >> >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Sun, 5 Feb 2023 at 13:38, Irene Markelic <ir...@markelic.de> wrote: >> >>> Hello, >>> >>> has anyone used spark with GPUs? I wonder if every worker node in a >>> cluster needs one GPU or if you can have several worker nodes of which >>> only one has a GPU. >>> >>> Thank you! >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>>
