hi,
I have configured the GPU scheduling for spark-3.0.0 on yarn following the
official document ,but the job seems not runing with GPU . Do I need to
modify my code to invoke CUDA ? Is there any tutorial can be shared ?
running logs:
...
2020-06-13 10:58:01,938 INFO spark.SparkContext: Running Spark version
3.0.0-preview2
2020-06-13 10:58:04,101 INFO resource.ResourceUtils:
==============================================================
2020-06-13 10:58:04,105 INFO resource.ResourceUtils: Resources for
spark.driver:
gpu -> [name: gpu, addresses: 0]
spark-default.conf:
...
spark.executor.resource.gpu.amount 1
spark.worker.resource.gpu.amount 1
spark.driver.resource.gpu.amount 1
spark.driver.resource.gpu.discoveryScript
/usr/local/spark-3.0.0/examples/src/main/scripts/getGpusResources.sh
spark.worker.resource.gpu.discoveryScript
/usr/local/spark-3.0.0/examples/src/main/scripts/getGpusResources.sh
nodemanager log:
...
2020-06-13 10:55:07,702 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.ResourcePluginManager:
Found Resource plugins from configuration: [yarn.io/gpu]
2020-06-13 10:55:07,745 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer:
Trying to discover GPU information ...
2020-06-13 10:55:10,601 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer:
Discovered GPU information: === GPUs in the system ===
Driver Version:440.82
ProductName=GeForce GTX 950M, MinorNumber=0, TotalMemory=2004MiB,
Utilization=2.0%
Thanks
charles
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
