"So if I am
going to use GPU in my job running on the spark , I still need to code the
map and reduce function in cuda or in c++ and then invoke them throught jni
or something like GPUEnabler , is that right ?"
Sort of. You could go through all of that work yourself, or you could use
the plugin
Bobby
Thanks for your answer, it seems that I have misunderstood this paragraph in
the website : *"GPU-accelerate your Apache Spark 3.0 data science
pipelines—without code changes—and speed up data processing and model
training while substantially lowering infrastructure costs."* . So if I am
Charles,
I am sorry that you got the idea that Apache Spark is GPU accelerated out
of the box. Where did you get that information so we can try to make it
more clear? Apache Spark 3.0 opens up a set of plugin APIs that allow for
a plugin to provide GPU acceleration. You can look at SPARK-27396
hi,
I have configured the GPU scheduling for spark-3.0.0 on yarn following the
official document ,but the job seems not runing with GPU . Do I need to
modify my code to invoke CUDA ? Is there any tutorial can be shared ?
running logs:
...
2020-06-13 10:58:01,938 INFO spark.SparkContext: Running
