Hi, completely agree with Hao. In case you are using YARN try to see the EMR documentation on how to enable GPU as resource in YARN before trying to use that in SPARK.
This is one of the most exciting features of SPARK 3, and you can reap huge benefits out of it :) Regards, Gourav Sengupta On Fri, Apr 9, 2021 at 6:10 PM HaoZ <ha...@nvidia.com> wrote: > Hi Martin, > > I tested the local mode in Spark on Rapids Accelerator and it works fine > for > me. > The only possible issue is the CUDA 11.2 however the supported CUDA version > as per https://nvidia.github.io/spark-rapids/docs/download.html is 11.0. > > Here is a quick test using Spark local mode. > Note: When I was testing this local mode, I make sure there is nothing in > spark-defaults.conf so everything is clean. > > ====== > scala> val df = sc.makeRDD(1 to 100, 6).toDF > df: org.apache.spark.sql.DataFrame = [value: int] > > scala> val df2 = sc.makeRDD(1 to 100, 6).toDF > df2: org.apache.spark.sql.DataFrame = [value: int] > > scala> df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" > === $"b").count > res0: Long = 100 > scala> df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" > === $"b").explain() > == Physical Plan == > GpuColumnarToRow false > +- GpuShuffledHashJoin [a#29], [b#31], Inner, GpuBuildRight, false > :- GpuShuffleCoalesce 2147483647 > : +- GpuColumnarExchange gpuhashpartitioning(a#29, 10), > ENSURE_REQUIREMENTS, [id=#221] > : +- GpuProject [value#2 AS a#29] > : +- GpuRowToColumnar TargetSize(2147483647) > : +- *(1) SerializeFromObject [input[0, int, false] AS > value#2] > : +- Scan[obj#1] > +- GpuCoalesceBatches RequireSingleBatch > +- GpuShuffleCoalesce 2147483647 > +- GpuColumnarExchange gpuhashpartitioning(b#31, 10), > ENSURE_REQUIREMENTS, [id=#228] > +- GpuProject [value#8 AS b#31] > +- GpuRowToColumnar TargetSize(2147483647) > +- *(2) SerializeFromObject [input[0, int, false] AS > value#8] > +- Scan[obj#7] > ====== > > Thanks, > Hao > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
