[
https://issues.apache.org/jira/browse/SPARK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-6922.
------------------------------
Resolution: Duplicate
Yes, a likely subset of SPARK-6307. I'm closing this since I don't see it's
likely that there will be separate activity here.
> RDD.cartesian is much slower than join
> --------------------------------------
>
> Key: SPARK-6922
> URL: https://issues.apache.org/jira/browse/SPARK-6922
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Environment: Ubuntu 12.04.5, Spark 1.3.0 CDH 4 binary, standalone
> Reporter: David Tolnay
>
> Cartesian takes 3 minutes to join 500x500 partitions. Join with a constant
> key takes only 4 seconds. Here is a deterministic repro:
> {code}
> val lst = List.fill(500)(Tuple1(0))
> val df = sqlContext.createDataFrame(lst).repartition(500)
> df.select($"_1".as("a")).saveAsParquetFile("file:///tmp/parquet/left")
> df.select($"_1".as("b")).saveAsParquetFile("file:///tmp/parquet/right")
> val left = sqlContext.parquetFile("file:///tmp/parquet/left")
> val right = sqlContext.parquetFile("file:///tmp/parquet/right")
> def time[A](f: => A) = {
> val start = System.nanoTime
> f
> (System.nanoTime-start)/1e6
> }
> time { left.rdd.cartesian(right.rdd).count } // 3 minutes
> time { left.rdd.keyBy(_=>0).join(right.rdd.keyBy(_=>0)).count } // 4 seconds
> {code}
> Possibly related to SPARK-6307 in which cartesian causes the block manager to
> fetch the same blocks over and over.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
