Hey everyone,
Consider the following use of spark.sql.shuffle.partitions:
case class Data(A:String = f"${(math.random*1e8).toLong}%09.0f", B: String
= f"${(math.random*1e8).toLong}%09.0f")
val dataFrame = (1 to 1000).map(_ => Data()).toDF
dataFrame.registerTempTable("data")
sqlContext.setConf( "spark.sql.shuffle.partitions", "10")
val a = sqlContext.sql("SELECT * FROM data CLUSTER BY A")
sqlContext.setConf( "spark.sql.shuffle.partitions", "20")
val b = sqlContext.sql("SELECT * FROM data CLUSTER BY A")
a.rdd.partitions.size
b.rdd.partitions.size
Expected result:
10
20
Actual result:
20
20
The bad thing about this is that the way the DataFrame is evaluated
currently depends on the global state. Multiple actions on the same
DataFrame may hence behave nondeterministically.
Furthermore, imagine a user has a job that includes multiple shuffles each
of which might require different settings. It is very hard to determine
(due to lazyness) at which place the configuration is read and hence to
place the setConf call.
Instead I propose to ensure that all relevant configuration parameters are
stored "in the lineage" at the place where the DataFrame is created so that
every call to that DataFrame at a later point will deterministically use
the context that the DataFrame was initially created in.
Best regards,
Daniel
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-SQL-parameters-like-shuffle-partitions-should-be-stored-in-the-lineage-tp13240.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
