I was able to resolve this use case (Thanks Cheng Lian) where I wanted to
launch executor on just the specific partition while also getting the batch
pruning optimisations of Spark SQL by doing following :-
val query = sql("SELECT * FROM cac
hedTable WHERE key = 1")
val plannedRDD = query.queryExecution.toRdd
val prunedRDD = PartitionPruningRDD.create(plannedRDD, _ == 3)
prunedRDD.collect()
Thanks a lot Cheng for suggesting the approach to do things other way round.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Point-lookup-optimisation-in-SchemaRDD-tp21555p21613.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
