Can you show your full code. How are you partitioning the data? How are you reading it? What is the resulting query plan (run explain() or EXPLAIN).
On Tue, Apr 5, 2016 at 10:02 AM, dsing001 <darshan.m...@gmail.com> wrote: > HI, > > I am using 1.5.2. I have a dataframe which is partitioned based on the > country. So I have around 150 partition in the dataframe. When I run > sparksql and use country = 'UK' it still reads all partitions and not able > to prune other partitions. Thus all the queries run for similar times > independent of what country I pass. Is it desired? > > Is there a way to fix this in 1.5.2 by using some parameter or is it fixed > in latest versions? > > Thanks > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Partition-pruning-in-spark-1-5-2-tp26682.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
