Partition pruning in spark 1.5.2

2016-04-05 Thread dsing001
HI, I am using 1.5.2. I have a dataframe which is partitioned based on the country. So I have around 150 partition in the dataframe. When I run sparksql and use country = 'UK' it still reads all partitions and not able to prune other partitions. Thus all the queries run for similar times

Plan issue with spark 1.5.2

2016-04-05 Thread dsing001
I am using spark 1.5.2. I have a question regarding plan generated by spark. I have 3 data-frames which has the data for different countries. I have around 150 countries and data is skewed. My 95% queries will have country as criteria. However, I have seen issues with the plans generated for