This is related: SPARK-11087 FYI
On Mon, Nov 30, 2015 at 5:56 PM, Tejas Patil <tejas.patil...@gmail.com> wrote: > Hi, > I am trying to use predicate pushdown in ORC and I was expecting that it > would be used when one tries to query an ORC table in Hive. But based on > the query plan generated, I don't see that happening. I am missing some > configurations or this is not supported at all. > > PS: I have tried the following over 1.5.1 and even 1.6 release branch. > > val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) > hiveContext.setConf("spark.sql.orc.filterPushdown", "true") > // orc_table is a hive table > val query = hiveContext.sql("SELECT COUNT(*) FROM orc_table WHERE key > > 10") > > scala> query.explain > == Physical Plan == > TungstenAggregate(key=[], > functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#41L]) > TungstenExchange SinglePartition > TungstenAggregate(key=[], > functions=[(count(1),mode=Partial,isDistinct=false)], > output=[currentCount#44L]) > Project > Filter (key#39 > 10) > HiveTableScan [key#39], (MetastoreRelation default, orc_table, None) > > > Thanks, > Tejas > >