We are working to improve the integration here, but I can recommend the
following when running spark 1.1:  create an external table and
set spark.sql.hive.convertMetastoreParquet=true

Note that even with a HiveContext we don't support window functions yet.


On Wed, Oct 8, 2014 at 10:41 AM, Anand Mohan <[email protected]> wrote:

> We have our analytics infra built on Spark and Parquet.
> We are trying to replace some of our queries based on the direct Spark RDD
> API to SQL based either on Spark SQL/HiveQL.
> Our motivation was to take advantage of the transparent projection &
> predicate pushdown that's offered by Spark SQL and eliminate the need for 
> persisting
> the RDD in memory. (Cache invalidation turned out to be a big problem for
> us)
>
> The below tests are done with Spark 1.1.0 on CDH 5.1.0
>
>
> 1. Spark SQL's (SQLContext) Parquet support was excellent for our case.
> The ability to query in SQL and apply scala functions as UDFs in the SQL is
> extremely convenient. Project pushdown works flawlessly, not much sure
> about predicate pushdown
> (we have 90% optional fields in our dataset and I remember Michael
> Armbrust telling me that this is a bug in Parquet in that it doesnt allow
> predicate pushdown for optional fields.)
> However we have timestamp based duplicate removal which requires windowing
> queries which are not working in SQLContext.sql parsing mode.
>
> 2. We then tried HiveQL using HiveContext by creating a Hive external
> table backed by the same Parquet data. However, in this mode, projection
> pushdown doesnt seem to work and it ends up reading the whole Parquet data
> for each query.(which slows down a lot)
> Please see attached the screenshot of this.
> Hive itself doesnt seem to have any issues with the projection pushdown.
> So this is weird. Is this due to any configuration problem?
>
> Thanks in advance,
> Anand Mohan
>
> *SparkSQLHiveParquet.png* (316K) Download Attachment
> <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/15953/0/SparkSQLHiveParquet.png>
>
> ------------------------------
> View this message in context: Spark SQL HiveContext Projection Pushdown
> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-HiveContext-Projection-Pushdown-tp15953.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>

Reply via email to