|spark.sql.parquet.filterPushdown| defaults to |false| because there’s a
bug in Parquet which may cause NPE, please refer to
http://spark.apache.org/docs/latest/sql-programming-guide.html#configuration
This bug hasn’t been fixed in Parquet master. We’ll turn this on once
the bug is fixed.
In Spark SQL, Parquet filter pushdown doesn’t cover |HiveTableScan| for
now. May I ask why do you prefer |HiveTableScan| rather than
|ParquetTableScan|?
Cheng
On 1/19/15 5:02 PM, Xiaoyu Wang wrote:
The *spark.sql.parquet.**filterPushdown=true *has been turned on. But
set
The *spark.sql.parquet.**filterPushdown=true *has been turned on. But set
*spark.sql.hive.**convertMetastoreParquet *to *false*. the first parameter
is lose efficacy!!!
2015-01-20 6:52 GMT+08:00 Yana Kadiyska yana.kadiy...@gmail.com:
If you're talking about filter pushdowns for parquet files
If you're talking about filter pushdowns for parquet files this also has to
be turned on explicitly. Try *spark.sql.parquet.**filterPushdown=true . *It's
off by default
On Mon, Jan 19, 2015 at 3:46 AM, Xiaoyu Wang wangxy...@gmail.com wrote:
Yes it works!
But the filter can't pushdown!!!
If
Yes it works!
But the filter can't pushdown!!!
If custom parquetinputformat only implement the datasource API?
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala
2015-01-16 21:51 GMT+08:00 Xiaoyu Wang wangxy...@gmail.com:
Thanks
I think you might need to set
spark.sql.hive.convertMetastoreParquet to false if I understand that flag
correctly
Sent on the new Sprint Network from my Samsung Galaxy S®4.
div Original message /divdivFrom: Xiaoyu Wang
wangxy...@gmail.com /divdivDate:01/16/2015 5:09 AM
Thanks yana!
I will try it!
在 2015年1月16日，20:51，yana yana.kadiy...@gmail.com
mailto:yana.kadiy...@gmail.com 写道：
I think you might need to set
spark.sql.hive.convertMetastoreParquet to false if I understand that flag
correctly
Sent on the new Sprint Network from my Samsung Galaxy S®4.
