Johannes Mayer created SPARK-27956:
--------------------------------------
Summary: Allow subqueries as partition filter
Key: SPARK-27956
URL: https://issues.apache.org/jira/browse/SPARK-27956
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.3.0
Reporter: Johannes Mayer
Subqueries are not pushed down as partition filters. See following example
{code:java}
create table user_mayerjoh.tab (c1 string)
partitioned by (c2 string)
stored as parquet;
{code}
{code:java}
explain select * from user_mayerjoh.tab where c2 < 1;{code}
== Physical Plan ==
*(1) FileScan parquet user_mayerjoh.tab[c1#22,c2#23] Batched: true, Format:
Parquet, Location: PrunedInMemoryFileIndex[], PartitionCount: 0,
*PartitionFilters: [isnotnull(c2#23), (cast(c2#23 as int) < 1)]*,
PushedFilters: [], ReadSchema: struct<c1:string>
{code:java}
explain select * from user_mayerjoh.tab where c2 < (select 1);{code}
== Physical Plan ==
+- *(1) FileScan parquet user_mayerjoh.tab[c1#30,c2#31] Batched: true, Format:
Parquet, Location: PrunedInMemoryFileIndex[], PartitionCount: 0,
*PartitionFilters: [isnotnull(c2#31)]*, PushedFilters: [], ReadSchema:
struct<c1:string>
Is it possible to first execute the subquery and use the result as partition
filter?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
