[ https://issues.apache.org/jira/browse/SPARK-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128451#comment-14128451 ]
Cody Koeninger commented on SPARK-3462: --------------------------------------- Created a PR for feedback. https://github.com/apache/spark/pull/2345 Seems to do the right thing locally, will see about testing on a cluster > parquet pushdown for unionAll > ----------------------------- > > Key: SPARK-3462 > URL: https://issues.apache.org/jira/browse/SPARK-3462 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 1.1.0 > Reporter: Cody Koeninger > > http://apache-spark-developers-list.1001551.n3.nabble.com/parquet-predicate-projection-pushdown-into-unionAll-td8339.html > // single table, pushdown > scala> p.where('age < 40).select('name) > res36: org.apache.spark.sql.SchemaRDD = > SchemaRDD[97] at RDD at SchemaRDD.scala:103 > == Query Plan == > == Physical Plan == > Project [name#3] > ParquetTableScan [name#3,age#4], (ParquetRelation /var/tmp/people, > Some(Configuration: core-default.xml, core-site.xml, mapred-default.xml, > mapred-site.xml), org.apache.spark.sql.SQLContext@6d7e79f6, []), [(age#4 < > 40)] > // union of 2 tables, no pushdown > scala> b.where('age < 40).select('name) > res37: org.apache.spark.sql.SchemaRDD = > SchemaRDD[99] at RDD at SchemaRDD.scala:103 > == Query Plan == > == Physical Plan == > Project [name#3] > Filter (age#4 < 40) > Union [ParquetTableScan [name#3,age#4,phones#5], (ParquetRelation > /var/tmp/people, Some(Configuration: core-default.xml, core-site.xml, > mapred-default.xml, mapred-site.xml), > org.apache.spark.sql.SQLContext@6d7e79f6, []), [] > ,ParquetTableScan [name#0,age#1,phones#2], (ParquetRelation /var/tmp/people2, > Some(Configuration: core-default.xml, core-site.xml, mapred-default.xml, > mapred-site.xml), org.apache.spark.sql.SQLContext@6d7e79f6, []), [] > ] -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org