[jira] [Commented] (SPARK-22913) Hive Partition Pruning, Fractional and Timestamp types
[ https://issues.apache.org/jira/browse/SPARK-22913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305018#comment-16305018 ] Ameen Tayyebi commented on SPARK-22913: --- Note: I'll be away since December 30th until January 18th so I'll be checking up on this issue and the pull request at that time. > Hive Partition Pruning, Fractional and Timestamp types > -- > > Key: SPARK-22913 > URL: https://issues.apache.org/jira/browse/SPARK-22913 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ameen Tayyebi > Fix For: 2.3.0 > > > Spark currently pushes the predicates it has in the SQL query to Hive > Metastore. This only applies to predicates that are placed on top of > partitioning columns. As more and more hive metastore implementations come > around, this is an important optimization to allow data to be prefiltered to > only relevant partitions. Consider the following example: > Table: > create external table data (key string, quantity long) > partitioned by (processing-date timestamp) > Query: > select * from data where processing-date = '2017-10-23 00:00:00' > Currently, no filters will be pushed to the hive metastore for the above > query. The reason is that the code that tries to compute predicates to be > sent to hive metastore, only deals with integral and string column types. It > doesn't know how to handle fractional and timestamp columns. > I have tables in my metastore (AWS Glue) with millions of partitions of type > timestamp and double. In my specific case, it takes Spark's master node about > 6.5 minutes to download all partitions for the table, and then filter the > partitions client-side. The actual processing time of my query is only 6 > seconds. In other words, without partition pruning, I'm looking at 6.5 > minutes of processing and with partition pruning, I'm looking at 6 seconds > only. > I have a fix for this developed locally that I'll provide shortly as a pull > request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22913) Hive Partition Pruning, Fractional and Timestamp types
[ https://issues.apache.org/jira/browse/SPARK-22913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305017#comment-16305017 ] Apache Spark commented on SPARK-22913: -- User 'ameent' has created a pull request for this issue: https://github.com/apache/spark/pull/20100 > Hive Partition Pruning, Fractional and Timestamp types > -- > > Key: SPARK-22913 > URL: https://issues.apache.org/jira/browse/SPARK-22913 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ameen Tayyebi > Fix For: 2.3.0 > > > Spark currently pushes the predicates it has in the SQL query to Hive > Metastore. This only applies to predicates that are placed on top of > partitioning columns. As more and more hive metastore implementations come > around, this is an important optimization to allow data to be prefiltered to > only relevant partitions. Consider the following example: > Table: > create external table data (key string, quantity long) > partitioned by (processing-date timestamp) > Query: > select * from data where processing-date = '2017-10-23 00:00:00' > Currently, no filters will be pushed to the hive metastore for the above > query. The reason is that the code that tries to compute predicates to be > sent to hive metastore, only deals with integral and string column types. It > doesn't know how to handle fractional and timestamp columns. > I have tables in my metastore (AWS Glue) with millions of partitions of type > timestamp and double. In my specific case, it takes Spark's master node about > 6.5 minutes to download all partitions for the table, and then filter the > partitions client-side. The actual processing time of my query is only 6 > seconds. In other words, without partition pruning, I'm looking at 6.5 > minutes of processing and with partition pruning, I'm looking at 6 seconds > only. > I have a fix for this developed locally that I'll provide shortly as a pull > request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
