[ https://issues.apache.org/jira/browse/SPARK-22006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-22006. ---------------------------------- Resolution: Invalid Please see https://github.com/apache/spark/blob/183d4cb71fbcbf484fc85d8621e1fe04cbbc8195/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L119-L121: {code} // We should cast all relative timestamp/date/string comparison into string comparisons // This behaves as a user would expect because timestamp strings sort lexicographically. // i.e. TimeStamp(2013-01-01 00:00 ...) < "2014" = true {code} I think direct comparison between {{datetime}} and {{date}} is not even allowed in Python itself: {code} >>> import datetime >>> datetime.date(2017, 1, 1) > datetime.datetime(2017, 1, 1) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can't compare datetime.datetime to datetime.date {code} > date/datetime comparisons should avoid casting > ---------------------------------------------- > > Key: SPARK-22006 > URL: https://issues.apache.org/jira/browse/SPARK-22006 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.2.0 > Reporter: Adrian Bridgett > Priority: Minor > Labels: performance > > I believe there's a relatively simple optimisation that can be done here - > comparing timestamps with dates involves a cast whereas comparing with > datetimes avoids this (and pushes the query down into parquet: > {code} > df.filter(df['local_read_at'] > datetime.date('2017-01-01')).count() > {code} > Results in a plan of: > {code} > +- *Filter (isnotnull(local_read_at#324) && (cast(local_read_at#324 > as string) > 2017-01-01)) > +- *FileScan parquet [local_read_at#324] Batched: true, Format: > Parquet, Location: InMemoryFileIndex[s3a://...], PartitionFilters: [], > PushedFilters: [IsNotNull(local_read_at)], ReadSchema: > struct<local_read_at:timestamp> > {code} > Whereas: > {code} > df.filter(df['local_read_at'] > datetime.datetime(2017,1,1)).count() > {code} > Results in: > {code} > +- *Filter (isnotnull(local_read_at#324) && (local_read_at#324 > > 1483228800000000)) > +- *FileScan parquet [local_read_at#324] Batched: true, Format: > Parquet, Location: InMemoryFileIndex[s3a://...], PartitionFilters: [], > PushedFilters: [IsNotNull(local_read_at), > GreaterThan(local_read_at,2017-01-01 00:00:00.0)], ReadSchema: > struct<local_read_at:timestamp> > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org