[
https://issues.apache.org/jira/browse/SPARK-22006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-22006.
----------------------------------
Resolution: Invalid
Please see
https://github.com/apache/spark/blob/183d4cb71fbcbf484fc85d8621e1fe04cbbc8195/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L119-L121:
{code}
// We should cast all relative timestamp/date/string comparison into string
comparisons
// This behaves as a user would expect because timestamp strings sort
lexicographically.
// i.e. TimeStamp(2013-01-01 00:00 ...) < "2014" = true
{code}
I think direct comparison between {{datetime}} and {{date}} is not even allowed
in Python itself:
{code}
>>> import datetime
>>> datetime.date(2017, 1, 1) > datetime.datetime(2017, 1, 1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't compare datetime.datetime to datetime.date
{code}
> date/datetime comparisons should avoid casting
> ----------------------------------------------
>
> Key: SPARK-22006
> URL: https://issues.apache.org/jira/browse/SPARK-22006
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Adrian Bridgett
> Priority: Minor
> Labels: performance
>
> I believe there's a relatively simple optimisation that can be done here -
> comparing timestamps with dates involves a cast whereas comparing with
> datetimes avoids this (and pushes the query down into parquet:
> {code}
> df.filter(df['local_read_at'] > datetime.date('2017-01-01')).count()
> {code}
> Results in a plan of:
> {code}
> +- *Filter (isnotnull(local_read_at#324) && (cast(local_read_at#324
> as string) > 2017-01-01))
> +- *FileScan parquet [local_read_at#324] Batched: true, Format:
> Parquet, Location: InMemoryFileIndex[s3a://...], PartitionFilters: [],
> PushedFilters: [IsNotNull(local_read_at)], ReadSchema:
> struct<local_read_at:timestamp>
> {code}
> Whereas:
> {code}
> df.filter(df['local_read_at'] > datetime.datetime(2017,1,1)).count()
> {code}
> Results in:
> {code}
> +- *Filter (isnotnull(local_read_at#324) && (local_read_at#324 >
> 1483228800000000))
> +- *FileScan parquet [local_read_at#324] Batched: true, Format:
> Parquet, Location: InMemoryFileIndex[s3a://...], PartitionFilters: [],
> PushedFilters: [IsNotNull(local_read_at),
> GreaterThan(local_read_at,2017-01-01 00:00:00.0)], ReadSchema:
> struct<local_read_at:timestamp>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
