dongjoon-hyun commented on code in PR #59699:
URL: https://github.com/apache/spark/pull/59699#discussion_r4348448496
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala:
##########
@@ -333,6 +338,69 @@ class OrcFilterSuite extends OrcTest with
SharedSparkSession {
}
}
+ test("SPARK-60449: filter pushdown - timestamp with sub-microsecond digits")
{
+ // Another engine (e.g. Hive) can write sub-microsecond digits to an ORC
timestamp column, and
+ // Spark reads the column as TimestampType by truncating every value to
microseconds. So the
+ // pushed down predicates must not skip the stripes whose truncated values
satisfy them.
+ withTempPath { dir =>
+ val schema = TypeDescription.fromString("struct<ts:timestamp>")
+ Seq(
+ "part-0.orc" -> Seq("2020-05-25 10:11:12.123456789", null),
Review Comment:
Thank you. In 246b3c941a8, I added `2020-05-25 10:11:12.123456999` and
`2020-05-25 10:11:12.123457` as the only values of their own files, with `=`,
`<=`, `>`, and `IN` expectations and `stripSparkFilter` checks for `=` and `>`.
Now the test fails when the offset is changed to `+ 800` or `+ 1000`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
