ump45nose commented on code in PR #7227:
URL: https://github.com/apache/datafusion-comet/pull/7227#discussion_r4381640320
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -678,13 +677,45 @@ fn spark_has_updater(
return false;
}
+ // A timestamp leaf is INT96, or INT64 annotated as a timestamp. Spark's
INT96 updaters
+ // produce only timestamps, and its INT64 updaters produce a timestamp or
a long (a decimal
+ // needs a decimal annotation), so a timestamp read as a string, a
narrower integer, a float
+ // or a decimal has no updater. Without this, the cast returns text, raw
numbers or NULL.
+ // See #7214.
+ if matches!(physical_type, DataType::Timestamp(_, _))
+ && (is_string_or_binary(target_type)
+ || matches!(
+ target_type,
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Float32
+ | DataType::Float64
+ | DataType::Decimal128(_, _)
+ | DataType::Decimal256(_, _)
+ ))
+ {
+ return false;
Review Comment:
Thanks, folded in with cb442c8b. The separate block is gone: `Date32` and
`Timestamp` joined the list for the numeric-to-string rule, and the
`(Timestamp, Int8 | Int16 | Int32 | Float32 | Float64 | Decimal*)` pairs joined
the `is_spark_rejected_conversion` table. I also covered the DATE gap there
with `(Date32, Float32 | Decimal*)`, and the string half comes from the same
list. The new `parquet_date_read_as_string_float_or_decimal_errors` test reads
a DATE column as `string`, `float` and `decimal(20,0)` and gets the
Spark-shaped error; `int` is still accepted. The INT96 `bigint` rule stays
separate since it needs the footer marker. The `parquet::` lib tests pass (332)
and clippy is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
