viirya commented on code in PR #56334:
URL: https://github.com/apache/spark/pull/56334#discussion_r3532947354
##########
sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:
##########
@@ -38,6 +38,50 @@ private[sql] object ArrowUtils {
// todo: support more types.
+ /**
+ * Check if a Spark DataType is supported by Arrow. This recursively checks
complex types
+ * (Array, Struct, Map).
+ *
+ * Note: This checks compatibility with toArrowField(), not toArrowType().
Types like
+ * GeometryType, GeographyType, and VariantType are not supported by
toArrowType() (which only
+ * handles primitive Arrow types), but ARE supported by toArrowField() which
converts them to
+ * Arrow Struct representations with metadata. Since Arrow cache uses
toArrowField() via
+ * toArrowSchema() to create the schema, these types are supported.
+ */
+ def isSupportedByArrow(dt: DataType): Boolean = {
+ dt match {
+ // Primitive types
+ case BooleanType | ByteType | ShortType | IntegerType | LongType |
FloatType | DoubleType |
+ _: StringType | BinaryType | NullType =>
+ true
+
+ // Decimal
+ case _: DecimalType => true
+
+ // Temporal types
+ case DateType | TimestampType | TimestampNTZType | _: TimeType => true
Review Comment:
The prerequisite is now built: SPARK-57975 /
https://github.com/apache/spark/pull/57053 (approved, pending CI) adds an
opt-in lossless Arrow representation for the nanosecond timestamp types -- a
struct of `(epochMicros: int64, nanosWithinMicro: int16)`, i.e.
`TimestampNanosVal`'s own layout with no unit conversion -- covering the full
0001-9999 domain, including the `9999-12-31T23:59:59.999999999` value from your
repro. The interchange mapping stays untouched: its consumers (pandas
`datetime64[ns]` etc.) are themselves int64-bound, so the reduced domain there
is inherent to the destination; the struct is for internal storage only.
Once that lands, this PR will opt in via `losslessTimestampNanos = true` at
the cache's schema-construction sites. The struct is self-describing (tagged
child metadata), so both read paths already work through `ArrowColumnVector`
with no format flag; the stats collector will read the struct children directly
(which also removes the epoch-nanos conversion from the stats path); and the
zero-copy eligibility check will route native int64-nanos Arrow input through
the row-conversion path, like the existing large-var-types handling. That
restores full value-domain parity with the default cache for these types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
