viirya commented on code in PR #56334:
URL: https://github.com/apache/spark/pull/56334#discussion_r3532948093
##########
sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:
##########
@@ -38,6 +38,50 @@ private[sql] object ArrowUtils {
// todo: support more types.
+ /**
+ * Check if a Spark DataType is supported by Arrow. This recursively checks
complex types
+ * (Array, Struct, Map).
+ *
+ * Note: This checks compatibility with toArrowField(), not toArrowType().
Types like
+ * GeometryType, GeographyType, and VariantType are not supported by
toArrowType() (which only
+ * handles primitive Arrow types), but ARE supported by toArrowField() which
converts them to
+ * Arrow Struct representations with metadata. Since Arrow cache uses
toArrowField() via
+ * toArrowSchema() to create the schema, these types are supported.
+ */
+ def isSupportedByArrow(dt: DataType): Boolean = {
+ dt match {
+ // Primitive types
+ case BooleanType | ByteType | ShortType | IntegerType | LongType |
FloatType | DoubleType |
+ _: StringType | BinaryType | NullType =>
+ true
+
+ // Decimal
+ case _: DecimalType => true
+
+ // Temporal types
+ case DateType | TimestampType | TimestampNTZType | _: TimeType => true
+
+ // Interval types
+ case _: YearMonthIntervalType | _: DayTimeIntervalType |
CalendarIntervalType => true
Review Comment:
Confirmed -- `SparkArithmeticException` extends `ArithmeticException`, and
the batch-loop try covers `rowIter.next()` where upstream lazy evaluation runs,
so an unrelated ANSI error can indeed be re-labeled. Adding a harmless interval
column must not change the error condition.
Rather than narrowing the catch, we plan to make the translation unnecessary
and delete it outright: following the same approach as SPARK-57975 /
https://github.com/apache/spark/pull/57053 (opt-in lossless struct
representation for nanosecond timestamps), a follow-up will store
`CalendarInterval` losslessly as a struct of `(months: int32, days: int32,
microseconds: int64)` -- the type's own field layout, mirroring the default
cache's `CALENDAR_INTERVAL` `ColumnType`. With no `* 1000` conversion on the
cache path, the overflow (and `withIntervalOverflowTranslation` along with this
mis-attribution bug) ceases to exist, and the cache regains the default
serializer's full value domain for this type as well. The shared
`IntervalMonthDayNanoWriter` will also get a structured `DATETIME_OVERFLOW` at
the `Math.multiplyExact` site (the `TimestampNTZNanosWriter` pattern) for the
interchange paths that keep the standard MonthDayNano encoding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
