cloud-fan commented on code in PR #58584:
URL: https://github.com/apache/spark/pull/58584#discussion_r3992124832
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala:
##########
@@ -245,7 +250,7 @@ private[hive] case class HiveGenericUDTF(
private lazy val wrappers = children.map(x => wrapperFor(toInspector(x),
x.dataType)).toArray
@transient
- private lazy val unwrapper = unwrapperFor(outputInspector)
+ private lazy val unwrapper = unwrapperFor(outputInspector, elementSchema)
Review Comment:
**Non-blocking (P2):** `GenerateExec` binds this generator with
`BindReferences`, which calls `withNewChildrenInternal` and creates a copy
whose lazy `elementSchema` is unset. For a persisted view created with
first-class CHAR/VARCHAR and queried under legacy settings, that copy can
rebuild the runtime unwrapper with STRING targets, losing CHAR padding and
length checks even though analysis used the captured schema. Please retain the
resolved element schema across copies, as the GenericUDF and UDAF paths now
retain their analyzed types, and add the corresponding opposite-config
persisted-view UDTF coverage.
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:
##########
@@ -897,6 +953,19 @@ private[hive] trait HiveInspectors {
(value: Any, row: InternalRow, ordinal: Int) => row(ordinal) =
unwrapper(value)
}
+ /**
+ * Builds an in-place unwrapper that also honors target-type-specific
conversions.
+ */
+ def unwrapperFor(
+ field: HiveStructField,
+ dataType: DataType): (Any, InternalRow, Int) => Unit = dataType match {
+ case _: CharType | _: VarcharType =>
+ val unwrapper = unwrapperFor(field.getFieldObjectInspector, dataType)
+ (value: Any, row: InternalRow, ordinal: Int) => row(ordinal) =
unwrapper(value)
+ case _ =>
Review Comment:
**Non-blocking (P2):** This fallback drops the Catalyst child types for
ARRAY/MAP/STRUCT outputs, even though the ObjectInspector/DataType overload
above already recurses with them. A SerDe exposing a nested generic string
inspector can therefore return an unpadded CHAR value or let an over-length
CHAR/VARCHAR value through without `EXCEED_LIMIT_LENGTH`. Please select the
target-aware unwrapper whenever the declared field type contains CHAR/VARCHAR
recursively, while retaining the specialized field setters for types that do
not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
