cloud-fan commented on code in PR #58584:
URL: https://github.com/apache/spark/pull/58584#discussion_r4068179406
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:
##########
@@ -829,6 +869,26 @@ private[hive] trait HiveInspectors {
null
}
}
+ case (_, c: CharType) =>
Review Comment:
**Non-blocking (P2):** This typed unwrapper is also used by Hive ORC
whenever a complex field contains a nanosecond timestamp. With preserve-only
types and `spark.sql.readSideCharPadding=false`, recursion now reaches this
unconditional CHAR branch and pads an adjacent nested value anyway; the same
CHAR remains raw when the unrelated nanos sibling is absent. Please separate
the ORC nanos-only conversion policy from the strict Hive-function CHAR/VARCHAR
policy, propagate that choice through nested containers, and cover this
disabled-padding ORC interaction.
**Recommended change:** Separate nanos-only recursive conversion from
Hive-function bounded-string enforcement, make the selected policy propagate
through array/map/struct recursion, route the Hive ORC nanos path through the
nanos-only policy, and add caller-level coverage for preserve-only read padding
disabled alongside nested nanos timestamps.
**Why this works:** Introduce an explicit internal conversion policy or
distinct recursive entry point in HiveInspectors. Hive function evaluators
select the policy that applies CHAR/VARCHAR read checks and validated bounded
map-key conversion. Hive ORC selects the policy that preserves nanos timestamps
but delegates CHAR/VARCHAR leaves to raw untyped extraction;
ApplyCharTypePadding remains the authority that conditionally pads table reads.
Ensure every recursive container call carries the selected policy rather than
falling back to the all-checks default.
**Scope:** Disentangle Hive function result validation from Hive ORC nanos
decoding without changing either feature's supported type surface.
**Compatibility:** Legacy disabled-mode Hive function results remain
unbounded StringType; enabled Hive function boundaries remain checked
first-class CHAR/VARCHAR; Script TRANSFORM remains excluded.
**Risks:** A policy flag that is not propagated into nested arrays, maps,
and structs would leave the regression in deeper shapes. Disabling bounded
checks for Hive function callers would lose required padding, length
enforcement, and normalized map-key validation. Changing the nanos timestamp
arms could reintroduce microsecond truncation in Hive ORC.
**Constraints:** Hive UDF/UDAF/UDTF outputs analyzed as CHAR/VARCHAR must
retain their declared kind, length, padding, default collation, and overflow
behavior. Hive ORC must preserve TimestampNTZNanosType and
TimestampLTZNanosType precision recursively. Table-read padding must continue
to follow readSideCharPadding, with standard semantics overriding a disabled
setting through ApplyCharTypePadding.
**Success:** In preserve-only mode with readSideCharPadding=false, a Hive
ORC CHAR/VARCHAR value remains raw whether or not an adjacent nested field uses
a nanos timestamp. When read-side padding is enabled or standard semantics
requires it, table reads still expose the expected CHAR padding through the
established logical padding rule. Hive UDF/UDAF/UDTF typed conversion continues
to enforce bounded-string padding, overflow, runtime-inspector compatibility,
and duplicate normalized map keys. Hive ORC recursively round-trips both nanos
timestamp kinds without precision loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
