srielau commented on code in PR #58545:
URL: https://github.com/apache/spark/pull/58545#discussion_r3960907837
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala:
##########
@@ -156,6 +157,25 @@ object CharVarcharUtils extends Logging with
SparkCharVarcharUtils {
StructType(fields)
}
+ /**
+ * Applies CHAR padding and VARCHAR length checks when parsing text into a
typed schema.
+ * Null stays null. Unbounded STRING is unchanged. This is assignment
semantics
+ * (overflow raises EXCEED_LIMIT_LENGTH), not explicit CAST truncation.
+ */
+ def applyTextParseSemantics(value: UTF8String, dt: DataType): UTF8String = {
+ if (value == null) {
+ null
+ } else {
+ dt match {
+ case c: CharType =>
+ CharVarcharCodegenUtils.charTypeWriteSideCheck(value, c.length)
Review Comment:
Fixed in f7ff449eb64. The JSON, CSV, and XML parser regression now runs the
same non-space overflow value through both CHAR(5) and VARCHAR(5), checking a
null field in PERMISSIVE mode and EXCEED_LIMIT_LENGTH in FAILFAST mode. The
focused SPARK-59274 suite passes all 3 tests.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -617,16 +629,20 @@ class JacksonParser(
*/
private def convertMap(
parser: JsonParser,
- fieldConverter: ValueConverter): MapData = {
+ fieldConverter: ValueConverter,
+ keyType: DataType,
+ valueType: DataType): MapData = {
val keys = ArrayBuffer.empty[UTF8String]
val values = ArrayBuffer.empty[Any]
var badRecordException: Option[Throwable] = None
while (nextUntil(parser, JsonToken.END_OBJECT)) {
- keys += UTF8String.fromString(parser.currentName)
+ keys += CharVarcharUtils.applyTextParseSemantics(
Review Comment:
Fixed in f7ff449eb64. convertMap now invokes the value converter once,
recovers a partial value when available, then normalizes the key and appends
the key/value pair together. When conversion has no partial value, neither side
is appended. Added coverage showing an oversized CHAR key no longer hides a
later sibling field, plus a malformed entry followed by two valid keys that
normalize to the same key; EXCEPTION now reports DUPLICATED_MAP_KEY and
LAST_WIN keeps the last value. I used three entries for the latter because the
malformed integer produces no partial value, so the requested atomic semantics
intentionally retain no key from that entry. All three focused SPARK-59274
tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
