srielau commented on code in PR #59167:
URL: https://github.com/apache/spark/pull/59167#discussion_r4169052261
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -659,6 +716,60 @@ class JacksonParser(
}
}
+ /**
+ * CHAR/VARCHAR JSON maps apply assignment to object names, then
+ * `spark.sql.mapKeyDedupPolicy` to collisions created by that normalization.
+ * Exact repeated names stay historical last-wins.
+ */
+ private def convertConstrainedMap(
+ parser: JsonParser,
+ fieldConverter: ValueConverter,
+ keyType: DataType,
+ valueType: DataType,
+ mapDepth: Int): MapData = {
+ val lastEntries =
+ mutable.LinkedHashMap.empty[UTF8String, (UTF8String, Option[Any])]
+ var partialResultException: Option[Throwable] = None
+ var badMapException: Option[Throwable] = None
+
+ while (nextUntil(parser, JsonToken.END_OBJECT)) {
+ val rawKey = UTF8String.fromString(parser.currentName)
+ val value = try {
+ Some(fieldConverter.apply(parser))
+ } catch {
+ case err: PartialValueException if enablePartialResults =>
+ partialResultException =
partialResultException.orElse(Some(err.cause))
+ Some(err.partialResult)
+ case DuplicateMapKeyUtils(e) => throw e
+ case NonFatal(e) =>
+ badMapException = badMapException.orElse(Some(e))
+ skipRemainingValue(parser, mapDepth)
+ None
+ }
+ try {
+ val key = CharVarcharUtils.applyTextParseSemantics(rawKey, keyType)
+ lastEntries.update(rawKey, (key, value))
Review Comment:
Thanks. JSON now matches XML: `lastEntries.remove(rawKey)` before `update`
so LAST_WIN follows the latest occurrence of an exact name.
`from_json('{"a":1,"a ":2,"a":3}', 'MAP<CHAR(2), INT>')` keeps `a ` -> 3 under
LAST_WIN and still raises `DUPLICATED_MAP_KEY` under EXCEPTION.
`interleavedCharJsonQuery` now expects `[("b ", 2), ("a ", 3)]`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -635,9 +692,9 @@ class JacksonParser(
case err: PartialValueException if enablePartialResults =>
badRecordException = badRecordException.orElse(Some(err.cause))
values += err.partialResult
- case NonFatal(e) if enablePartialResults =>
+ case NonFatal(e) if drainErrors =>
badRecordException = badRecordException.orElse(Some(e))
- parser.skipChildren()
+ skipRemainingValue(parser, mapDepth)
Review Comment:
Added the STRING-keyed case with `enablePartialResults=true`:
`from_json('{"a":[1,"bad",99],"b":[3]}', 'MAP<STRING, ARRAY<INT>>')` now keeps
only `b` (inner tokens do not leak). `convertOrdinaryMap` omits the failed pair
instead of leaving an unpaired key, so the partial map can retain later
siblings. Duplicate STRING cardinality is unchanged. The PR description now
mentions this drain change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
