cloud-fan commented on code in PR #58581:
URL: https://github.com/apache/spark/pull/58581#discussion_r4057254064
##########
sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:
##########
@@ -469,8 +539,10 @@ object DataType {
/**
* Returns a map of field path to collation name.
*/
- private def getCollationsMap(metadataFields: List[JField]): Map[String,
String] = {
- val collationsJsonOpt = metadataFields.find(_._1 ==
COLLATIONS_METADATA_KEY).map(_._2)
+ private def getCollationsMap(
+ metadataFields: List[JField],
+ metadataKey: String): Map[String, String] = {
+ val collationsJsonOpt = metadataFields.find(_._1 == metadataKey).map(_._2)
Review Comment:
**Non-blocking (P2):** This treats a malformed value under the new reserved
key as if the key were absent. If `__CHAR_VARCHAR_COLLATIONS` is a string or
another non-object, this falls through to `Map.empty`; if the object contains
non-string values, `collect` silently skips them. `parseStructField` then
removes the reserved key, so inputs such as
`{"__CHAR_VARCHAR_COLLATIONS":"caller"}` deserialize to an uncollated type with
the metadata lost. Python rejects those same shapes, and the compatibility
contract calls for malformed restoration encodings to fail explicitly. Please
validate that the dedicated value is an object and that every entry is a
provider-qualified string before stripping it, with Scala/Python regressions
for non-object and non-string values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
