srielau opened a new pull request, #58581: URL: https://github.com/apache/spark/pull/58581
### What changes were proposed in this pull request? Carry CHAR/VARCHAR collation through Spark Connect type conversion, matching STRING: - Add optional `collation` to Connect proto `Char` and `VarChar`. - Encode and decode it in Scala `DataTypeProtoConverter` and Python Connect type converters. - Let Python `CharType`/`VarcharType` carry collation, and parse collated CHAR/VARCHAR in DDL/JSON. - Use the result-schema encoder for Connect `createDataFrame` so collated CHAR/VARCHAR are accepted independently of the client's local first-class-types flag. - Older clients ignore the new optional field and keep uncollated CHAR/VARCHAR. JIRA: https://issues.apache.org/jira/browse/SPARK-59276 ### Why are the changes needed? CHAR/VARCHAR are first-class types and can have collations, but Connect dropped collation on proto round-trip. Result schemas, `createDataFrame`, and Python type objects then lost collation or failed to encode collated CHAR/VARCHAR. ### Does this PR introduce _any_ user-facing change? Yes, for Spark Connect clients talking to a server that uses collated CHAR/VARCHAR. Result schemas and Python `CharType`/`VarcharType` now preserve collation. Older clients still receive CHAR/VARCHAR length and ignore the new optional proto field. Classic Spark and the default uncollated types are unchanged. ### How was this patch tested? Added/updated coverage for: - Scala proto round-trip of collated and uncollated CHAR/VARCHAR. - JSON schema round-trip of collated CHAR/VARCHAR. - JVM Connect result schema and `createDataFrame`. - Arrow encoder round-trip with collated CHAR/VARCHAR. - Python type JSON/repr and Connect proto conversion. Ran focused sbt tests: ``` sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 -Dsbt.override.build.repos=true \ 'connect-client-jvm/testOnly org.apache.spark.sql.connect.ClientE2ETestSuite -- -z SPARK-59276' ``` Both SPARK-59276 JVM Connect tests passed. Python `test_char_varchar_type_collations` passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
