srielau commented on code in PR #58581:
URL: https://github.com/apache/spark/pull/58581#discussion_r3960855730
##########
python/pyspark/sql/types.py:
##########
@@ -1366,10 +1390,15 @@ def processDataType(dt: DataType, fieldPath: str) ->
None:
return collationMetadata
def _isCollatedString(self, dt: DataType) -> bool:
- return isinstance(dt, StringType) and not dt.isUTF8BinaryCollation()
+ if isinstance(dt, StringType):
+ return not dt.isUTF8BinaryCollation()
+ if isinstance(dt, (CharType, VarcharType)):
+ return not dt.isUTF8BinaryCollation()
Review Comment:
Fixed in 922d99a3501. Scala and Python now test CHAR/VARCHAR
optional-collation presence before applying the existing StringType
default-collation rule. I added a mixed MapType round trip with an explicit
UTF8_BINARY CHAR key and a UNICODE_CI VARCHAR value in both test suites. The
focused Scala and Python tests pass.
##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala:
##########
@@ -223,7 +223,9 @@ class SparkSession private[sql] (
/** @inheritdoc */
def createDataFrame(rows: java.util.List[Row], schema: StructType):
DataFrame = {
- createDataset(RowEncoder.encoderFor(schema),
rows.iterator().asScala).toDF()
+ // The client cannot observe the server's CHAR/VARCHAR configuration.
Encode an explicitly
+ // provided schema independently of the client's local configuration, as
for result schemas.
+ createDataset(RowEncoder.encoderForResultSchema(schema),
rows.iterator().asScala).toDF()
Review Comment:
Fixed in 922d99a3501. The schema-only server path now builds an empty
physical-string relation and routes it through buildLocalRelationFromRows(...,
Some(schema)), reusing the same Dataset.to(schema) reconciliation as populated
input. The Connect E2E test now checks that empty and populated inputs are both
preserved with standard semantics and both rejected by the default policy. The
focused E2E test passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
