srielau commented on code in PR #58317:
URL: https://github.com/apache/spark/pull/58317#discussion_r3896519066
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:
##########
@@ -437,6 +437,11 @@ object OrcUtils extends Logging {
s"array<${getOrcSchemaString(a.elementType)}>"
case m: MapType =>
s"map<${getOrcSchemaString(m.keyType)},${getOrcSchemaString(m.valueType)}>"
+ // Under standard semantics, keep Spark responsible for CHAR/VARCHAR
assignment and scan
+ // checks. Native ORC would truncate or pad before Spark can validate the
original value.
+ // Preserve-only mode retains the native constrained schema and its legacy
enforcement.
+ case _: CharType | _: VarcharType if
SQLConf.get.charVarcharStandardSemantics =>
Review Comment:
Addressed in 7a02fee8f3d. Analysis now binds the effective
standard-semantics mode to the cleaned scan relation instead of letting ORC
consult the later caller SQLConf. The binding reaches both V1 HadoopFsRelation
schemas and V2 ORC scan construction, while preserve-only scans retain native
CHAR/VARCHAR enforcement. Added a permanent-view regression that creates the
view with standard semantics, disables that setting in the caller, and verifies
EXCEED_LIMIT_LENGTH across V1/V2 and vectorized/row readers.
##########
connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala:
##########
@@ -3724,6 +3724,82 @@ abstract class AvroSuite
}
}
+ test("SPARK-58814: Avro infers nested CHAR/VARCHAR schema and values") {
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ val input = spark.range(1).selectExpr(
+ "cast('ab' AS CHAR(4)) AS c",
+ "cast('xy' AS VARCHAR(3)) AS v",
+ "named_struct('c', cast('z' AS CHAR(2))) AS s",
+ "array(cast('q' AS VARCHAR(2))) AS a",
+ "map(cast('k' AS CHAR(2)), cast('v' AS VARCHAR(2))) AS m")
+ input.write.mode("overwrite").format("avro").save(path)
+
+ val readBack = spark.read.format("avro").load(path)
+ assert(DataType.equalsIgnoreNullability(readBack.schema, input.schema))
+ checkAnswer(
+ readBack.selectExpr("concat('<', c, '>')", "v", "concat('<', s.c,
'>')"),
Review Comment:
Addressed in 7a02fee8f3d. The Avro round-trip projection now materializes
and asserts both collection fields: Seq(\"q\") for the array and Map(\"k \" ->
\"v\") for the CHAR-keyed map. The targeted Avro V1 and V2 tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
