srielau commented on code in PR #58317:
URL: https://github.com/apache/spark/pull/58317#discussion_r4186481125
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:
##########
@@ -536,11 +558,14 @@ object OrcUtils extends Logging {
resultSchema: StructType,
partitionSchema: StructType,
conf: Configuration): String = {
- val resultSchemaString = if (canPruneCols) {
- OrcUtils.getOrcSchemaString(resultSchema)
+ val readSchema = if (canPruneCols) {
+ resultSchema
} else {
- OrcUtils.getOrcSchemaString(StructType(dataSchema.fields ++
partitionSchema.fields))
+ StructType(dataSchema.fields ++ partitionSchema.fields)
}
+ // Spark CHAR/VARCHAR in the read schema are Catalyst constraints; ask ORC
for STRING.
+ val resultSchemaString =
+ OrcUtils.getOrcSchemaString(orcReadSchemaWithoutCharVarchar(readSchema))
Review Comment:
Addressed in e7556da9857 without changing the ORC reader.
The STRING MAPRED schema stays mode-independent. I documented that
preserve-only ORC CHAR/VARCHAR reads over STRING storage return stored values
(no ORC truncation), matching Parquet, and that read-side length checks apply
only under `spark.sql.charVarchar.standardSemantics.enabled`:
- `PRESERVE_CHAR_VARCHAR_TYPE_INFO` config doc
- SQL migration guide (4.3 to 4.4)
- PR user-facing section
`SPARK-58814: preserve-only ORC CHAR/VARCHAR inference and STRING values`
writes first-class CHAR/VARCHAR under preserve-only and infers
`CHAR(4)`/`VARCHAR(4)`, and reads stored `abcdef` with CHAR(4)/VARCHAR(4)
schemas as `abcdef` for V1/V2 and vectorized/row readers. Standard-semantics
`EXCEED_LIMIT_LENGTH` is unchanged.
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveCharVarcharTestSuite.scala:
##########
@@ -54,12 +55,22 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite
with TestHiveSinglet
}
}
- test("SPARK-35700: Read char/varchar orc table with created and written by
external systems") {
+ test("SPARK-35700, SPARK-58814: read native char/varchar ORC") {
withTable("t") {
hiveClient.runSqlHive("CREATE TABLE t(c CHAR(5), v VARCHAR(7)) STORED AS
ORC")
- hiveClient.runSqlHive("INSERT INTO t VALUES('Spark', 'kyuubi')")
- checkAnswer(sql("SELECT c, v from t"), Row("Spark", "kyuubi"))
- checkAnswer(sql("SELECT v from t where c = 'Spark' and v = 'kyuubi'"),
Row("kyuubi"))
+ hiveClient.runSqlHive("INSERT INTO t VALUES('Sp', 'kyuubi')")
+ Seq(true, false).foreach { vectorizedReaderEnabled =>
+ withSQLConf(
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true",
Review Comment:
Addressed in e7556da9857.
The original SPARK-35700 default-configuration case is back: Hive-written
native ORC `CHAR(5)`/`VARCHAR(7)` with `'Spark'`/`'kyuubi'` and `v = 'kyuubi'`
through `convertMetastoreOrc`, vectorized and row. The standard-semantics
padding case stays next to it and still uses `'Sp'` so CHAR padding remains
covered.
##########
sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala:
##########
@@ -1976,17 +2138,6 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
assert(spark.read.orc(path).schema.head.dataType === StringType)
}
}
- // preserveCharVarcharTypeInfo also keeps first-class types, so write
still stamps CHAR.
Review Comment:
Addressed in e7556da9857.
`SPARK-58814: preserve-only ORC CHAR/VARCHAR inference and STRING values`
writes CHAR(4)/VARCHAR(4) under preserve-only (standard semantics off) and
asserts a file-only read infers those types, so a missing
`spark.sql.catalyst.type` stamp fails the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
