srielau opened a new pull request, #58317:
URL: https://github.com/apache/spark/pull/58317
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR:
https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a
faster review.
7. If you want to add a new configuration, please read the guideline first
for naming configurations in
'common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the
guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Parent: [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794)
(first-class CHAR/VARCHAR under
`spark.sql.charVarchar.standardSemantics.enabled`).
- Keep Spark responsible for CHAR/VARCHAR assignment and scan checks on ORC.
`OrcUtils.getOrcSchemaString` now maps requested `CHAR`/`VARCHAR` to ORC STRING
so native ORC does not truncate or pad before Spark can validate the original
value. Write-side already uses ORC STRING plus `spark.sql.catalyst.type`.
- Add SPARK-58814 coverage for major formats under `standardSemantics`:
- Parquet/ORC nested CHAR/VARCHAR (struct/array/map keys) on file-only
inference, V1 and V2
- Avro nested inference plus explicit-schema pad/overflow (V1 and V2)
- CSV/JSON user-schema scans, length checks, and catalog INSERT
assignment/overflow
- Cross-flag: `standardSemantics=false` collapses to STRING;
`preserveCharVarcharTypeInfo` keeps types
Parquet already embeds Spark schema JSON in footer metadata, so nested
CHAR/VARCHAR inference needed tests rather than a writer change.
### Why are the changes needed?
Without this, Spark-written files and user-specified schemas do not keep
first-class CHAR/VARCHAR through round-trips in the remaining format gaps. For
ORC specifically, an explicit `schema("c CHAR(4)")` read sent native ORC
`char(n)` via `MAPRED_INPUT_SCHEMA`, so ORC truncated `"abcdef"` to `"abcd"`
before `charTypeReadSideCheck` and `EXCEED_LIMIT_LENGTH` never fired.
### Does this PR introduce _any_ user-facing change?
Yes, when `spark.sql.charVarchar.standardSemantics.enabled` is true
(unreleased / master):
- ORC no longer silently truncates oversized CHAR/VARCHAR values on
schema-specified reads; Spark rejects them with `EXCEED_LIMIT_LENGTH`.
- Parquet/ORC/Avro file-only inference and CSV/JSON user schemas preserve
CHAR/VARCHAR types and apply pad/length and INSERT assignment as covered by the
new tests.
### How was this patch tested?
New tests in `BasicCharVarcharTestSuite` (`SPARK-58814: major formats
preserve CHAR/VARCHAR schemas and values`) and `AvroSuite` (V1 and V2). Also
ran existing ORC query suites and scalastyle:
```
build/sbt "sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z
SPARK-58814"
build/sbt "avro/testOnly org.apache.spark.sql.avro.AvroV1Suite -- -z
SPARK-58814"
build/sbt "avro/testOnly org.apache.spark.sql.avro.AvroV2Suite -- -z
SPARK-58814"
build/sbt "sql/testOnly
org.apache.spark.sql.execution.datasources.orc.OrcV1QuerySuite"
build/sbt "sql/testOnly
org.apache.spark.sql.execution.datasources.orc.OrcV2QuerySuite"
build/sbt "sql/scalastyle" "sql/Test/scalastyle" "avro/scalastyle"
"avro/Test/scalastyle"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
