[
https://issues.apache.org/jira/browse/SPARK-56819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-56819:
-----------------------------------
Labels: pull-request-available (was: )
> Add an option to trim trailing spaces when reading CHAR columns
> ---------------------------------------------------------------
>
> Key: SPARK-56819
> URL: https://issues.apache.org/jira/browse/SPARK-56819
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.1, 4.1.1
> Environment: spark-4.0.1
> Reporter: XiaodongHuan
> Priority: Major
> Labels: pull-request-available
>
> Spark currently enforces CHAR(N) fixed-length semantics by padding CHAR
> values on write, and by applying read-side padding when
> spark.sql.readSideCharPadding is enabled. This behavior is different from
> MySQL, where CHAR values normally have trailing spaces removed on retrieval
> unless PAD_CHAR_TO_FULL_LENGTH is enabled.
> This difference makes MySQL-to-Spark migration harder for workloads that rely
> on MySQL's default CHAR retrieval behavior. Users may observe different
> results for functions such as length(), concat(), comparisons in application
> code, or downstream BI/reporting queries, unless they manually wrap CHAR
> columns with rtrim() in every query.
> This proposal is to add an opt-in SQL configuration that trims trailing
> spaces from CHAR(N) columns/fields when reading table data. The default
> should preserve the current Spark behavior for compatibility. The new option
> should only affect CHAR types on the read path, and should not change
> VARCHAR/STRING semantics or write-side CHAR/VARCHAR length checks.
> The interaction with the existing spark.sql.readSideCharPadding option should
> be clearly defined, so users can choose between Spark's fixed-length CHAR
> behavior and MySQL-compatible CHAR retrieval behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
