[
https://issues.apache.org/jira/browse/SPARK-61588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61588:
-----------------------------------
Labels: pull-request-available (was: )
> Preserve persisted string defaults in legacy escaping mode
> ----------------------------------------------------------
>
> Key: SPARK-61588
> URL: https://issues.apache.org/jira/browse/SPARK-61588
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Yan Yan
> Priority: Minor
> Labels: pull-request-available
>
> With spark.sql.parser.escapedStringLiterals=true, Spark can reconstruct a
> persisted column default with extra backslashes. Rows written before ALTER
> TABLE ADD COLUMN can therefore return a different value from rows inserted
> later using DEFAULT.
> h3. Reproduction
> Run these statements in a fresh database. chr(39) constructs the apostrophe
> without depending on SQL string escaping in the reproduction itself.
> {code:sql}
> SET spark.sql.parser.escapedStringLiterals=true;
> CREATE TABLE default_escape_repro (id INT) USING parquet;
> INSERT INTO default_escape_repro VALUES (1);
> ALTER TABLE default_escape_repro ADD COLUMNS (
> s STRING DEFAULT concat('Dianne', chr(39), 's horse')
> );
> INSERT INTO default_escape_repro VALUES (2, DEFAULT);
> SELECT id, s, hex(s) FROM default_escape_repro ORDER BY id;
> {code}
> Expected: both rows contain {{Dianne's horse}}, with hex
> {{4469616E6E65277320686F727365}}.
> Actual before the fix: the old row can contain {{Dianne\'s horse}}, with hex
> {{4469616E6E655C277320686F727365}}, while the new row contains the expected
> value.
> Strings containing literal backslashes are also affected. The problem extends
> to strings nested in ARRAY, MAP and STRUCT defaults and to persisted default
> values converted for V2 catalogs, including CREATE TABLE LIKE.
> h3. Cause and fix
> Spark folds a default and stores its value as SQL in EXISTS_DEFAULT metadata.
> Literal.sql emits escaped SQL independently of the session's string-literal
> parsing setting. Reading that SQL through the legacy parser mode retains
> serialization escapes as value characters.
> Decode serialized literals using the escaping policy that Literal.sql emits,
> including recursive literal-only constructors and collated strings. Share
> this decoding with V2 column conversion while retaining its full analysis and
> constant validation. Other session settings, including the time zone, remain
> active. Expression-shaped external metadata, including casts containing
> functions, keeps session parsing and existing reader evaluation behavior.
> h3. Affected versions and validation
> The reader path was introduced in Spark 3.4.0 (SPARK-39211 / SPARK-39862);
> source-history inspection establishes the affected-version field. The V2 path
> that reads EXISTS_DEFAULT dates to Spark 3.5.2 (SPARK-48286). Historical
> releases were inspected statically; runtime validation was performed on the
> PR checkout.
> 313 tests pass across StructTypeSuite, CatalogV2UtilSuite,
> LiteralExpressionSuite, ExpressionParserSuite, InsertSuite and
> CreateTableLikeSuite. Regressions cover both parser modes, schema
> persistence, nested strings and collations, external CAST compatibility, V2
> analysis, real Parquet reads and V1-to-V2 CREATE TABLE LIKE. Catalyst and SQL
> Core production/test scalastyle checks pass.
> PR: https://github.com/apache/spark/pull/59820
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]