[
https://issues.apache.org/jira/browse/SPARK-60479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-60479:
-----------------------------------
Assignee: Serge Rielau
> Push down CHAR filters when read-side CHAR padding is enabled
> -------------------------------------------------------------
>
> Key: SPARK-60479
> URL: https://issues.apache.org/jira/browse/SPARK-60479
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Serge Rielau
> Assignee: Serge Rielau
> Priority: Major
> Labels: pull-request-available
>
> spark.sql.readSideCharPadding defaults to true. For CHAR columns,
> ApplyCharTypePadding wraps the scan in
> StaticInvoke(CharVarcharCodegenUtils.readSidePadding, col, n) and strips CHAR
> metadata from the scan output.
> JDBC / DSv2 filter translation (V2ExpressionBuilder / PushablePredicate) does
> not understand that StaticInvoke. A predicate such as CHAR_COL = 'Y' stays on
> the Spark side. Only pushable pieces such as IS NOT NULL go to the remote
> system, so JDBC tables (including Oracle CHAR(n)) are fully scanned.
> With the flag off, the same predicate is rewritten as RPAD(col, n, ' ') =
> 'Y'. That form is translatable after SPARK-50692 / SPARK-51321, and dialects
> that implement RPAD (H2, Postgres, SQL Server) push it. Oracle's OSS dialect
> still does not list RPAD in isSupportedFunction, so Oracle JDBC would not
> push even that form today.
> The padding rewrite is required for CHAR semantics. Pushdown should still
> happen, using an equivalent remote predicate.
> Expected
> With spark.sql.readSideCharPadding=true (and with
> spark.sql.charVarchar.standardSemantics.enabled=true, which also pads on
> read), CHAR comparisons and similar predicates against JDBC/DSv2 CHAR columns
> should push down when the dialect can evaluate the equivalent padding.
> Example: WHERE prod_version_current = 'Y' on Oracle CHAR(1) should appear in
> the remote SQL as something like RPAD("PROD_VERSION_CURRENT", 1, ' ') = 'Y'
> (or an equivalent CHAR comparison), not only IS NOT NULL.
> Spark-side read padding of the returned values can stay.
> Proposed approach
> Translate StaticInvoke of readSidePadding (and, if safe,
> charTypeReadSideCheck) in V2ExpressionBuilder to the same V2 RPAD expression
> already used for StringRPad, so existing RPAD-capable dialects keep working
> with the default config.
> Add RPAD to OracleDialect.isSupportedFunction (Oracle already has RPAD).
> Tests: JDBC V2 CHAR column + equality / IN / IS NOT NULL with the flag on and
> off; assert the pushed SQL contains the comparison, not only null checks.
> Cover at least H2; Oracle if the docker suite is in reach.
> Do not push a bare col = 'Y' and rely on remote CHAR padding. Spark with
> read-side padding uses string equality on the padded value (RPAD(col, n, ' ')
> = lit). For CHAR(n) vs a shorter literal those are not the same as Oracle
> CHAR comparison.
> Related
> SPARK-40697: read-side CHAR padding (default true)
> SPARK-50692: RPAD pushdown for predicate-side CHAR padding
> SPARK-51321: SQL Server LPAD/RPAD
> SPARK-58794: full CHAR/VARCHAR (this bug is independent of
> standard-semantics; it hits the default 3.4+ path)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
