[
https://issues.apache.org/jira/browse/SPARK-59943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated SPARK-59943:
-----------------------------------------
Description:
SPARK-34586 (https://github.com/apache/spark/pull/58153) adds
writeDistributionMode and writeOrdering as parameters of CreateTable,
CreateTableAsSelect, ReplaceTable and
ReplaceTableAsSelect, and of the CREATE/REPLACE TABLE execs. That changes the
plan and exec signatures, so extensions that match or construct them
positionally stop compiling (for example Apache Paimon's
PaimonCreateTableAsSelectStrategy).
This ticket covers two changes:
1. Carry the pair on TableSpec instead, as the SPARK-43529 follow-up
(7e94f2a5433) did for its parameter, so the four plan signatures and the exec
signatures go back to what they were before SPARK-34586.
TableSpec.withNewLocation also rebuilds TableSpec positionally and has to keep
the pair.
2. Move the capability check,
WriteDistributionAndOrdering.validateCatalogForWriteDistributionAndOrdering,
from DataSourceV2Strategy to the `case create: V2CreateTablePlan` block of
CheckAnalysis.
SparkPlanner runs extraStrategies and injected planner strategies before
DataSourceV2Strategy, and the first match wins. A custom strategy that plans
these commands with its own exec therefore
skips the check, so CREATE TABLE ... ORDERED BY on a catalog without
SUPPORTS_CREATE_TABLE_WITH_WRITE_DISTRIBUTION_AND_ORDERING succeeds without the
layout instead of
failing with UNSUPPORTED_FEATURE.TABLE_OPERATION. This already happens with
SPARK-34586 as merged: a Scala case-class pattern compiles to a type test plus
accessor calls, so an extension built
against an earlier release, e.g. `case c: CreateTableAsSelect =>
MyCtasExec(...) :: Nil`, still links and never reads the new fields. Every
planner path goes through CheckAnalysis; a check in Spark's execs would not
cover a custom exec. The sibling checks for default values, constraints and
generated columns also sit in DataSourceV2Strategy and may be worth moving the
same way.
was:
SPARK-34586 (https://github.com/apache/spark/pull/58153) adds
writeDistributionMode and writeOrdering as parameters of CreateTable,
CreateTableAsSelect, ReplaceTable and ReplaceTableAsSelect, and of the
CREATE/REPLACE TABLE execs. That changes the plan and exec signatures, so
extensions that match or construct them positionally stop compiling (for
example Apache Paimon's PaimonCreateTableAsSelectStrategy).
Carry the pair on TableSpec instead, as the SPARK-43529 follow-up (7e94f2a5433)
did for its parameter, so the four plan signatures and the exec signatures go
back to what they were before SPARK-34586.
Once a custom strategy compiles again it skips
WriteDistributionAndOrdering.validateCatalogForWriteDistributionAndOrdering,
which runs in DataSourceV2Strategy, so move that check into the execs or into
CheckAnalysis. TableSpec.withNewLocation also rebuilds TableSpec positionally
and has to keep the pair. Should land before 4.4.0.
> Carry the CREATE/REPLACE TABLE write distribution and ordering on TableSpec
> ---------------------------------------------------------------------------
>
> Key: SPARK-59943
> URL: https://issues.apache.org/jira/browse/SPARK-59943
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Anurag Mantripragada
> Priority: Major
>
> SPARK-34586 (https://github.com/apache/spark/pull/58153) adds
> writeDistributionMode and writeOrdering as parameters of CreateTable,
> CreateTableAsSelect, ReplaceTable and
> ReplaceTableAsSelect, and of the CREATE/REPLACE TABLE execs. That changes the
> plan and exec signatures, so extensions that match or construct them
> positionally stop compiling (for example Apache Paimon's
> PaimonCreateTableAsSelectStrategy).
> This ticket covers two changes:
> 1. Carry the pair on TableSpec instead, as the SPARK-43529 follow-up
> (7e94f2a5433) did for its parameter, so the four plan signatures and the exec
> signatures go back to what they were before SPARK-34586.
> TableSpec.withNewLocation also rebuilds TableSpec positionally and has to
> keep the pair.
> 2. Move the capability check,
> WriteDistributionAndOrdering.validateCatalogForWriteDistributionAndOrdering,
> from DataSourceV2Strategy to the `case create: V2CreateTablePlan` block of
> CheckAnalysis.
> SparkPlanner runs extraStrategies and injected planner strategies before
> DataSourceV2Strategy, and the first match wins. A custom strategy that plans
> these commands with its own exec therefore
> skips the check, so CREATE TABLE ... ORDERED BY on a catalog without
> SUPPORTS_CREATE_TABLE_WITH_WRITE_DISTRIBUTION_AND_ORDERING succeeds without
> the layout instead of
> failing with UNSUPPORTED_FEATURE.TABLE_OPERATION. This already happens with
> SPARK-34586 as merged: a Scala case-class pattern compiles to a type test
> plus accessor calls, so an extension built
> against an earlier release, e.g. `case c: CreateTableAsSelect =>
> MyCtasExec(...) :: Nil`, still links and never reads the new fields. Every
> planner path goes through CheckAnalysis; a check in Spark's execs would not
> cover a custom exec. The sibling checks for default values, constraints and
> generated columns also sit in DataSourceV2Strategy and may be worth moving
> the same way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
