[
https://issues.apache.org/jira/browse/SPARK-59943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated SPARK-59943:
-----------------------------------------
Description:
SPARK-34586 ([https://github.com/apache/spark/pull/58153]) adds
writeDistributionMode and writeOrdering as parameters of CreateTable,
CreateTableAsSelect, ReplaceTable and
ReplaceTableAsSelect, and of the CREATE/REPLACE TABLE execs. That changes the
plan and exec signatures, so extensions that match or construct them
positionally stop compiling (for example Apache Paimon's
PaimonCreateTableAsSelectStrategy).
This ticket covers two changes:
# Carry the pair on TableSpec instead, as the SPARK-43529 follow-up
(7e94f2a5433) did for its parameter, so the four plan signatures and the exec
signatures go back to what they were before SPARK-34586.
TableSpec.withNewLocation also rebuilds TableSpec positionally and has to keep
the pair.
# Move the capability check,
WriteDistributionAndOrdering.validateCatalogForWriteDistributionAndOrdering,
from DataSourceV2Strategy to the `case create: V2CreateTablePlan` block of
CheckAnalysis. SparkPlanner runs extraStrategies and injected planner
strategies before DataSourceV2Strategy, and the first match wins. A custom
strategy that plans these commands with its own exec therefore skips the check,
so CREATE TABLE ... ORDERED BY on a catalog without
SUPPORTS_CREATE_TABLE_WITH_WRITE_DISTRIBUTION_AND_ORDERING succeeds without the
layout instead of failing with UNSUPPORTED_FEATURE.TABLE_OPERATION. This
already happens with SPARK-34586 as merged: a Scala case-class pattern compiles
to a type test plus accessor calls, so an extension built against an earlier
release, e.g. `case c: CreateTableAsSelect => MyCtasExec(...) :: Nil`, still
links and never reads the new fields. Every planner path goes through
CheckAnalysis; a check in Spark's execs would not cover a custom exec. The
sibling checks for default values, constraints and generated columns also sit
in DataSourceV2Strategy and may be worth moving the same way.
was:
SPARK-34586 ([https://github.com/apache/spark/pull/58153]) adds
writeDistributionMode and writeOrdering as parameters of CreateTable,
CreateTableAsSelect, ReplaceTable and
ReplaceTableAsSelect, and of the CREATE/REPLACE TABLE execs. That changes the
plan and exec signatures, so extensions that match or construct them
positionally stop compiling (for example Apache Paimon's
PaimonCreateTableAsSelectStrategy).
This ticket covers two changes:
# Carry the pair on TableSpec instead, as the SPARK-43529 follow-up
(7e94f2a5433) did for its parameter, so the four plan signatures and the exec
signatures go back to what they were before SPARK-34586.
TableSpec.withNewLocation also rebuilds TableSpec positionally and has to keep
the pair.
# Move the capability check,
WriteDistributionAndOrdering.validateCatalogForWriteDistributionAndOrdering,
from DataSourceV2Strategy to the `case create: V2CreateTablePlan` block of
CheckAnalysis. SparkPlanner runs extraStrategies and injected planner
strategies before DataSourceV2Strategy, and the first match wins. A custom
strategy that plans these commands with its own exec therefore skips the check,
so CREATE TABLE ... ORDERED BY on a catalog without
SUPPORTS_CREATE_TABLE_WITH_WRITE_DISTRIBUTION_AND_ORDERING succeeds without the
layout instead of failing with UNSUPPORTED_FEATURE.TABLE_OPERATION. This
already happens with SPARK-34586 as merged: a Scala case-class pattern compiles
to a type test plus accessor calls, so an extension built
against an earlier release, e.g. `case c: CreateTableAsSelect =>
MyCtasExec(...) :: Nil`, still links and never reads the new fields. Every
planner path goes through CheckAnalysis; a check in Spark's execs would not
cover a custom exec. The sibling checks for default values, constraints and
generated columns also sit in DataSourceV2Strategy and may be worth moving the
same way.
> Carry the CREATE/REPLACE TABLE write distribution and ordering on TableSpec
> ---------------------------------------------------------------------------
>
> Key: SPARK-59943
> URL: https://issues.apache.org/jira/browse/SPARK-59943
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Anurag Mantripragada
> Priority: Major
>
> SPARK-34586 ([https://github.com/apache/spark/pull/58153]) adds
> writeDistributionMode and writeOrdering as parameters of CreateTable,
> CreateTableAsSelect, ReplaceTable and
> ReplaceTableAsSelect, and of the CREATE/REPLACE TABLE execs. That changes the
> plan and exec signatures, so extensions that match or construct them
> positionally stop compiling (for example Apache Paimon's
> PaimonCreateTableAsSelectStrategy).
> This ticket covers two changes:
> # Carry the pair on TableSpec instead, as the SPARK-43529 follow-up
> (7e94f2a5433) did for its parameter, so the four plan signatures and the exec
> signatures go back to what they were before SPARK-34586.
> TableSpec.withNewLocation also rebuilds TableSpec positionally and has to
> keep the pair.
> # Move the capability check,
> WriteDistributionAndOrdering.validateCatalogForWriteDistributionAndOrdering,
> from DataSourceV2Strategy to the `case create: V2CreateTablePlan` block of
> CheckAnalysis. SparkPlanner runs extraStrategies and injected planner
> strategies before DataSourceV2Strategy, and the first match wins. A custom
> strategy that plans these commands with its own exec therefore skips the
> check, so CREATE TABLE ... ORDERED BY on a catalog without
> SUPPORTS_CREATE_TABLE_WITH_WRITE_DISTRIBUTION_AND_ORDERING succeeds without
> the layout instead of failing with UNSUPPORTED_FEATURE.TABLE_OPERATION. This
> already happens with SPARK-34586 as merged: a Scala case-class pattern
> compiles to a type test plus accessor calls, so an extension built against an
> earlier release, e.g. `case c: CreateTableAsSelect => MyCtasExec(...) ::
> Nil`, still links and never reads the new fields. Every planner path goes
> through CheckAnalysis; a check in Spark's execs would not cover a custom
> exec. The sibling checks for default values, constraints and generated
> columns also sit in DataSourceV2Strategy and may be worth moving the same way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
