[
https://issues.apache.org/jira/browse/SPARK-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784736#comment-16784736
]
Ryan Blue commented on SPARK-23521:
-----------------------------------
I've turned off commenting on the google doc to preserve its state, with the
existing comments. I'm also adding a PDF of the final proposal to this issue.
> SPIP: Standardize SQL logical plans with DataSourceV2
> -----------------------------------------------------
>
> Key: SPARK-23521
> URL: https://issues.apache.org/jira/browse/SPARK-23521
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Ryan Blue
> Priority: Major
> Labels: SPIP
> Attachments: SPIP_ Standardize logical plans.pdf
>
>
> Executive Summary: This SPIP is based on [discussion about the DataSourceV2
> implementation|https://lists.apache.org/thread.html/55676ec1f5039d3deaf347d391cf82fe8574b8fa4eeab70110ed5b2b@%3Cdev.spark.apache.org%3E]
> on the dev list. The proposal is to standardize the logical plans used for
> write operations to make the planner more maintainable and to make Spark's
> write behavior predictable and reliable. It proposes the following principles:
> # Use well-defined logical plan nodes for all high-level operations: insert,
> create, CTAS, overwrite table, etc.
> # Use planner rules that match on these high-level nodes, so that it isn’t
> necessary to create rules to match each eventual code path individually.
> # Clearly define Spark’s behavior for these logical plan nodes. Physical
> nodes should implement that behavior so that all code paths eventually make
> the same guarantees.
> # Specialize implementation when creating a physical plan, not logical
> plans. This will avoid behavior drift and ensure planner code is shared
> across physical implementations.
> The SPIP doc presents a small but complete set of those high-level logical
> operations, most of which are already defined in SQL or implemented by some
> write path in Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
