[
https://issues.apache.org/jira/browse/SPARK-52223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aakash Japi updated SPARK-52223:
--------------------------------
Description:
SDP is a Spark Connect-only feature. We need to add the following APIs to cover
the pipeline lifecycle:
# {{CreateDataflowGraph}} creates a new graph in the registry.
# {{DefineDataset}} and {{DefineFlow}} register elements to the created graph.
Datasets are the nodes of the dataflow graph, and are either tables or views,
and flows are the edges connecting them.
# {{StartRun}} starts a run, which is a single execution of a graph.
# {{StopRun}} stops an existing run, while {{DropPipeline}} stops any current
runs and drops the pipeline.
# `PipelineCommand`, which contains a oneof that contains one of the above
protos. This is the interface exposed to the SC command itself.
We also need to add the new {{PipelineCommand}} object to the
{{ExecutePlanRequest}} and the {{PipelineCommand.Response}} to the
{{ExecutePlanResponse}} object.
> [SDP] Create spark connect API for SDP
> --------------------------------------
>
> Key: SPARK-52223
> URL: https://issues.apache.org/jira/browse/SPARK-52223
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: Aakash Japi
> Priority: Major
>
> SDP is a Spark Connect-only feature. We need to add the following APIs to
> cover the pipeline lifecycle:
> # {{CreateDataflowGraph}} creates a new graph in the registry.
> # {{DefineDataset}} and {{DefineFlow}} register elements to the created
> graph. Datasets are the nodes of the dataflow graph, and are either tables or
> views, and flows are the edges connecting them.
> # {{StartRun}} starts a run, which is a single execution of a graph.
> # {{StopRun}} stops an existing run, while {{DropPipeline}} stops any
> current runs and drops the pipeline.
> # `PipelineCommand`, which contains a oneof that contains one of the above
> protos. This is the interface exposed to the SC command itself.
> We also need to add the new {{PipelineCommand}} object to the
> {{ExecutePlanRequest}} and the {{PipelineCommand.Response}} to the
> {{ExecutePlanResponse}} object.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
