Anish Mahto created SPARK-52393:
-----------------------------------
Summary: [SDP] Pipeline SQL Graph Element Registration
Key: SPARK-52393
URL: https://issues.apache.org/jira/browse/SPARK-52393
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.1.0
Reporter: Anish Mahto
Add functionality to register graph elements (tables, views, flows) in a
declarative pipeline's DataflowGraph object, from the SQL files sent on
`DefineSqlDataset` requests to the spark connect backend.
This involves parsing the SQL text, interpreting the extracted logical plans,
and constructing appropriate graph element objects (Table, View, Flow).
The consequence is when a pipeline is eventually run, the registered graph
elements from SQL files will actually materialize and produce the correct
streaming table/materialized view/temporary view during execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
