[
https://issues.apache.org/jira/browse/SPARK-48338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Milicevic updated SPARK-48338:
------------------------------------
Description:
Design doc for this feature is in attachment.
*High level example of Sql Script:*
{code:java}
BEGIN
DECLARE c INT = 10;
WHILE c > 0 DO
INSERT INTO tscript VALUES (c);
SET c = c - 1;
END WHILE;
END{code}
*High level motivation behind this feature:*
SQL Scripting gives customers the ability to develop complex ETL and analysis
entirely in SQL. Until now, customers have had to write verbose SQL statements
or combine SQL + Python to efficiently write business logic. Coming from
another system, customers have to choose whether or not they want to migrate to
pyspark. Some customers end up not using Spark because of this gap. SQL
Scripting is a key milestone towards enabling SQL practitioners to write
sophisticated queries, without the need to use pyspark. Further, SQL Scripting
is a necessary step towards support for SQL Stored Procedures, and along with
SQL Variables (released) and Temp Tables (in progress), will allow for more
seamless data warehouse migrations.
*Work items classification:*
* M0 - basic support
* M1 - features and changes required to enable SQL Scripting by default
* M2 - follow-up improvements and additional functionalities that are
non-fundamental and should not block M1
* M3 - potential improvements for the future, need investigation
was:
Design doc for this feature is in attachment.
*High level example of Sql Script:*
{code:java}
BEGIN
DECLARE c INT = 10;
WHILE c > 0 DO
INSERT INTO tscript VALUES (c);
SET c = c - 1;
END WHILE;
END{code}
*High level motivation behind this feature:*
SQL Scripting gives customers the ability to develop complex ETL and analysis
entirely in SQL. Until now, customers have had to write verbose SQL statements
or combine SQL + Python to efficiently write business logic. Coming from
another system, customers have to choose whether or not they want to migrate to
pyspark. Some customers end up not using Spark because of this gap. SQL
Scripting is a key milestone towards enabling SQL practitioners to write
sophisticated queries, without the need to use pyspark. Further, SQL Scripting
is a necessary step towards support for SQL Stored Procedures, and along with
SQL Variables (released) and Temp Tables (in progress), will allow for more
seamless data warehouse migrations.
*Work items classification:*
* M0 - basic support
* M1 - features and changes required to enable SQL Scripting by default
* M2 - follow-ups and additional functionalities that are non-fundamental and
should not block M1
* M3 - potential improvements for the future, need investigation
> Sql Scripting support for Spark SQL
> -----------------------------------
>
> Key: SPARK-48338
> URL: https://issues.apache.org/jira/browse/SPARK-48338
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Aleksandar Tomic
> Priority: Major
> Labels: pull-request-available
> Attachments: Sql Scripting - OSS.odt, [Design Doc] Sql Scripting -
> OSS.pdf
>
>
> Design doc for this feature is in attachment.
> *High level example of Sql Script:*
> {code:java}
> BEGIN
> DECLARE c INT = 10;
> WHILE c > 0 DO
> INSERT INTO tscript VALUES (c);
> SET c = c - 1;
> END WHILE;
> END{code}
> *High level motivation behind this feature:*
> SQL Scripting gives customers the ability to develop complex ETL and analysis
> entirely in SQL. Until now, customers have had to write verbose SQL
> statements or combine SQL + Python to efficiently write business logic.
> Coming from another system, customers have to choose whether or not they want
> to migrate to pyspark. Some customers end up not using Spark because of this
> gap. SQL Scripting is a key milestone towards enabling SQL practitioners to
> write sophisticated queries, without the need to use pyspark. Further, SQL
> Scripting is a necessary step towards support for SQL Stored Procedures, and
> along with SQL Variables (released) and Temp Tables (in progress), will allow
> for more seamless data warehouse migrations.
> *Work items classification:*
> * M0 - basic support
> * M1 - features and changes required to enable SQL Scripting by default
> * M2 - follow-up improvements and additional functionalities that are
> non-fundamental and should not block M1
> * M3 - potential improvements for the future, need investigation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
