[
https://issues.apache.org/jira/browse/SPARK-48338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085966#comment-18085966
]
Wenchen Fan commented on SPARK-48338:
-------------------------------------
We should also probably make the rule clear. This umbrella ticket has subtasks
with fixed versions from 4.0 to 4.2. A major SQL feature like scripting can
have endless subtasks, but we can't keep the umbrella open forever. So my
intuition is to resolve the umbrella ticket once the feature is available to
users (turn on by default in this case, which is 4.1.0)
However, what if people keep creating new subtasks under a resolved umbrella
ticket? shall we allow it or not? I don't have a strong opinion, Jira system
does not forbid it, so I treat it as personal preference. But the bottom line
is: the umbrella ticket has been mentioned already in the previous release, so
new subtasks won't trigger another mentioning of the umbrella ticket in the new
release notes.
> Sql Scripting support for Spark SQL
> -----------------------------------
>
> Key: SPARK-48338
> URL: https://issues.apache.org/jira/browse/SPARK-48338
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Aleksandar Tomic
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
> Attachments: Sql Scripting - OSS.odt, [Design Doc] Sql Scripting -
> OSS.pdf
>
>
> Design doc for this feature is in attachment.
> *High level example of Sql Script:*
> {code:java}
> BEGIN
> DECLARE c INT = 10;
> WHILE c > 0 DO
> INSERT INTO tscript VALUES (c);
> SET c = c - 1;
> END WHILE;
> END{code}
> *High level motivation behind this feature:*
> SQL Scripting gives customers the ability to develop complex ETL and analysis
> entirely in SQL. Until now, customers have had to write verbose SQL
> statements or combine SQL + Python to efficiently write business logic.
> Coming from another system, customers have to choose whether or not they want
> to migrate to pyspark. Some customers end up not using Spark because of this
> gap. SQL Scripting is a key milestone towards enabling SQL practitioners to
> write sophisticated queries, without the need to use pyspark. Further, SQL
> Scripting is a necessary step towards support for SQL Stored Procedures, and
> along with SQL Variables (released) and Temp Tables (in progress), will allow
> for more seamless data warehouse migrations.
> *Work items classification:*
> * M0 - basic support
> * M1 - features and changes required to enable SQL Scripting by default
> * M2 - follow-up improvements and additional functionalities that are
> non-fundamental and should not block M1
> * M3 - potential improvements for the future, need investigation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
