[
https://issues.apache.org/jira/browse/SPARK-54914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shubham_bhusate updated SPARK-54914:
------------------------------------
Description:
The pipe syntax DROP operator currently fails with a parse error when trying to
drop nested struct fields using qualified names (e.g., `drop col.i1`).
*Current Behavior:*
>>> spark.sql("""
... CREATE OR REPLACE TEMP VIEW st AS
... SELECT 1 as x, named_struct('i1', 2, 'i2', 3) as col
... """)
>>> spark.sql("""
... TABLE st
... |> DROP col.i1
... """)
Error: PARSE_SYNTAX_ERROR: Syntax error at or near '.'
*Expected Behavior:*
>>> spark.sql("""
... TABLE st
... |> DROP col.i1
... """)
Should succeed and return:
{+}--{-}{-}{+}-------+
| x| col|
{+}--{-}{-}{+}-------+
| 1|{i2: 3}|
{+}--{-}{-}{+}-------+
was:
The pipe syntax DROP operator currently fails with a parse error when trying to
drop nested struct fields using qualified names (e.g., `drop col.i1`).
*Current Behavior:*
>>> spark.sql("""
... CREATE OR REPLACE TEMP VIEW st AS
... SELECT 1 as x, named_struct('i1', 2, 'i2', 3) as col
... """)
DataFrame[]
>>> spark.sql("""
... TABLE st
... |> DROP col.i1
... """)
Error: PARSE_SYNTAX_ERROR: Syntax error at or near '.'
*Expected Behavior:*
>>> spark.sql("""
... TABLE st
... |> DROP col.i1
... """)
Should succeed and return:
+---+--------+
| x| col|
+---+--------+
| 1|\{i2: 3} |
+---+--------+
> Pipe Syntax - Qualified Column Names in DROP Operator
> -----------------------------------------------------
>
> Key: SPARK-54914
> URL: https://issues.apache.org/jira/browse/SPARK-54914
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: shubham_bhusate
> Priority: Major
> Labels: pull-request-available
>
> The pipe syntax DROP operator currently fails with a parse error when trying
> to drop nested struct fields using qualified names (e.g., `drop col.i1`).
> *Current Behavior:*
> >>> spark.sql("""
> ... CREATE OR REPLACE TEMP VIEW st AS
> ... SELECT 1 as x, named_struct('i1', 2, 'i2', 3) as col
> ... """)
> >>> spark.sql("""
> ... TABLE st
> ... |> DROP col.i1
> ... """)
> Error: PARSE_SYNTAX_ERROR: Syntax error at or near '.'
> *Expected Behavior:*
> >>> spark.sql("""
> ... TABLE st
> ... |> DROP col.i1
> ... """)
> Should succeed and return:
> {+}--{-}{-}{+}-------+
> | x| col|
> {+}--{-}{-}{+}-------+
> | 1|{i2: 3}|
> {+}--{-}{-}{+}-------+
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
