[
https://issues.apache.org/jira/browse/SPARK-38087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated SPARK-38087:
-----------------------------------
Component/s: SQL
(was: Spark Core)
> select doesnt validate if the column already exists
> ---------------------------------------------------
>
> Key: SPARK-38087
> URL: https://issues.apache.org/jira/browse/SPARK-38087
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.1
> Environment: Version{{{}v3.2.1{}}}
> {{}}
> {{{}{}}}Master{{{}local[*]{}}}
> {{(Reproducible in any environment)}}
> Reporter: Deepa Vasanthkumar
> Priority: Minor
> Attachments: select vs drop.png
>
>
>
> Select doesnt validate whether the alias column is already present in the
> dataframe.
> After which, we cannot do anything in that dataframe on that column.
> df4 = df2.select(df2.firstname, df2.lastname) --> throws analysis exception
> df4.show()
>
> However drop will not let you drop the said column.
>
> Scenario to reproduce :
> df2 = df1.select("*", (df1.firstname).alias("firstname")) ---> this will
> add same column
> df2.show()
> df2.drop(df2.firstname) --> this will give AnalysisException: Reference
> 'firstname' is ambiguous, could be: firstname, firstname.
>
>
> Is this expected behavior .
> !select vs drop.png!
> !image-2022-02-02-06-28-23-543.png!
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
