[
https://issues.apache.org/jira/browse/SPARK-38087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepa Vasanthkumar updated SPARK-38087:
---------------------------------------
Description:
Select doesnt validate whether the alias column is already present in the
dataframe.
However drop will not let you drop the said column.
Scenario to reproduce :
df2 = df1.select("*", (df1.firstname).alias("firstname")) ---> this will add
same column
df2.show()
df2.drop(df2.firstname) --> this will give AnalysisException: Reference
'firstname' is ambiguous, could be: firstname, firstname.
Is this expected behavior .
!select vs drop.png!
!image-2022-02-02-06-28-23-543.png!
was:
Select doesnt validate whether the alias column is already present in the
dataframe.
However drop will not let you drop the said column.
Scenario to reproduce :
df2 = df1.select("*", (df1.firstname).alias("firstname")) ---> this will add
same column
df2.show()
df2.drop(df2.firstname) --> this will give AnalysisException: Reference
'firstname' is ambiguous, could be: firstname, firstname.
Is this expected behavior .
!image-2022-02-02-06-28-23-543.png!
> select doesnt validate if the column already exists
> ---------------------------------------------------
>
> Key: SPARK-38087
> URL: https://issues.apache.org/jira/browse/SPARK-38087
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.2.1
> Environment: Version{{{}v3.2.1{}}}
> {{}}
> {{{}{}}}Master{{{}local[*]{}}}
> {{(Reproducible in any environment)}}
> Reporter: Deepa Vasanthkumar
> Priority: Minor
> Fix For: 3.3
>
> Attachments: select vs drop.png
>
>
>
> Select doesnt validate whether the alias column is already present in the
> dataframe.
> However drop will not let you drop the said column.
>
> Scenario to reproduce :
> df2 = df1.select("*", (df1.firstname).alias("firstname")) ---> this will
> add same column
> df2.show()
> df2.drop(df2.firstname) --> this will give AnalysisException: Reference
> 'firstname' is ambiguous, could be: firstname, firstname.
>
>
> Is this expected behavior .
> !select vs drop.png!
> !image-2022-02-02-06-28-23-543.png!
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
