[ https://issues.apache.org/jira/browse/SPARK-38087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486167#comment-17486167 ]
Dongjoon Hyun commented on SPARK-38087: --------------------------------------- I removed the fixed version field, [~deepa.vasanthkumar]. > select doesnt validate if the column already exists > --------------------------------------------------- > > Key: SPARK-38087 > URL: https://issues.apache.org/jira/browse/SPARK-38087 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.2.1 > Environment: Version{{{}v3.2.1{}}} > {{}} > {{{}{}}}Master{{{}local[*]{}}} > {{(Reproducible in any environment)}} > Reporter: Deepa Vasanthkumar > Priority: Minor > Attachments: select vs drop.png > > > > Select doesnt validate whether the alias column is already present in the > dataframe. > After which, we cannot do anything in that dataframe on that column. > df4 = df2.select(df2.firstname, df2.lastname) --> throws analysis exception > df4.show() > > However drop will not let you drop the said column. > > Scenario to reproduce : > df2 = df1.select("*", (df1.firstname).alias("firstname")) ---> this will > add same column > df2.show() > df2.drop(df2.firstname) --> this will give AnalysisException: Reference > 'firstname' is ambiguous, could be: firstname, firstname. > > > Is this expected behavior . > !select vs drop.png! > !image-2022-02-02-06-28-23-543.png! > > > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org