[ https://issues.apache.org/jira/browse/SPARK-31123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikel San Vicente updated SPARK-31123: -------------------------------------- Description: Hi, I am seeing a really strange behaviour in drop method after a join with aliases. It doesn't seem to find the column when I reference to it using dataframe("columnName") syntax, but it does work with other combinators like select {code:java} case class Record(a: String, dup: String) case class Record2(b: String, dup: String) val df = Seq(Record("a", "dup")).toDF val joined = df.alias("a").join(df2.alias("b"), df("a") === df2("b")) val dupCol = df("dup") joined.drop(dupCol) // Does not drop anything joined.drop(func.col("a.dup")) // It drops the column joined.select(dupCol) // It selects the column {code} was: Hi, I am seeing a really strange behaviour in drop method after a join with aliases. It doesn't seem to find the column when I reference to it using dataframe("columnName") syntax, but it does work with other combinators like select {code:java} case class Record(a: String, dup: String) case class Record2(b: String, dup: String) val df = Seq(Record("a", "dup")).toDF val joined = df.alias("a").join(df2.alias("b"), df("a") === df2("b")) val dupCol = df("dup") joined.drop(dupCol) // Does not drop anything joined.drop(func.col("a.dup")) // It works! joined.select(dupCol) // It works! {code} > Drop does not work after join with aliases > ------------------------------------------ > > Key: SPARK-31123 > URL: https://issues.apache.org/jira/browse/SPARK-31123 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.2 > Reporter: Mikel San Vicente > Priority: Minor > > > Hi, > I am seeing a really strange behaviour in drop method after a join with > aliases. It doesn't seem to find the column when I reference to it using > dataframe("columnName") syntax, but it does work with other combinators like > select > {code:java} > case class Record(a: String, dup: String) > case class Record2(b: String, dup: String) > val df = Seq(Record("a", "dup")).toDF > val joined = df.alias("a").join(df2.alias("b"), df("a") === df2("b")) > val dupCol = df("dup") > joined.drop(dupCol) // Does not drop anything > joined.drop(func.col("a.dup")) // It drops the column > joined.select(dupCol) // It selects the column > {code} > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org