[ https://issues.apache.org/jira/browse/SPARK-43439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Frederik Paradis updated SPARK-43439: ------------------------------------- Description: When passing a string to the drop method, if the string contains an alias, the column is not dropped. However, passing a column object with the same name and alias, it works. {code:python} from pyspark.sql import SparkSession import pyspark.sql.functions as F spark = SparkSession.builder.master("local[1]").appName("local-spark-session").getOrCreate() df = spark.createDataFrame([(1, 10)], ["any", "hour"]).alias("a") j = df.drop("a.hour") print(j) # DataFrame[any: bigint, hour: bigint] jj = df.drop(F.col("a.hour")) print(jj) # DataFrame[any: bigint] {code} Related issues: https://issues.apache.org/jira/browse/SPARK-31123 https://issues.apache.org/jira/browse/SPARK-14759 was: When passing a string to the drop method, if the string contains an alias, the column is not dropped. However, passing a column object with the same name and alias, it works. {code:python} from pyspark.sql import SparkSession import pyspark.sql.functions as F spark = SparkSession.builder.master("local[1]").appName("local-spark-session").getOrCreate() df = spark.createDataFrame([(1, 10)], ["any", "hour"]).alias("a") j = df.drop("a.hour") print(j) # DataFrame[any: bigint, hour: bigint] jj = df.drop(F.col("a.hour")) print(jj) # DataFrame[any: bigint] {code} Related issues: https://issues.apache.org/jira/browse/SPARK-31123 https://issues.apache.org/jira/browse/SPARK-14759 > Drop does not work when passed a string with an alias > ----------------------------------------------------- > > Key: SPARK-43439 > URL: https://issues.apache.org/jira/browse/SPARK-43439 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 3.3.2 > Reporter: Frederik Paradis > Priority: Major > > When passing a string to the drop method, if the string contains an alias, > the column is not dropped. However, passing a column object with the same > name and alias, it works. > {code:python} > from pyspark.sql import SparkSession > import pyspark.sql.functions as F > spark = > SparkSession.builder.master("local[1]").appName("local-spark-session").getOrCreate() > df = spark.createDataFrame([(1, 10)], ["any", "hour"]).alias("a") > j = df.drop("a.hour") > print(j) # DataFrame[any: bigint, hour: bigint] > jj = df.drop(F.col("a.hour")) > print(jj) # DataFrame[any: bigint] > {code} > > Related issues: > https://issues.apache.org/jira/browse/SPARK-31123 > https://issues.apache.org/jira/browse/SPARK-14759 > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org