[ https://issues.apache.org/jira/browse/SPARK-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021093#comment-16021093 ]
lyc commented on SPARK-20681: ----------------------------- As said in spark source code, `drop` can only be used to drop top level columns. Why did you think this will work? > DataFram.Drop doesn't take effective, neither does error > -------------------------------------------------------- > > Key: SPARK-20681 > URL: https://issues.apache.org/jira/browse/SPARK-20681 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.0 > Reporter: Xi Wang > Priority: Critical > > I am running the following code trying to drop nested columns, but it > doesn't work, it doesn't return error either. > *I read the DF from this json:* > {'parent':{'child':{'grandchild':{'val':'1',' val_to_be_deleted':'0'}}}} > scala> spark.read.format("json").load("c:/tmp/spark_issue.json") > res0: org.apache.spark.sql.DataFrame = [father: struct<child: > struct<grandchild: struct<val: bigint, val_to_be_deleted: bigint>>>] > *read the df:* > scala> res0.printSchema > root > |-- parent: struct (nullable = true) > | |-- child: struct (nullable = true) > | | |-- grandchild: struct (nullable = true) > | | | |-- val: long (nullable = true) > | | | |-- val_to_be_deleted: long (nullable = true) > *drop the column (I tried different ways, "quote", `back-tick`, col(object) > ...) column remains anyway:* > scala> res0.drop(col("father.child.grandchild.val_to_be_deleted")).printSchema > root > |-- father: struct (nullable = true) > | |-- child: struct (nullable = true) > | | |-- grandchild: struct (nullable = true) > | | | |-- val: long (nullable = true) > | | | |-- val_to_be_deleted: long (nullable = true) > scala> res0.drop("father.child.grandchild.val_to_be_deleted").printSchema > root > |-- father: struct (nullable = true) > | |-- child: struct (nullable = true) > | | |-- grandchild: struct (nullable = true) > | | | |-- val: long (nullable = true) > | | | |-- val_to_be_deleted: long (nullable = true) > Any help is appreciated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org