[jira] [Closed] (SPARK-20681) DataFram.Drop doesn't take effective, neither does error

2017-07-11 Thread Xi Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xi Wang closed SPARK-20681.
---
Resolution: Fixed

> DataFram.Drop doesn't take effective, neither does error
> 
>
> Key: SPARK-20681
> URL: https://issues.apache.org/jira/browse/SPARK-20681
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
>Reporter: Xi Wang
>Priority: Critical
>
> I am running the following code trying to drop nested columns,  but it 
> doesn't work, it doesn't return error either.
> *I read the DF from this json:*
> {'parent':{'child':{'grandchild':{'val':'1',' val_to_be_deleted':'0'
> scala> spark.read.format("json").load("c:/tmp/spark_issue.json")
> res0: org.apache.spark.sql.DataFrame = [father: struct struct>>]
> *read the df:*
> scala> res0.printSchema
> root
> |-- parent: struct (nullable = true)
> ||-- child: struct (nullable = true)
> |||-- grandchild: struct (nullable = true)
> ||||-- val: long (nullable = true)
> ||||-- val_to_be_deleted: long (nullable = true)
> *drop the column (I tried different ways, "quote", `back-tick`, col(object) 
> ...) column remains anyway:*
> scala> res0.drop(col("father.child.grandchild.val_to_be_deleted")).printSchema
> root
> |-- father: struct (nullable = true)
> ||-- child: struct (nullable = true)
> |||-- grandchild: struct (nullable = true)
> ||||-- val: long (nullable = true)
> ||||-- val_to_be_deleted: long (nullable = true)
> scala> res0.drop("father.child.grandchild.val_to_be_deleted").printSchema
> root
> |-- father: struct (nullable = true)
> ||-- child: struct (nullable = true)
> |||-- grandchild: struct (nullable = true)
> ||||-- val: long (nullable = true)
> ||||-- val_to_be_deleted: long (nullable = true)
> Any help is appreciated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-20681) DataFram.Drop doesn't take effective, neither does error

2017-05-09 Thread Xi Wang (JIRA) 
Xi Wang created SPARK-20681:
---

 Summary: DataFram.Drop doesn't take effective, neither does error
 Key: SPARK-20681
 URL: https://issues.apache.org/jira/browse/SPARK-20681
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.1.0
Reporter: Xi Wang
Priority: Critical


I am running the following code trying to drop nested columns,  but it doesn't 
work, it doesn't return error either.

*I read the DF from this json:*
{'parent':{'child':{'grandchild':{'val':'1',' val_to_be_deleted':'0'
scala> spark.read.format("json").load("c:/tmp/spark_issue.json")
res0: org.apache.spark.sql.DataFrame = [father: struct>>]

*read the df:*
scala> res0.printSchema
root
|-- parent: struct (nullable = true)
||-- child: struct (nullable = true)
|||-- grandchild: struct (nullable = true)
||||-- val: long (nullable = true)
||||-- val_to_be_deleted: long (nullable = true)


*drop the column (I tried different ways, "quote", `back-tick`, col(object) 
...) column remains anyway:*
scala> res0.drop(col("father.child.grandchild.val_to_be_deleted")).printSchema
root
|-- father: struct (nullable = true)
||-- child: struct (nullable = true)
|||-- grandchild: struct (nullable = true)
||||-- val: long (nullable = true)
||||-- val_to_be_deleted: long (nullable = true)


scala> res0.drop("father.child.grandchild.val_to_be_deleted").printSchema
root
|-- father: struct (nullable = true)
||-- child: struct (nullable = true)
|||-- grandchild: struct (nullable = true)
||||-- val: long (nullable = true)
||||-- val_to_be_deleted: long (nullable = true)



Any help is appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org