[
https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593119#comment-15593119
]
Suresh Thalamati commented on SPARK-17916:
------------------------------------------
Thank you for trying out the different scenarios. I think output you are
getting after setting he quote to empty is not what is expected in the case.
You want "" to be recognized as empty string, not actual quotes in the output.
Example (Before my changes on 2.0.1 branch):
input:
col1,col2
1,"-"
2,""
3,
4,"A,B"
val df = spark.read.format("csv").option("nullValue", "\"-\"").option("quote",
"").option("header", true).load("/Users/suresht/sparktests/emptystring.csv")
df: org.apache.spark.sql.DataFrame = [col1: string, col2: string]
scala> df.selectExpr("length(col2)").show
+------------+
|length(col2)|
+------------+
| null|
| 2|
| null|
| 2|
+------------+
scala> df.show
+----+----+
|col1|col2|
+----+----+
| 1|null|
| 2| ""|
| 3|null|
| 4| "A|
+----+----+
> CSV data source treats empty string as null no matter what nullValue option is
> ------------------------------------------------------------------------------
>
> Key: SPARK-17916
> URL: https://issues.apache.org/jira/browse/SPARK-17916
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.1
> Reporter: Hossein Falaki
>
> When user configures {{nullValue}} in CSV data source, in addition to those
> values, all empty string values are also converted to null.
> {code}
> data:
> col1,col2
> 1,"-"
> 2,""
> {code}
> {code}
> spark.read.format("csv").option("nullValue", "-")
> {code}
> We will find a null in both rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
