[
https://issues.apache.org/jira/browse/SPARK-17878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567046#comment-15567046
]
Hyukjin Kwon commented on SPARK-17878:
--------------------------------------
Maybe it'd be nicer if options allow list or nested map (if possible). I
noticed {{read.csv}} in R can use {{na.strings}} as a list which is actually
currently being mapped to {{nullValue}} as string..
> Support for multiple null values when reading CSV data
> ------------------------------------------------------
>
> Key: SPARK-17878
> URL: https://issues.apache.org/jira/browse/SPARK-17878
> Project: Spark
> Issue Type: Story
> Components: SQL
> Affects Versions: 2.0.1
> Reporter: Hossein Falaki
>
> There are CSV files out there with multiple values that are supposed to be
> interpreted as null. As a result, multiple spark users have asked for this
> feature built into the CSV data source. It can be easily implemented in a
> backwards compatible way:
> - Currently CSV data source supports an option named {{nullValue}}.
> - We can add logic in {{CSVOptions}} to understands option names that match
> {{nullValue[\d]}}. This way user can specify a query with multiple or one
> null value.
> {code}
> val df = spark.read.format("CSV").option("nullValue1",
> "-").option("nullValue2", "*")....
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
