[
https://issues.apache.org/jira/browse/SPARK-17878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567104#comment-15567104
]
Hossein Falaki commented on SPARK-17878:
----------------------------------------
That would require API change in SparkSQL. Otherwise, we need to split the
string passed to {{nullValue}} on a delimiter. Neither these are safe choices.
I think accepting {{nullValue1}}, {{nullValue2}}, etc (along with existing
{{nullValue}}) is:
* backwards compatible
* clear
* extensible for other options in future. E.g., quoteCharacter, etc.
> Support for multiple null values when reading CSV data
> ------------------------------------------------------
>
> Key: SPARK-17878
> URL: https://issues.apache.org/jira/browse/SPARK-17878
> Project: Spark
> Issue Type: Story
> Components: SQL
> Affects Versions: 2.0.1
> Reporter: Hossein Falaki
>
> There are CSV files out there with multiple values that are supposed to be
> interpreted as null. As a result, multiple spark users have asked for this
> feature built into the CSV data source. It can be easily implemented in a
> backwards compatible way:
> - Currently CSV data source supports an option named {{nullValue}}.
> - We can add logic in {{CSVOptions}} to understands option names that match
> {{nullValue[\d]}}. This way user can specify a query with multiple or one
> null value.
> {code}
> val df = spark.read.format("CSV").option("nullValue1",
> "-").option("nullValue2", "*")....
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
