[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584357#comment-16584357 ]
Maxim Gekk commented on SPARK-17916: ------------------------------------ > he default behavior in 2.3.x for csv format is that when i write out null >value, it comes back in as null. when i write out empty string, it also comes >back in as null. [~koert] Please, have a look at the added test: [https://github.com/apache/spark/pull/21273/files#diff-219ac8201e443435499123f96e94d29fR1355] . It checks exactly what you described. If you have something different, please, leave the code here. > CSV data source treats empty string as null no matter what nullValue option is > ------------------------------------------------------------------------------ > > Key: SPARK-17916 > URL: https://issues.apache.org/jira/browse/SPARK-17916 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.1 > Reporter: Hossein Falaki > Assignee: Maxim Gekk > Priority: Major > Fix For: 2.4.0 > > > When user configures {{nullValue}} in CSV data source, in addition to those > values, all empty string values are also converted to null. > {code} > data: > col1,col2 > 1,"-" > 2,"" > {code} > {code} > spark.read.format("csv").option("nullValue", "-") > {code} > We will find a null in both rows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org