Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22367#discussion_r216196505 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala --- @@ -91,9 +91,10 @@ abstract class CSVDataSource extends Serializable { } row.zipWithIndex.map { case (value, index) => - if (value == null || value.isEmpty || value == options.nullValue) { - // When there are empty strings or the values set in `nullValue`, put the - // index as the suffix. + if (value == null || value.isEmpty || value == options.nullValue || + value == options.emptyValueInRead) { --- End diff -- @MaxGekk, can we get rid of this one (and the one in `CSVInferSchema.scala`) for now since we target 2.4? IIRC (need to double check), this behaviour by `makeSafeHeader` is from R's `read.csv`. We should check if it is coherent or not.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org