Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/14118 FYI, before [SPARK-14143](https://issues.apache.org/jira/browse/SPARK-14143), null values had been handled this way: : ```scala if (datum == options.nullValue && nullable && (!castType.isInstanceOf[StringType])) ``` Then in [SPARK-14143](https://issues.apache.org/jira/browse/SPARK-14143), it was first broken down into numeric data types in https://github.com/apache/spark/pull/11947/commits/93ac6bb3eb63efb775b48af090a37a6cbe4f30c4 to handle byte-specific null value, short-specific null value, int-specific null value, ... : ```scala case _: ByteType => if (datum == params.byteNullValue && nullable) null else datum.toByte case _: ShortType => if (datum == params.shortNullValue && nullable) null else datum.toShort case _: IntegerType => if (datum == params.integerNullValue && nullable) null else datum.toInt ... ``` then in https://github.com/apache/spark/pull/11947/commits/698b4b41baa1ebd5d66ea6242bcb39bcd0887f8b byte-specific null value, short-specific null value, int-specific null value, ... were reduced back to one single null value: ```scala case _: ByteType => if (datum == params.nullValue && nullable) null else datum.toByte case _: ShortType => if (datum == params.nullValue && nullable) null else datum.toShort case _: IntegerType => if (datum == params.nullValue && nullable) null else datum.toInt ``` Along with that change, we had introduced regression handling non-numeric data types like `BooleanType` etc. Since we don't need to handle type-specific null values, this patch switchs back to the way we handled null values in the 1.6 days (and thus fixes the regression): ```scala if (datum == options.nullValue && nullable && (!castType.isInstanceOf[StringType])) ```
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org