Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22787#discussion_r226871288
--- Diff: docs/sql-migration-guide-upgrade.md ---
@@ -11,6 +11,8 @@ displayTitle: Spark SQL Upgrading Guide
- In PySpark, when creating a `SparkSession` with
`SparkSession.builder.getOrCreate()`, if there is an existing `SparkContext`,
the builder was trying to update the `SparkConf` of the existing `SparkContext`
with configurations specified to the builder, but the `SparkContext` is shared
by all `SparkSession`s, so we should not update them. Since 3.0, the builder
comes to not update the configurations. This is the same behavior as Java/Scala
API in 2.3 and above. If you want to update them, you need to update them prior
to creating a `SparkSession`.
+ - In Spark version 2.4 and earlier, the parser of JSON data source
treats empty strings as null for some data types like `IntegerType`. For
`FloatType` and `DoubleType`, it fails on empty strings and throws exceptions.
Since Spark 3.0, we disallow empty strings and will throw exceptions for data
types except for `StringType` and `BinaryType`.
--- End diff --
In the rest of this PR, we are saying `non string type` while we are
mentioning `StringType` and `BinaryType` in this migration doc. Do we need to
mention `binary` type in the following lines, too?
-
https://github.com/apache/spark/pull/22787/files#diff-635e02b2d1ce4ad1675b0350ccac0c10R313
-
https://github.com/apache/spark/pull/22787/files#diff-635e02b2d1ce4ad1675b0350ccac0c10R320
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
