GitHub user stephend-realitymine opened a pull request:
https://github.com/apache/spark/pull/9245
[SPARK-10947] [SQL] With schema inference from JSON into a Dataframe, add
option to infer all primitive object types as strings
Currently, when a schema is inferred from a JSON file using
sqlContext.read.json, the primitive object types are inferred as string, long,
boolean, etc.
However, if the inferred type is too specific (JSON obviously does not
enforce types itself), this can cause issues with merging dataframe schemas.
This pull request adds the option "primitivesAsString" to the JSON
DataFrameReader which when true (defaults to false if not set) will infer all
primitives as strings.
Below is an example usage of this new functionality.
```
val jsonDf = sqlContext.read.option("primitivesAsString",
"true").json(primitiveFieldAndType)
scala> jsonDf.printSchema()
root
|-- bigInteger: string (nullable = true)
|-- boolean: string (nullable = true)
|-- double: string (nullable = true)
|-- integer: string (nullable = true)
|-- long: string (nullable = true)
|-- null: string (nullable = true)
|-- string: string (nullable = true)
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/RealityMineLtd/spark
stephend-primitivesAsString
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9245.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9245
----
commit 79b68a886a3a6e324e682709af346e599b3efd57
Author: RealityMine Ltd Coordinator
<realitymine-coordina...@users.noreply.github.com>
Date: 2015-10-06T11:16:13Z
Merge pull request #1 from apache/master
Resyncing to master
commit ec74be6f92ef4940bacb9455989c473ed3a1539a
Author: Stephen De Gennaro <steph...@realitymine.com>
Date: 2015-10-15T12:19:34Z
SPARK-10947 Added option to json schema primativesAsString when true will
infer primative types as strings
commit 8a879c80a87e1bf9fe17fd58b18bce36294bc17b
Author: Ewan Leith <ewan.le...@realitymine.com>
Date: 2015-10-22T16:07:39Z
Fixing spelling of primitive from primative
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
