GitHub user hvanhovell opened a pull request:
https://github.com/apache/spark/pull/14938
Fix ArrayType and MapType CatalogString.
## What changes were proposed in this pull request?
the `catalogString` for `ArrayType` and `MapType` currently calls the
`simpleString` method on its children. This is a problem when the child is a
struct, the `struct.simpleString` implementation truncates the number of fields
it shows (25 at max). This breaks the generation of a proper `catalogString`,
and has shown to cause errors while writing to Hive.
This PR fixes this by providing proper `catalogString` implementations for
`ArrayData` or `MapData`.
## How was this patch tested?
Added testing for `catalogString` to `DataTypeSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hvanhovell/spark SPARK-17335
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14938.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14938
----
commit 7ab9de0770387b10e5e5094ac1f1ee95d8a53f1b
Author: Herman van Hovell <hvanhov...@databricks.com>
Date: 2016-09-02T15:05:20Z
Fix ArrayType and MapType catalogString.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
