Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51918543 @ueshin If we are changing the default value of `ArrayType.containsNull` to `true`, we may also need to add logic to set that value based on data sources. In https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetConverter.scala#L78, currently, there is no pattern for `ArrayType.containsNull=true`. If we do not add logic to set `ArrayType.containsNull` or support null values in arrays in our Parquet support (I am not sure if Parquet can do this, but seems people are working on it for Hive https://issues.apache.org/jira/browse/HIVE-6994), seems some existing working cases will break.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org