Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/21952 Hi @dbtsai , nice catch! I think we can also check the nullability here: https://github.com/apache/spark/pull/21952/files#diff-01fea32e6ec6bcf6f34d06282e08705aR160 If the input data is from data source, I doubt the improvement of this PR. As the data schema is always nullable for data source: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L379 Anyway we should add these checks.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org