Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/20929#discussion_r180291939
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -887,6 +887,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val IGNORE_NULL_FIELDS_STREAMING_SCHEMA_INFERENCE =
--- End diff --
* I was expecting a configuration for the JSON data source instead of a
Spark SQL conf. Say, how to implement this feature if the data source is
outside Spark?
* Since Spark SQL cannot control the behavior of external data source
implementations, the flag won't work unless external data sources recognize it.
This is another reason to put it under JSON data source.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
