Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/20929#discussion_r180291983 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -887,6 +887,14 @@ object SQLConf { .booleanConf .createWithDefault(false) + val IGNORE_NULL_FIELDS_STREAMING_SCHEMA_INFERENCE = + buildConf("spark.sql.streaming.schemaInference.ignoreNullFields") --- End diff -- `ignoreNullFields` is not precise. For example ```json {"a": 1} {"a": null} ``` The `null` value is not ignored. I would suggest `dropFieldIfAllNull`.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org