Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/22238#discussion_r213123711 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -2812,6 +2812,12 @@ See [Input Sources](#input-sources) and [Output Sinks](#output-sinks) sections f # Additional Information +**Gotchas** --- End diff -- I was going to add the explanation to `doc()` of `spark.sql.shuffle.partitions`, but looks like what we explained in `doc()` would not be published automatically. (Please correct me if I'm missing here.) SQLConf is even not exposed to scaladoc. That's why I'm adding this to structured streaming guide doc. Actually I think most of end users only take a look at this doc for structured streaming, and we can't (and shouldn't) expect end users to take a look at source code to find it. But also actually I didn't notice that `spark.sql.shuffle.partitions` is explained in `sql-programming-guide.md` but I also think we need to explain all configs here if they work differently with batch query. `spark.sql.shuffle.partitions` is the case. Btw, `Gotchas` looks like funny though. Maybe having section would be better. Maybe like `## Other Configuration Options` in `sql-programming-guide.md`?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org