This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push: new 1c7d78042f9 [SPARK-42731][CONNECT][DOCS] Document Spark Connect configurations 1c7d78042f9 is described below commit 1c7d78042f94b8177609d1f8c4ce2617432715c7 Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Wed Mar 15 08:40:30 2023 +0900 [SPARK-42731][CONNECT][DOCS] Document Spark Connect configurations ### What changes were proposed in this pull request? This PR proposes to document the configuration of Spark Connect defined in https://github.com/apache/spark/blob/master/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala ### Why are the changes needed? To let users know which configuration are supported for Spark Connect. ### Does this PR introduce _any_ user-facing change? Yes, it documents the configurations for Spark Connect. ### How was this patch tested? Linters in CI should verify this change. Also manually built the docs as below: ![Screen Shot 2023-03-14 at 8 24 51 PM](https://user-images.githubusercontent.com/6477701/224986645-3e3abfe3-4f6b-4810-8887-24cf24532f5e.png) Closes #40416 from HyukjinKwon/SPARK-42731. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> (cherry picked from commit e986fb0767eb64f1db6cf30f8c9f3e01c192171d) Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- docs/configuration.md | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 932cd4d503c..0ca53acbeb6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -3138,6 +3138,69 @@ like shuffle, just replace "rpc" with "shuffle" in the property names except The default value for number of thread-related config keys is the minimum of the number of cores requested for the driver or executor, or, in the absence of that value, the number of cores available for the JVM (with a hardcoded upper limit of 8). +### Spark Connect + +#### Server Configuration + +Server configurations are set in Spark Connect server, for example, when you start the Spark Connect server with `./sbin/start-connect-server.sh`. +They are typically set via the config file and command-lineoptions with `--conf/-c`. + +<table class="table"> +<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr> +<tr> + <td><code>spark.connect.grpc.binding.port</code></td> + <td> + 15002 + </td> + <td>Port for Spark Connect server to bind.</td> + <td>3.4.0</td> +</tr> +<tr> + <td><code>spark.connect.grpc.interceptor.classes</code></td> + <td> + (none) + </td> + <td>Comma separated list of class names that must implement the <code>io.grpc.ServerInterceptor</code> interface</td> + <td>3.4.0</td> +</tr> +<tr> + <td><code>spark.connect.grpc.arrow.maxBatchSize</code></td> + <td> + 4m + </td> + <td>When using Apache Arrow, limit the maximum size of one arrow batch that can be sent from server side to client side. Currently, we conservatively use 70% of it because the size is not accurate but estimated.</td> + <td>3.4.0</td> +</tr> +<tr> + <td><code>spark.connect.extensions.relation.classes</code></td> + <td> + (none) + </td> + <td>Comma separated list of classes that implement the trait <code>org.apache.spark.sql.connect.plugin.RelationPlugin</code> to support custom +Relation types in proto.</td> + <td>3.4.0</td> +</tr> +<tr> + <td><code>spark.connect.extensions.expression.classes</code></td> + <td> + (none) + </td> + <td>Comma separated list of classes that implement the trait +<code>org.apache.spark.sql.connect.plugin.ExpressionPlugin</code> to support custom +Expression types in proto.</td> + <td>3.4.0</td> +</tr> +<tr> + <td><code>spark.connect.extensions.command.classes</code></td> + <td> + (none) + </td> + <td>Comma separated list of classes that implement the trait +<code>org.apache.spark.sql.connect.plugin.CommandPlugin</code> to support custom +Command types in proto.</td> + <td>3.4.0</td> +</tr> +</table> ### Security --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org