dongjoon-hyun opened a new pull request, #505:
URL: https://github.com/apache/spark-connect-swift/pull/505
### What changes were proposed in this pull request?
This PR proposes to support the `grpc_max_message_size` parameter in the
Spark
Connect connection string.
- `SparkConnectClient.init(remote:)` parses it now. Previously it fell
through to
the `default` branch and only printed `Unknown parameter: ...`, leaving the
existing `URIParams.PARAM_GRPC_MAX_MESSAGE_SIZE` constant unused.
- A non-positive or non-integer value throws
`SparkConnectError.InvalidArgument`,
like the other invalid inputs in the same initializer.
- The value is applied to the gRPC transport through a default `MethodConfig`
(an empty service name applies to all services and methods) in a
`ServiceConfig`,
at both places creating the transport: `SparkConnectClient.withGPRC` and
`DataFrame.withGPRC`. Both `maxRequestMessageBytes` and
`maxResponseMessageBytes`
are set, matching PySpark which sets both `grpc.max_send_message_length`
and
`grpc.max_receive_message_length`.
When the parameter is absent, an empty `ServiceConfig` is used and the
behavior is
unchanged (gRPC's 4MiB default).
### Why are the changes needed?
`grpc_max_message_size` is supported by the PySpark and Scala clients and is
used to
avoid hitting the default gRPC message size limit when collecting large
results.
The Swift client silently dropped it, so users had no way to raise the limit.
### Does this PR introduce _any_ user-facing change?
Yes. `sc://localhost:15002/;grpc_max_message_size=268435456` is now honored
instead
of ignored, and an invalid value such as `grpc_max_message_size=abc` now
throws
`SparkConnectError.InvalidArgument` instead of being ignored.
### How was this patch tested?
Pass the CIs with the newly added test cases in `SparkConnectClientTests`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
