[
https://issues.apache.org/jira/browse/SPARK-53525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-53525:
----------------------------------
Parent Issue: SPARK-54357 (was: SPARK-51166)
> Spark Connect ArrowBatch Result Chunking
> ----------------------------------------
>
> Key: SPARK-53525
> URL: https://issues.apache.org/jira/browse/SPARK-53525
> Project: Spark
> Issue Type: Sub-task
> Components: Connect
> Affects Versions: 4.1.0
> Reporter: Xi Lyu
> Assignee: Xi Lyu
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Currently, we enforce gRPC message limits on both the client and the server.
> These limits are largely meant to protect both sides from potential OOMs by
> rejecting abnormally large messages. However, there are cases in which the
> server incorrectly sends oversized messages that exceed these limits and
> cause execution failures.
> Specifically, the large message issue from the server to the client we’re
> solving here, comes from the Arrow batch data in ExecutePlanResponse being
> too large. It’s caused by a single arrow row exceeding the 128MB message
> limit, and Arrow cannot partition further and return the single large row in
> one gRPC message.
> To improve Spark Connect stability, this PR implements chunking large Arrow
> batches when returning query results from the server to the client, ensuring
> each ExecutePlanResponse chunk remains within the size limit, and the chunks
> from a batch will be reassembled on the client when parsing as an arrow batch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
