[
https://issues.apache.org/jira/browse/SPARK-61837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61837:
------------------------------------
Assignee: Lavan Vivekanandasarma
> Avoid per-row Python calls during result materialization
> --------------------------------------------------------
>
> Key: SPARK-61837
> URL: https://issues.apache.org/jira/browse/SPARK-61837
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Lavan Vivekanandasarma
> Assignee: Lavan Vivekanandasarma
> Priority: Major
>
> The Rust-backed Python client invokes Python Row-construction helpers for
> each result row, adding overhead during large result materialization.
> Replacing these helpers with native tuple construction removes frequent
> Python execution checkpoints and can starve background Python threads and
> delay Ctrl+C during conversion. Releasing the GIL during the earlier RPC does
> not cover this later conversion phase.
> The proposed change uses the native tuple constructor while preserving the
> public tuple-subclass Row and shared field-name lists. It adds a cached
> Python-frame checkpoint and explicit signal checks every 1,024
> Row/nested-value conversion entries, including values inside large arrays and
> maps. This preserves opportunities for thread progress and interruption. The
> budget bounds conversion work, not elapsed time; GC, large scalar allocations
> and user-defined callbacks can still cause pauses.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]