[ 
https://issues.apache.org/jira/browse/SPARK-61837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61837:
-----------------------------------
    Labels: pull-request-available  (was: )

> Avoid per-row Python calls during result materialization
> --------------------------------------------------------
>
>                 Key: SPARK-61837
>                 URL: https://issues.apache.org/jira/browse/SPARK-61837
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Lavan Vivekanandasarma
>            Assignee: Lavan Vivekanandasarma
>            Priority: Major
>              Labels: pull-request-available
>
> The Rust-backed Python client invokes Python Row-construction helpers for 
> each result row, adding overhead during large result materialization. 
> Replacing these helpers with native tuple construction removes frequent 
> Python execution checkpoints and can starve background Python threads and 
> delay Ctrl+C during conversion. Releasing the GIL during the earlier RPC does 
> not cover this later conversion phase.
> The proposed change uses the native tuple constructor while preserving the 
> public tuple-subclass Row and shared field-name lists. It adds a cached 
> Python-frame checkpoint and explicit signal checks every 1,024 
> Row/nested-value conversion entries, including values inside large arrays and 
> maps. This preserves opportunities for thread progress and interruption. The 
> budget bounds conversion work, not elapsed time; GC, large scalar allocations 
> and user-defined callbacks can still cause pauses.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to