[ 
https://issues.apache.org/jira/browse/SPARK-61469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61469:
-----------------------------------
    Labels: pull-request-available  (was: )

> Drive the future of a blocking call from the heap in Spark Connect Rust, so 
> that a request does not overflow a small stack of the calling thread
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61469
>                 URL: https://issues.apache.org/jira/browse/SPARK-61469
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> runtime::block_on and block_on_abortable of Spark Connect Rust move the 
> future they drive by value through several frames on the calling thread 
> (block_on, drive, the runtime's block_on, the closure of block_in_place). The 
> future of a request holds the state of its gRPC call and of its retries, tens 
> of KiB (the analyze_plan future is about 45-75 KiB), so each of those frames 
> keeps a copy of it on the calling thread unless the compiler inlines it. 
> Which frames are inlined depends on the code of the calling crate, so an 
> unrelated change can tip it: on macOS arm64 release builds, DataFrame::schema 
> of a shallow plan used about 355 KiB of the stack before the request was 
> cloned, and with other open changes (#495, #503, #515) about 526 KiB, so the 
> deep_request_stack_tests of SPARK-61400, which run on a 512 KiB thread (the 
> smallest stack a Python thread may have), overflow (SIGABRT/SIGBUS), even for 
> spark.range(1).schema().
> Fix: pin the future on the heap in block_on and block_on_abortable, so that 
> those frames hold only its pointer. The calling thread then uses about 127 
> KiB up to the request clone in schema(), and schema() and collect() of 
> shallow and 746-level plans pass down to a 192 KiB stack. The cost is one 
> heap allocation per blocking call.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to