niko86 commented on issue #10439: URL: https://github.com/apache/arrow-rs/issues/10439#issuecomment-5274711050
Withdrawing this — the fault is not in arrow-rs. Sorry for the noise, and thanks to anyone who spent time on it. ## Root cause pyarrow bundles **its own mimalloc** as its default memory pool (`pa.default_memory_pool().backend_name` → `mimalloc`). The reproducer above links a **second** mimalloc via the `mimalloc` crate, and `libmimalloc-sys` 0.1.49 defaults to **mimalloc v3**. Two co-resident v3 instances hand out overlapping memory — [microsoft/mimalloc#1287](https://github.com/microsoft/mimalloc/issues/1287), fixed on `dev3` but not in any stable release. [apache/datafusion-python#1607](https://github.com/apache/datafusion-python/issues/1607) is the same fault in the same shape. ## Controlled experiment The reproducer in the issue body, unchanged, at the versions originally reported (`pandas` 2.3.3, `pyarrow` 25.0.0, Python 3.12, macOS arm64) — now on `arrow` 59.2.0. **The only variable is the mimalloc major:** | `Cargo.toml` | result | |---|---| | `mimalloc = "0.1"` (→ v3.3.2, as filed) | **SIGSEGV, rc=139** — 2/2 | | `mimalloc = { version = "0.1", features = ["v2"] }` (→ v2.3.2) | **survives 500 iters** — 3/3 | arrow-rs is byte-identical across those two runs. A `polars` producer survives in both. ## Where my isolation reasoning went wrong Every "key isolation" bullet in the report is equally explained by allocator co-residency, which I never tested: - *"only manifests under mimalloc; the system allocator masks it"* — it needs **two** mimallocs. A cross-matrix of consumer allocator × pyarrow pool shows only mimalloc×mimalloc corrupts; `mimalloc × system`, `system × mimalloc` and `system × system` are all clean. - *"polars-produced streams are never affected"* — polars does not allocate from pyarrow's pool, so no co-resident heap is involved. That is not evidence about the consume path. - *"reproduces via arrow-rs's own `from_pyarrow_bound`, so pyo3-arrow is exonerated"* — correct, but it exonerates arrow-rs by the same argument. What both paths have in common is that **pyarrow is imported**, not which API consumed the stream. Two further checks in my own project, where this originally surfaced: an instrumented `GlobalAlloc` wrapping `MiMalloc` and watching pyarrow's buffer address **never fired** while corruption still occurred — so the Rust allocator never receives the foreign pointer, consistent with `Deallocation::Custom`'s drop arm being a no-op. And the corruption reproduces there with **no arrow-rs code on the path at all** — `import pyarrow` then importing the extension is sufficient to corrupt a subsequently-built `pa.table`. ## One honest caveat In *this* reproducer the arrow-rs consume call is **necessary** to trigger the crash: a control that keeps the co-residency and the heap churn but never calls `consume()` survives 500 iterations. I read that as arrow-rs being the workload that grows the Rust heap into the colliding region while pyarrow buffers are live, rather than as a defect in the consume path — the v2/v3 A/B above is what I'd weight, since it changes nothing else. Flagging it so maintainers can reach their own conclusion; happy to reopen with better evidence if anyone reads it differently. No action needed on arrow-rs. Closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
