pingzh opened a new pull request, #11873: URL: https://github.com/apache/arrow-rs/pull/11873
[Open Codex thread](https://chatgpt.com/codex/open-app?codex_thread_id=01a16a10-d957-7f83-96cf-a4ba164eab6c) # Rationale for this change Two individually valid batches can exceed a 32-bit Arrow offset when `BatchCoalescer` combines them. For example, combining two one-row `List<Null>` batches with 2^30 children each overflows the list offset. Flush the buffered rows before that limit so coalescing produces two valid batches while preserving row order and column alignment. # What changes are included in this PR? - Track cumulative logical spans for 32-bit offsets, including nested lists, maps, structs, and dictionary values, before modifying any column builder. - Flush and retry when another slice would exceed `i32::MAX`; pass unsupported offset layouts through as row-bounded slices. - Include the bookkeeping in retained-memory reporting and document that offset limits can produce smaller batches. # Are these changes tested? Four focused regressions cover the actual 32-bit list-offset boundary using allocation-free null children, sliced strings with nulls and aligned columns, nested map/struct/list/string payloads, and sparse filtered inputs. Small byte limits are test-only; production uses `i32::MAX`. # Are there any significant user-facing changes? Coalescing may emit smaller batches to preserve representable offsets. Rows and schema are preserved; no public API or configuration is added. The port and its review were assisted by Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
