andygrove opened a new issue, #7688: URL: https://github.com/apache/datafusion-comet/issues/7688
### What is the problem the feature request solves? The native Parquet scan registers no memory reservation, so its working set is invisible to the task memory pool and to Spark behind it. #6125 prototyped charging it and was closed after review. This issue records what the prototype showed and what review found, so a later attempt does not start from scratch. **The gap.** With tracing on (`jemalloc,alloc-accounting`), a TPC-H SF100 run (2 executors x 8 cores, 16g off-heap) showed scan-only queries such as q1, q6 and q19 allocating 100 to 150 MiB per executor natively while reserving almost nothing from the pool. The gap is a fixed working set per running scan, not something that grows over time. Per partition it is the compressed column chunks of the row group being decoded (arrow-rs fetches the projected column chunks of one row group before decoding it, and DataFusion reads one file per partition at a time), plus the decompressed pages and output batch the decoder builds. For q1 on SF100 `lineitem` the seven projected columns are 9.1 MiB compressed per row group, which matches the roughly 14 MiB per task that was unaccounted. The size is set by the file layout (row-group size times projected columns), so wide tables with large row groups can hold far more per task. ### Describe the potential solution The prototype is on the branch `parquet-scan-memory-accounting` of `andygrove/datafusion-comet` (head `3cb14c0f95`). All of it is in `EagerPageIndexReaderFactory`, which already owns the scan's `AsyncFileReader`: - **Data pages, charged exactly.** Each buffer returned by `get_byte_ranges` is wrapped in a `Bytes::from_owner` whose owner holds its share of a `MemoryReservation`. The reservation is released when arrow-rs drops the last reference to the buffer or to any page sliced from it, so the decoder needs no change. - **Decode buffers, charged as an estimate.** arrow-rs allocates decompressed pages and the output batch with no hook to observe them, so each open reader held one default page (1 MiB) per projected leaf column plus one `batch_size` output batch. - **Refusal does not fail the scan.** The prototype used `try_grow` and kept reading unaccounted when refused, counting the bytes in a new `scan_io_unreserved_bytes` metric. **Measured results** (TPC-H SF100, same commit with and without the change, native allocated bytes minus the pool's reserved total at the same instant, per executor): | | Median gap | p95 gap | |---|---|---| | Before | 21.9 MiB | 140.2 MiB | | After | 7.9 MiB | 94.0 MiB | For scan-heavy queries the median gap dropped from 71 to 127 MiB to 6 to 19 MiB. Total run time was unchanged (208.7 s vs 208.8 s) and all 22 result hashes matched. At reduced off-heap size (`fair_unified`), no reservation was refused even at 2g, so the refusal path was not exercised on this workload. Spark's disk spill rose at 2g, mostly in q9 (8.0 to 15.0 GiB) and q10 (5.6 to 6.9 GiB). That is consistent with operators seeing less free memory, but it was a single run, so it needs repeating. ### What review found These need to be resolved by any follow-up. Thanks to @comphead, @viirya, @sunchao, @parthchandra, @rich7420 and @dwsmith1983 for the detailed reviews. **Error handling** - `try_grow` can fail for reasons other than refusal. `CometFairMemoryPool::try_grow` returns `ResourcesExhausted` only for its own local-limit refusal and for a partial grant. Anything else (a `SparkOutOfMemoryError` from another consumer's spill, an `InterruptedException` from a killed task, a JNI failure) arrives as `DataFusionError::External`. The prototype used `is_err()`, which swallowed those, so a killed task kept reading. Only `ResourcesExhausted` should mean "keep reading unaccounted". Everything else must propagate, with a test using a pool whose `try_grow` returns `External`. - `grow` instead of `try_grow` is worth reconsidering. Since #6128 the unified pools no longer panic on `grow`. They record the bytes and carry what Spark declined as overcommit. The fetched pages already exist when charged, so this is not admission control. With `grow` the pressure lands on consumers that can spill, and both the error-classification question and `scan_io_unreserved_bytes` go away. The cost is that `grow` ignores the fair limit. - If `try_grow` stays, a scan that cannot act on a refusal keeps asking Spark for the whole overcommit debt on every call: a JNI round trip per row group, and possibly a JVM spill that buys nothing (Spark grants part, the unified pool hands the whole grant back, the scan reads anyway). Under `greedy_unified` each refusal also logs a warning and dumps all consumers' memory usage. The reader should back off after a refusal. **Consumer registration** - One `MemoryConsumer` per reader shrinks the fair pool for the whole task. The fair pool's limit is `pool_size / num`, so each registration lowers every other consumer's ceiling regardless of how many bytes the scan holds (roughly 25% in an aggregate + sort + shuffle-writer task). DataFusion also creates a second reader per file for bloom-filter pruning and keeps both alive across `load_bloom_filters`, so the count churns and the decode estimate is charged twice. Register once on the factory, one consumer per scan operator, and hand each reader a `new_empty()` handle. This is the same pattern #6247 used for the writers. This may also explain part of the q9/q10 spill increase. **Decode estimate** - `leaves * 1 MiB + batch_size * row_width` is fixed at plan time and charged before any metadata is read. A 500-column projection charged about 508 MiB per open reader (532,480,000 bytes in one reproduction) even for a one-row or 1 KiB file, and a scan over many small files pays it on every open. Capping by file size does not bound it either, since decompressed pages can exceed the file. - Size it from the file instead. The factory already loads the footer and offset index, so per projected leaf it could use the smaller of one page and the largest selected column chunk's uncompressed size, plus `min(batch_size, num_rows) * row_width`. Charge it at the first `get_byte_ranges` rather than in `create_reader`, which keeps it out of the planning step and means a fully pruned file never pays it. - `row_width` summed top-level fields (a struct counted as a flat 32 bytes) while the page term counted leaves, so the batch term under-counts wide nested rows. The test covered only `Struct`, not `List<Struct>` or `Map`. **Blocking and threading** - `create_reader` runs in `ParquetMorselizer::prepare_open_file`, which DataFusion documents as the CPU-only planning step. A JNI `acquireMemory` there can run other consumers' spills and park in `lock.wait()`. - Plans with no JVM sources are spawned onto the shared Tokio runtime, so grows in `reserve_buffers` can park a runtime worker too. This is most likely at the start of a task, which is exactly when a scan makes its first request. A task-kill interrupt is not delivered to a worker thread. #6261 since made a blocked acquire hand the worker's other tasks to another thread, but that thread still waits in Spark. **Release accounting** - Charging per buffer gives N shrinks (one JNI `release_memory` per column chunk, each taking the pool mutex). With the page index and a row selection, parquet fetches one range per selected page, so it can be one shrink per page. One `Arc<MemoryReservation>` shared by every buffer of a fetch gives one grow and one shrink, released when the last buffer drops. - Coalesced object-store fetches are undercharged. `get_ranges` coalesces nearby ranges, so each returned `Bytes` is a slice of one merged allocation and any surviving slice pins the gap bytes too. "Charged exactly" holds only for the local path. `scan_io_object_store_response_bytes_read` already measures the difference. - The design depends on an invariant that must be documented in the code: no page buffer may end up aliased by an Arrow array that crosses FFI. If it did, the final `shrink` would run from the FFI release callback when the JVM closes the batch. `CometFairMemoryPool::shrink` and `CometUnifiedMemoryPool::shrink` panic when the JNI release fails, and a panic in an `extern "C"` callback aborts the executor. Comet requests `Utf8`, not `Utf8View`, so this does not fire today. `ByteViewArrayDecoderPlain` can alias the page bytes, so a `Utf8View` hint in the file's `ARROW:schema` is safe only because the cast to the Spark type copies. **Scope** - `get_bytes` (bloom filters) and `get_metadata` (footer plus page index) stay unreserved, and `load_bloom_filters` materializes `BloomFilterStatistics` for every row group at once, which is not small on a wide table. - The change turned accounting on for every native Parquet scan with no switch. A config flag (default `true`) would give users a way out while the estimate is refined. - `scan_io_unreserved_bytes` was cumulative refused bytes, including the one-shot decode estimate, but rendered as a size in the SQL UI so it read as a live footprint (or a leak). Any metric of this kind should be named and documented as cumulative, or split. - `docs/source/contributor-guide/memory_management.md` lists "decompression buffers, Parquet metadata structures, `object_store` request buffers" as undeclared. It, and its cgroup diagram, must be updated by whichever change closes the gap. - Spark SQL validation was not done. A change to scan memory behaviour needs the `run-*` Spark SQL label or `dev/local-ci.sh spark sql_core-1`, because the pull request tier runs only the default profile. ### Suggested plan 1. Register one `MemoryConsumer` per scan operator on the factory, with a `new_empty()` handle per reader. 2. Share one `Arc<MemoryReservation>` per fetch for the data pages. 3. Decide `grow` versus `try_grow` for the data pages. If `try_grow`, propagate non-`ResourcesExhausted` errors and back off after a refusal. 4. Size the decode estimate from the file's metadata and charge it at the first fetch. 5. Charge bloom-filter and metadata buffers, or document them as uncharged. 6. Update `memory_management.md`, add a switch, and run Spark SQL validation. ### Additional context Closed attempt: #6125. Related: #6126 (the same gap in the native Iceberg scan, which cannot reuse the `AsyncFileReader` approach), #6127 and #6128 (pool `grow` behaviour). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
