andygrove opened a new issue, #2531: URL: https://github.com/apache/datafusion-ballista/issues/2531
## Motivation Ballista's executor allocation observability in #2530 logs outstanding Rust allocator bytes alongside current memory pool reservations and the resolved executor-wide pool budget. This makes the gap visible, but does not attribute growing drift to particular memory consumers or record peak drift. DataFusion added reusable `MemoryDriftTracker` and `PeakRecordingPool::with_drift_tracker` APIs in apache/datafusion#25651. These are not available in Ballista's currently locked DataFusion 55.1.0 dependency. The tracker accepts a caller-supplied allocated-byte counter, so Ballista should be able to provide its existing `alloc_accounting::current_balance` rather than introduce another allocator. ## Proposed follow-up Once Ballista upgrades to a DataFusion release containing these APIs: - Evaluate integrating `MemoryDriftTracker` with executor task memory pools, supplying the existing allocation counter. - Use a single tracker per executor to compare process-wide allocator bytes with reservations across its pools; do not compare the process-wide allocation count against individual task pools. - Use the upstream `PeakRecordingPool` hooks for consumer-attributed drift and peak reporting where appropriate, rather than copying DataFusion's tracking implementation. - Preserve the 10-second allocation/reservation/pool-budget log and its bounded, auto-sized, and unbounded behavior. - Account for shared/custom pools and session runtime caching without double-counting reservations or retaining completed pools. Check whether wrapping pools affects downcasting or other existing behavior. - Keep this observational: do not change allocation behavior, memory limits, or the global allocator of applications embedding Ballista. - Evaluate whether the existing weak pool registry can be simplified using the upstream tracker, retaining any functionality needed for budget and live-pool reporting. ## Acceptance criteria - The dependency version containing the required APIs is identified and available to Ballista. - Executor drift/peak reporting uses DataFusion's reusable tracker and Ballista's existing allocation counter. - Tests cover concurrent task pools, shared/custom pools, session runtime caching, cleanup, and shutdown. - Documentation explains the distinction between allocator bytes, pool reservations, pool budget, and drift. ## Related work - Ballista executor allocation and pool logging: #2530 - Upstream reusable drift tracking: apache/datafusion#25651 This issue tracks a future follow-up; it does not require upgrading DataFusion as part of #2530. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
