CryoThrust commented on issue #12266: URL: https://github.com/apache/seatunnel/issues/12266#issuecomment-5633651326
I'd be interested in helping with this follow-up. Before implementing, I'd propose making the durability verification adaptive rather than unconditionally reloading the full WAL: * Keep the current representative-key check as the low-memory baseline. * Add a full-batch `loadAll` verification mode only when the estimated batch footprint fits a configured/observed heap budget; otherwise verify the batch through bounded chunks or per-key round trips. * Run verification in tear-down/iteration hooks outside the measured `SingleShot` path and record the verification mode in benchmark output. * Assert that every key in the growth batch is present and durable, while retaining an explicit OOM/regression guard for `initialStoredJobCount=1000`. Would maintainers prefer the first PR to implement bounded chunked verification (deterministic memory) or a full-batch mode behind a benchmark parameter? I can prepare a focused PR once that direction is confirmed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
