DanielLeens commented on issue #12266: URL: https://github.com/apache/seatunnel/issues/12266#issuecomment-5635778932
Thanks for proposing the adaptive verification direction. On the current file-backed path, neither a benchmark-level chunk loop nor a full-batch mode behind a parameter satisfies the stated memory guarantee. `FileMapStore.loadAll(Collection)` first calls `mapStorage.loadAll()` and only then filters the requested keys. `IMapStorage` exposes only that whole-map operation. Therefore, splitting keys into chunks at the benchmark layer still replays the complete WAL into heap for every chunk, while a parameterized full reload leaves the OOM outcome configuration-dependent rather than proving the acceptance criteria. Please keep this follow-up in design until there is a narrow storage-level way to verify selected keys without materializing the complete WAL. The first proposal must show its peak-memory bound with `initialStoredJobCount=1000`, preserve write-through and cache semantics, and include deterministic storage coverage. Only then should the benchmark harness add full-batch durability assertions outside the measured `SingleShot` path. No full `loadAll` switch or heuristic heap estimate should be merged as a substitute for that contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
