Eric Wong <e...@80x24.org> wrote:
> This is needed to limit the RSS of processes and ensure the
> stored data in over.sqlite3 and Xapian DBs are consistent if
> interrupted. Without checkpoints, indexing lore causes shard
> workers to take several GB of memory and thrash/OOM smaller
> systems.
Ugh, the ~30 hours in the cover letter was without this patch.
Using even an 100m batch size(*) makes lore/* take ~70 hours :<
(*) 100m works fine for lore/lkml and even >10m is diminishing
returns...
--
unsubscribe: one-click, see List-Unsubscribe header
archive: https://public-inbox.org/meta/
