Re: [PATCH 51/52] extsearchidx: support --batch-size checkpoints

2020-11-03 Thread Eric Wong
Eric Wong wrote: > This is needed to limit the RSS of processes and ensure the > stored data in over.sqlite3 and Xapian DBs are consistent if > interrupted. Without checkpoints, indexing lore causes shard > workers to take several GB of memory and thrash/OOM smaller > systems. Ugh, the ~30 hours

[PATCH 51/52] extsearchidx: support --batch-size checkpoints

2020-10-27 Thread Eric Wong
This is needed to limit the RSS of processes and ensure the stored data in over.sqlite3 and Xapian DBs are consistent if interrupted. Without checkpoints, indexing lore causes shard workers to take several GB of memory and thrash/OOM smaller systems. --- lib/PublicInbox/ExtSearchIdx.pm | 20 +