nickva opened a new pull request, #5987: URL: https://github.com/apache/couchdb/pull/5987
Two small optimizations and a cleanup (Commits in order) 1. **Do not run full GC after each doc update in the indexer** Run it after 1000 docs, ensuring it still forced to run as a safe default. However, in OTP 27+ this should not be needed at all. So infinity is also a possible config setting, we may change to that in the future. Comments from John Hogberg in OTP repo regarding behavior change in OTP 27: https://github.com/erlang/otp/issues/8229#issuecomment-1988858134 > the GC pressure of off-heap binaries ("vheap") was vastly under-counted prior to 24ef4cb [1]. In these tests this caused it to GC less often and, crucially, when there was less live data to keep. [1] https://github.com/erlang/otp/commit/24ef4cbaeda9b9c26682cba75f2f15b0c58722aa 2. **Increase view queue size from 100KB to 10MB** 100KB was undersized given the 100 batch size used in the process_doc call. 100 docs as small as 1KB will already start to get serialized and not take advantage of the work queue buffering. Make it 10MB to fit about 10 batches of 1MB each and be a bit under the min_writer_size default of 16MB. 3. **Use config:get_integer/3 in couch_mrview_updater** This avoids re-parsing the string values into integers when defaults are used every time we run the check. It's more of a code cleanup but can regarded as a micro-optimization as well **Benchmark** With this bench script https://gist.github.com/nickva/0c00947fc7405d260e1d5226f7e9c104 got about a 10-12% speed improvement locally on my laptop 11k -> 12k docs indexed per second With main ``` % ./view_bench.py URL: http://localhost:15984 db: db ndocs: 500000 doc_size: ~1024 B batch: 500 Loading docs... Load done in 43.0s View URL: db/_design/ddoc/_view/view Timing build... Build time: 46638 ms (46.6 s) for 500000 docs (10,721 docs/s) % ./view_bench.py URL: http://localhost:15984 db: db ndocs: 500000 doc_size: ~1024 B batch: 500 Loading docs... Load done in 42.5s View URL: db/_design/ddoc/_view/view Timing build... Build time: 45955 ms (46.0 s) for 500000 docs (10,880 docs/s) % ./view_bench.py URL: http://localhost:15984 db: db ndocs: 500000 doc_size: ~1024 B batch: 500 Loading docs... Load done in 43.7s View URL: db/_design/ddoc/_view/view Timing build... Build time: 48753 ms (48.8 s) for 500000 docs (10,256 docs/s) ``` With the PR ``` % ./view_bench.py URL: http://localhost:15984 db: db ndocs: 500000 doc_size: ~1024 B batch: 500 Loading docs... Load done in 43.4s View URL: db/_design/ddoc/_view/view Timing build... Build time: 40627 ms (40.6 s) for 500000 docs (12,307 docs/s) % ./view_bench.py URL: http://localhost:15984 db: db ndocs: 500000 doc_size: ~1024 B batch: 500 Loading docs... Load done in 43.3s View URL: db/_design/ddoc/_view/view Timing build... Build time: 39337 ms (39.3 s) for 500000 docs (12,711 docs/s) % ./view_bench.py URL: http://localhost:15984 db: db ndocs: 500000 doc_size: ~1024 B batch: 500 Loading docs... Load done in 43.6s View URL: db/_design/ddoc/_view/view Timing build... Build time: 41181 ms (41.2 s) for 500000 docs (12,142 docs/s) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
