nickva opened a new pull request, #6039:
URL: https://github.com/apache/couchdb/pull/6039
Previously in `fabric_doc_update` we used a dict keyed by the whole `#doc{}`
record. Updating a large batch meant spening a decent amount of CPU time
repeatedly hashing the document bodies. To avoid that we use small and
lightweight integers as key and while at it, also switch to use maps instead of
dicts.
A few more details on the two optimizations
1) Use a lightweight integer tag to match and index documents instead of
using their whole `#doc{}` records Using `#doc{}` as a key means applying the
`phash/2` function to it, which, in turn, traverses the whole term to produce
the hash slot number for the dict implementation
https://github.com/erlang/otp/blob/OTP-29.0.2/lib/stdlib/src/dict.erl#L676.
A seemingly obvious choice to use might be a the DocId, however in a batch we
can have duplicate DocIds so that won't work. So we borrow an already used
pattern in couch_db local doc updater which tags docs in a batch with the `tag`
field in the `meta` proplist. This coordinator tag lives only in the
coordinator we strip away before sending the data to workers or returning it.
2) Use a map instead of a dict. This one is a no-brainer. A map scales
better and uses an internal C based structures. Dict is as just phashes of
tuples essentially
I benchmarked lazily with the built-in fabric bench and at last on my laptop
the difference is noticeable for the default 1000 doc batches. I ran the
benchmarks 3 times alternating between main and PR code.
Environment
----
```
fabric_bench:go().
*** Parameters
* batch_size : 1000
* doc_size : medium
* docs : 100000
* individual_docs : 1000
* n : default
* q : default
*** Environment
* Nodes : 1
* Bench ver. : 1
* N : 1
* Q : 2
* OS : unix/darwin
* Couch ver. : 3.5.2-25b1b9a
* Couch git sha: 25b1b9a
* VM details : Erlang/OTP 27 [erts-15.2.7.9] [source] [64-bit] [smp:12:12]
* [ds:12:12:16] [async-threads:1]
```
Main
----
```
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 440
* Get random doc 100000X (Hz): 1600
* All docs (Hz): 49000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 24000
* Single doc updates 1000X (Hz): 280
* Time to run all benchmarks (sec): 300
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 440
* Get random doc 100000X (Hz): 1600
* All docs (Hz): 46000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 16000
* Single doc updates 1000X (Hz): 290
* Time to run all benchmarks (sec): 301
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 460
* Get random doc 100000X (Hz): 1700
* All docs (Hz): 48000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 21000
* Single doc updates 1000X (Hz): 300
* Time to run all benchmarks (sec): 288
```
PR
---
```
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 540
* Get random doc 100000X (Hz): 1600
* All docs (Hz): 49000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 25000
* Single doc updates 1000X (Hz): 290
* Time to run all benchmarks (sec): 257
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 550
* Get random doc 100000X (Hz): 1700
* All docs (Hz): 45000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 22000
* Single doc updates 1000X (Hz): 300
* Time to run all benchmarks (sec): 253
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 560
* Get random doc 100000X (Hz): 1600
* All docs (Hz): 46000
* All docs w/ include_docs (Hz): 4000
* Changes (Hz): 29000
* Single doc updates 1000X (Hz): 300
* Time to run all benchmarks (sec): 249
```
If we approximate it's about 20% or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
