Hi, you need the commits from master listed below (which will be in
8.0), which disable sync writes whild doing bulk HDB loads. It's the
per-record fsync()s that kill performance.
If you copied the HDB with scp and loaded it with kadmin -l, you'd find
it's still slow -- the network has nothing to do with it :/
Our HDB is larger than yours, so we needed this sooner...
commit 7d5f8bb051ca84592d1196bf5d5522da5a50f9d6
Author: Nicolas Williams <n...@twosigma.com>
Date: Tue Oct 10 12:18:57 2017 -0500
Disable sync during kadmin load
commit 305dc816525f461f9bfe640d87f671f53f0e0fc6
Author: Nicolas Williams <n...@twosigma.com>
Date: Tue Oct 10 12:11:26 2017 -0500
Disable sync during iprop receive_everything()
Doing an fsync per-record when receiving the complete HDB is a performance
disaster. Among other things, if the HDB is very large, then one slave
receving a full HDB can cause other slaves to timeout and, if HDB write
activity is high enough to cause iprop log truncation, then also need full
syncs, which leads to a cycle of full syncs for all slaves until HDB write
activity drops.
Allowing the iprop log to be larger helps, but improving
receive_everything()
performance helps even more.
commit 5bcbe2125b18160f6ad348b15f8036ffedc15770
Author: Nicolas Williams <n...@twosigma.com>
Date: Tue Oct 10 13:06:21 2017 -0500
Add hdb_set_sync() method
