* Vivek Goyal (vgo...@redhat.com) wrote: > On Fri, Sep 18, 2020 at 05:34:36PM -0400, Vivek Goyal wrote: > > Hi All, > > > > virtiofsd default thread pool size is 64. To me it feels that in most of > > the cases thread pool size 1 performs better than thread pool size 64. > > > > I ran virtiofs-tests. > > > > https://github.com/rhvgoyal/virtiofs-tests > > I spent more time debugging this. First thing I noticed is that we > are using "exclusive" glib thread pool. > > https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new > > This seems to run pre-determined number of threads dedicated to that > thread pool. Little instrumentation of code revealed that every new > request gets assiged to new thread (despite the fact that previous > thread finished its job). So internally there might be some kind of > round robin policy to choose next thread for running the job. > > I decided to switch to "shared" pool instead where it seemed to spin > up new threads only if there is enough work. Also threads can be shared > between pools. > > And looks like testing results are way better with "shared" pools. So > may be we should switch to shared pool by default. (Till somebody shows > in what cases exclusive pools are better). > > Second thought which came to mind was what's the impact of NUMA. What > if qemu and virtiofsd process/threads are running on separate NUMA > node. That should increase memory access latency and increased overhead. > So I used "numactl --cpubind=0" to bind both qemu and virtiofsd to node > 0. My machine seems to have two numa nodes. (Each node is having 32 > logical processors). Keeping both qemu and virtiofsd on same node > improves throughput further. > > So here are the results. > > vtfs-none-epool --> cache=none, exclusive thread pool. > vtfs-none-spool --> cache=none, shared thread pool. > vtfs-none-spool-numa --> cache=none, shared thread pool, same numa node
Do you have the numbers for: epool epool thread-pool-size=1 spool ? Dave > > NAME WORKLOAD Bandwidth IOPS > > vtfs-none-epool seqread-psync 36(MiB/s) 9392 > > vtfs-none-spool seqread-psync 68(MiB/s) 17k > > vtfs-none-spool-numa seqread-psync 73(MiB/s) 18k > > > vtfs-none-epool seqread-psync-multi 210(MiB/s) 52k > > vtfs-none-spool seqread-psync-multi 260(MiB/s) 65k > > vtfs-none-spool-numa seqread-psync-multi 309(MiB/s) 77k > > > vtfs-none-epool seqread-libaio 286(MiB/s) 71k > > vtfs-none-spool seqread-libaio 328(MiB/s) 82k > > vtfs-none-spool-numa seqread-libaio 332(MiB/s) 83k > > > vtfs-none-epool seqread-libaio-multi 201(MiB/s) 50k > > vtfs-none-spool seqread-libaio-multi 254(MiB/s) 63k > > vtfs-none-spool-numa seqread-libaio-multi 276(MiB/s) 69k > > > vtfs-none-epool randread-psync 40(MiB/s) 10k > > vtfs-none-spool randread-psync 64(MiB/s) 16k > > vtfs-none-spool-numa randread-psync 72(MiB/s) 18k > > > vtfs-none-epool randread-psync-multi 211(MiB/s) 52k > > vtfs-none-spool randread-psync-multi 252(MiB/s) 63k > > vtfs-none-spool-numa randread-psync-multi 297(MiB/s) 74k > > > vtfs-none-epool randread-libaio 313(MiB/s) 78k > > vtfs-none-spool randread-libaio 320(MiB/s) 80k > > vtfs-none-spool-numa randread-libaio 330(MiB/s) 82k > > > vtfs-none-epool randread-libaio-multi 257(MiB/s) 64k > > vtfs-none-spool randread-libaio-multi 274(MiB/s) 68k > > vtfs-none-spool-numa randread-libaio-multi 319(MiB/s) 79k > > > vtfs-none-epool seqwrite-psync 34(MiB/s) 8926 > > vtfs-none-spool seqwrite-psync 55(MiB/s) 13k > > vtfs-none-spool-numa seqwrite-psync 66(MiB/s) 16k > > > vtfs-none-epool seqwrite-psync-multi 196(MiB/s) 49k > > vtfs-none-spool seqwrite-psync-multi 225(MiB/s) 56k > > vtfs-none-spool-numa seqwrite-psync-multi 270(MiB/s) 67k > > > vtfs-none-epool seqwrite-libaio 257(MiB/s) 64k > > vtfs-none-spool seqwrite-libaio 304(MiB/s) 76k > > vtfs-none-spool-numa seqwrite-libaio 267(MiB/s) 66k > > > vtfs-none-epool seqwrite-libaio-multi 312(MiB/s) 78k > > vtfs-none-spool seqwrite-libaio-multi 366(MiB/s) 91k > > vtfs-none-spool-numa seqwrite-libaio-multi 381(MiB/s) 95k > > > vtfs-none-epool randwrite-psync 38(MiB/s) 9745 > > vtfs-none-spool randwrite-psync 55(MiB/s) 13k > > vtfs-none-spool-numa randwrite-psync 67(MiB/s) 16k > > > vtfs-none-epool randwrite-psync-multi 186(MiB/s) 46k > > vtfs-none-spool randwrite-psync-multi 240(MiB/s) 60k > > vtfs-none-spool-numa randwrite-psync-multi 271(MiB/s) 67k > > > vtfs-none-epool randwrite-libaio 224(MiB/s) 56k > > vtfs-none-spool randwrite-libaio 296(MiB/s) 74k > > vtfs-none-spool-numa randwrite-libaio 290(MiB/s) 72k > > > vtfs-none-epool randwrite-libaio-multi 300(MiB/s) 75k > > vtfs-none-spool randwrite-libaio-multi 350(MiB/s) 87k > > vtfs-none-spool-numa randwrite-libaio-multi 383(MiB/s) 95k > > > Thanks > Vivek -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
