* Anthony Liguori <[EMAIL PROTECTED]> [2008-09-26 11:03]: > Revision: 5323 > http://svn.sv.gnu.org/viewvc/?view=rev&root=qemu&revision=5323 > Author: aliguori > Date: 2008-09-26 15:59:29 +0000 (Fri, 26 Sep 2008) > > Log Message: > ----------- > Implement an fd pool to get real AIO with posix-aio > > This patch implements a simple fd pool to allow many AIO requests with > posix-aio. The result is significantly improved performance (identical to > that > reported for linux-aio) for both cache=on and cache=off. > > The fundamental problem with posix-aio is that it limits itself to one thread > per-file descriptor. I don't know why this is, but this patch provides a > simple > mechanism to work around this (duplicating the file descriptor). > > This isn't a great solution, but it seems like a reasonable intermediate step > between posix-aio and a custom thread-pool to replace it. > > Ryan Harper will be posting some performance analysis he did comparing > posix-aio > with fd pooling against linux-aio. The size of the posix-aio thread pool and > the fd pool were largely determined by him based on this analysis.
I'll have some more data to post in a bit, but for now, bumping the fd pool up to 64 and ensuring we init aio to support a thread per fd, we mostly match linux aio performance with a simpler implementation. For randomwrites, fd_pool lags a bit, but I've got other data that shows in most scenarios, fd_pool matches linux aio performance and does so with less CPU consumption. Results: 16k randwrite 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms) -----------------------------------+------+------------------+------------------ baremetal (O_DIRECT, aka cache=off)| 61.2 | 13.07 | 19.59 kvm: cache=off posix-aio w/o patch | 4.7 | 3467.44 | 254.08 kvm: cache=off linux-aio | 61.1 | 75.35 | 19.57 kvm: cache=on posix-aio w/o patch |127.0 | 115.78 | 9.19 kvm: cache=on posix-aio w/ patch |126.0 | 67.35 | 9.30 ------------ new results ----------+------+------------------+------------------ kvm:cache=off posix-aio fd_pool[16]| 33.5 | 14.28 | 49.19 kvm:cache=off posix-aio fd_pool[64]| 51.1 | 14.86 | 23.66 16k write 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms) -----------------------------------+------+------------------+------------------ baremetal (O_DIRECT, aka cache=off)|128.1 | 10.90 | 9.45 kvm: cache=off posix-aio w/o patch | 5.1 | 3152.00 | 231.06 kvm: cache=off linux-aio |130.0 | 83.83 | 8.99 kvm: cache=on posix-aio w/o patch |184.0 | 80.46 | 6.35 kvm: cache=on posix-aio w/ patch |165.0 | 70.90 | 7.09 ------------ new results ----------+------+------------------+------------------ kvm:cache=off posix-aio fd_pool[16]| 78.2 | 58.24 | 15.43 kvm:cache=off posix-aio fd_pool[64]|129.0 | 71.62 | 9.11 -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [EMAIL PROTECTED] -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html