Direct I/O (assuming you mean O_DIRECT on open) can be a bit fiddly, but I doubt it's out of reach. The main difficulty is allocating a buffer with appropriate alignment.
I put together a library to facilitate O_DIRECT I/O a while back: http://stromberg.dnsalias.org/~dstromberg/odirect/ odirect is callable from C, C++ and CPython. I hope to add support for calling it via Python "ctypes" at some point in the future. On Sun, Sep 4, 2011 at 6:09 AM, Sandon Van Ness <san...@van-ness.com> wrote: > As far as I can tell rsync doesn't support this. How hard would it be to > implement this? Is it trivial enough to just change the calls in the code > with sed? I think this can significantly reduce CPU usage and increase I/O > speed when dealing with fast storage solutions. It can make a huge > difference when say transferring 30TB of data. > > Here are some tests I did. So far the only thing I know of that can use > direct I/O is dd which showed a huge improvement. CPU usage went from very > high down to under 15% and the speed almost doubled. > > Is there anything else I can do to tune rsync to give me better speed > and/or less CPU usage as I seem to be hitting a CPU bottleneck in the case > of rsync. > > In my tests I saw the highest CPU usage with rsync and lowest throughput: > > *cp (no direct I/O):* > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 23356 root 20 0 10740 676 584 D 74 0.0 0:16.48 cp > 23344 root 20 0 0 0 0 D 9 0.0 0:04.80 flush-8:64 > 639 root 20 0 0 0 0 R 9 0.0 0:12.28 kswapd0 > > > root@dekabutsu: 05:54 AM :/data2# time cp -avf /data/winxp.img /data2/ > `/data/winxp.img' -> `/data2/winxp.img' > > real 1m10.767s > user 0m0.692s > sys 0m47.401s > > *Rsync (no direct I/O):* > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 23397 root 20 0 11572 616 256 R 97 0.0 0:08.48 rsync > 23395 root 20 0 11864 1292 848 R 84 0.0 0:07.35 rsync > > > root@dekabutsu: 05:56 AM :/data2# time rsync --progress -av > /data/winxp.img /data2/ > building file list ... > 1 file to consider > winxp.img > 37589497344 100% 223.43MB/s 0:02:40 (xfer#1, to-check=0/1) > > sent 37594086010 bytes received 42 bytes 234231065.74 bytes/sec > total size is 37589497344 speedup is 1.00 > > real 2m40.552s > user 3m36.020s > sys 1m8.181s > > *dd (direct I/O)=* > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 23452 root 20 0 14960 1664 580 D 13 0.0 0:00.76 dd > > > root@dekabutsu: 06:00 AM :/data2# time dd bs=1M iflag=direct oflag=direct > if=/data/winxp.img of=/data2/winxp.img > 35848+1 records in > 35848+1 records out > 37589497344 bytes (38 GB) copied, 46.8669 s, 802 MB/s > > real 0m46.869s > user 0m0.031s > sys 0m6.043s > > > *dd (no direct I/O):* > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 23455 root 20 0 14960 1664 576 R 65 0.0 0:06.59 dd > 23344 root 20 0 0 0 0 S 20 0.0 0:17.88 flush-8:64 > > > root@dekabutsu: 06:01 AM :/data2# time dd bs=1M if=/data/winxp.img > of=/data2/winxp.img > 35848+1 records in > 35848+1 records out > 37589497344 bytes (38 GB) copied, 83.5282 s, 450 MB/s > > real 1m23.530s > user 0m0.048s > sys 0m54.304s > > (so pretty close to the speed of cp). > > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/** > mailman/listinfo/rsync <https://lists.samba.org/mailman/listinfo/rsync> > Before posting, read: > http://www.catb.org/~esr/faqs/**smart-questions.html<http://www.catb.org/%7Eesr/faqs/smart-questions.html> >
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html