> Note that posix_fadvise() only affects caching and read-ahead at the OS > level. While the use of posix_fadvise() may indeed improve i/o > performance for particular use cases, it is not parallelism and does not > cause multiple user-space threads to be executed in parallel. I believe > that Kern is referring to a multi-threaded approach in the bacula-fd, > where multiple threads are executing in parallel to read and process > files. > > Also, I believe that bacula-fd already does make use of posix_fadvise()
Yes, but - it does not deliver the correct pattern. It does it on open - the time spend in bacula between bopen and read it sub millisecond leaving the kernel zero time for actually async fetching data for you. > I would think that a reader-writer approach would be possible. A single > writer thread would perform all i/o with the SD while multiple reader > threads would read and process single files at a time. A single > management thread would manage the list of files to be backed up and > spawn reader threads to process them. This could improve FD performance, > particularly when compression and/or encryption is being used. yes, that would also help. > I am not sure this approach is always a good thing. It depends on the > client hardware. When backing up weak clients using compression or > encryption, it would bring them to their knees, although a mechanism to > limit the number of reader threads that may be spawned would fix that. > Also, with weak clients, the real problem is slow disks on the clients, > and no amount of parallelism will fix that. Remember: This is about optimizing small files (<4MB) - larger - readahead works nicely. Well, today bacula works as a single-thread - Queue Depth 1 client. Google disk benchmarks and see if throughput goes up when QD goes up. The current pattern in bacula does not leave any outstanding IO requests to the underlying hardware as the next is only issued when the first has completed. Despite 0 more concurrency in bacula-fd, this approach stacks a lot of out-standing IO requests to the kernel and lets the kernel exercise parallism which bacula reads the first files off disk. Attached is a - very rough - patch to bacula-fd that compiles and probably breaks all conventions in the codebase on its way. Note: It may eat you tapes and data as it is severely untested at the moment. -- Jesper
bacula-fadvise.patch
Description: Binary data
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
