On Saturday 10 April 2010 04:21:40 Craig Ringer wrote: > Phil Stracchino wrote: > > On 04/09/10 02:35, Henrik Johansen wrote: > >> I have spend some time profiling the bacula-sd under production load on > >> Solaris. Besides checking for signs of lock contention and memory / CPU > >> usage issues I also tracked the I/O statistics for each file descriptor > >> opened within the bacula-sd process while running. > >> > >> According to my notes there was a noticeable change in throughput when > >> testing 100 concurrent jobs against 100 separate devices (100 separate > >> volumes) compared against writing the same jobs against 20 concurrent > >> volumes. > >> > >> I/O throughput increased with more concurrency. > > > > That is an extremely interesting, and somewhat counter-intuitive, result. > > It's exactly what I expect to see on any decent hardware. Experience on > the PostgreSQL mailing list suggests that for database workloads at > least, optimal write throughput tends to be achieved at > nthreads=nspindles+ncpus. > > Of course database workloads tend to be a bit more random in their write > patterns and tend to hit the CPU more than backula-sd's write loads are > likely to. For bacula-sd I'd start at nthreads=nspindles and explore > around there to find where my hardware performed best. I've certainly > found that pure sequential write throughput also tends to increase with > number of writing threads/processes - up to a point that varies with > raid level, disk array layout, spindle count, controller/swraid type, etc. > > Some benefit may even be gained from multiple writing threads on > *single* *disk* setups if TCQ/NCQ is enabled and none of the programs > are using fsync() to force write ordering. Probably not for pure > sequential writes, though I'd have to test that. > > ( I almost wonder if it's worth adapting that tool to produce a quick > and dirty optimizer that searches for near-optimal writing thread count > on a given storage device. Hmm. Might be handy if bacula-sd ever gets a > "max concurrent writers" param or the like. )
That could be an interesting and useful tool. Bacula *does* have two directives to control concurrency in the SD. One is Maximum Concurrent Jobs in the Storage resource, which is an overall limitation on how many simultaneous jobs can be running. The second is the Maximum Concurrent Jobs (same name) in the Device resource, which limits how many simultaneous jobs can be using that particular Device. By the way, for those who do not know, the Storage daemon does use posix_fadvise() when writing the spooling file. However, at this time, the Storage daemon is not aware of how much data is coming from the FD at the beginning, so I have not used posix_fadvise() for writing Volumes. posix_fadvise() an fdatasync() are used when reading files during backup. Kern > > I need to put together a decent benchmark platform before I can do much > hard testing for bacula or with that test tool, as the main decent > arrays I have access to are always busy with backups, production loads, > etc. > > -- > Craig Ringer > > > --------------------------------------------------------------------------- >--- Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Bacula-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/bacula-devel ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Bacula-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/bacula-devel
