In response to "Anders Boström" <[EMAIL PROTECTED]>: > >>>>> "BM" == Bill Moran <[EMAIL PROTECTED]> writes: > > BM> In response to "Anders Boström" <[EMAIL PROTECTED]>: > >> I did some new performance-tests: > >> > >> All operations are against a directory-tree with 7,255,659,224 bytes > >> data in 98,025 files. > >> > >> | test1 | test2 | test3 | > >> --------------------------------+-------+-------+-------+ > >> bacula-fd, no compression, md5: | 10:25 | 10:42 | 10:15 | > >> bacula-fd, GZIP, md5: | 16:09 | 15:46 | 17:02 | > >> tar, local (1): | 8:37 | 8:53 | 8:54 | > >> tar + nc (2): | 9:48 | 9:52 | 9:43 | > >> tar + gzip + nc (3): | 14:11 | 14:26 | 15:03 | > >> --------------------------------+-------+-------+-------+ > > BM> OK. This indicates to me that bacula is doing a damn good job. Only > BM> 15% overhead to add checksumming and cataloging features to backup. > BM> If you ask me, that's a hell of a deal. > > >> (1) time /bin/sh -c "tar cf - directory | cat >/dev/null" > >> (2) time /bin/sh -c "tar cf - directory | nc -q 0 backup_server 4711" > >> (3) time /bin/sh -c "tar czf - directory | nc -q 0 backup_server 4711" > >> > >> This round of tests is more in line with what I expected, and the > >> bacula performance is quite good. The only major difference compared > >> to my previous tests is that the file-server disc-performance is much > >> better. It seems like bacula suffers much more than tar from slow > >> disc-performance on the file-server. backup-server and network > >> performance don't seem to be an issue at all in the tests, even if > >> write to TCP is a bit slower than /dev/null. > >> > >> However, both tar and bacula suffers from quite large slow-down when > >> gzip is used. This is on an Athlon 64 X2 3800+ (2-core), running > >> >50% idle during backup, leading me to believe that there are room for > >> improvement. But part of the problem might be in the linux-kernel > >> (2.6.17.8). At least when tar was running, the gzip process seemed to > >> move from one CPU-core to the other very frequently. > > BM> Improvement, maybe, but not for Bacula, as far as I can see. If a > BM> dual-core system is running at 50%, then 1 core is maxed out. Since > > No, this was not the case. Most of the time, both CPU's was idle >30% > of the time, according to top.
It's hard to help when your facts keep changing. Further up, you mention that the CPU is >50% idle, now you say that each independent core is >30% idle. This is not meant as an attack, but you will get absolutely nowhere in performance optimization if you can't get the details right. Details are terribly important. > BM> the gzip process is serialized, it can only run on one core at a time, > BM> which means the CPU is the limiting factor at this time. > > gzip on this computer, on one CPU, reach about 18 Mbyte/s. bacula with > gzip only reach ~7.7 Mbyte/s. This leads me to believe that there are > room for improvement. Again, the story changes. Above, you indicate that tar+gzip ran about 15% faster than bacula with gzip, which seems reasonable. Now you're saying that gzip is ~twice as fast as Bacula + gzip. Where did this new number come from? Are you taking in to account networking on this new test? One thing to consider is that network has an impact, even when the physical speed of the network is faster than the gzip process. There is extra processing involved in packing up the packets, and delays (even if minimal) introduced by networking add latency to the process that isn't always intuitive. > BM> What was your performance goal anyway? If you actually thought you'd > BM> get backup throughput at wire speed on 1g network, that was your first > BM> mistake. I don't know of any disks that can feed data that fast. Hell, > BM> from your experiment above, those disks can feed data at about 13M/sec, > BM> which is closer to 100mb than gig, and that's the absolute fastest > BM> you're going to get. > > No, I don't expect better performance than the disc-performance. In > fact, a lot lower than disc-performance is acceptable. The 7.7 Mbyte/s > we reach now is OK. However, full backup times are long, but as long > as we can run a full backup in less than 48 hours, it's OK. Again, I'm glad it's working for you. However, as someone who would otherwise be interested in researching this, I don't have any motivation to do so. "full backup times are long" sounds arbitrary enough to disinterest me. > >> I can share one experience with you regarding > >> disc-performance: > >> > >> Both our two Seagate ST3500641AS discs (500Gb Barracuda 7200.9, SATA) > >> never completes the SMART extended self-test. It worked fine for ~6 > >> months, and now both run forever. The drives report 30% remaining > >> of the self-test, then ~2 hours later 10% remaining. After that it > >> goes up to 40% remaining and the cycle repeats. Also when running the > >> SMART extended self-test, the IO-performance is more than 10 times > >> lower than normal (leading to very long backup-time). > >> > >> We have been in contact with Seagate about this, and have upgraded to > >> the latest firmware, without any success. So I guess we will have to > >> RMA the discs. > > BM> Are these the disks you're using to test Bacula? Please tell me you're > BM> not using hardware known to be broken as a test bed. > > When I run the original tests reported on this list (yes on these > discs), I didn't know about this problem. And unfortunately, this > isn't a test-bed, it is our file-server. Yuk. Here I feel your pain. Keep in mind that there _could_ be disk problems skewing your tests. -- Bill Moran Collaborative Fusion Inc. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
