Re: [Bacula-users] Strange backups
On 8/23/07, Angel Mieres [EMAIL PROTECTED] wrote: Hi all, Im testing bacula with two jobs. One of them, backup over 70.000 files and have 2 Gb. Second one have over 100 files and have 1Gb. Why the first job is getting speed of 3.000 KB/sec and the second one 25.000 KB/sec?(the backup is to a file on both cases) Have bacula less performance with small files? Although my guess would be also oabout some database related hits, did you try to accomplish (just for testing purposes :) the same file copy via scp or ftp, so to check whether disk I/O or network bottlenecks can be responsible for the issue? -- Maxx - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
Diference between SCP and bacula goes from 15 min with bacula and 3 min with scp. I suspect it's some database problem. Im using postgresql 7.4.8. Maybe with Sqlite will increase the rates? Must I try with them? MaxxAtWork escribió: On 8/23/07, Angel Mieres [EMAIL PROTECTED] wrote: Hi all, Im testing bacula with two jobs. One of them, backup over 70.000 files and have 2 Gb. Second one have over 100 files and have 1Gb. Why the first job is getting speed of 3.000 KB/sec and the second one 25.000 KB/sec?(the backup is to a file on both cases) Have bacula less performance with small files? Although my guess would be also oabout some database related hits, did you try to accomplish (just for testing purposes :) the same file copy via scp or ftp, so to check whether disk I/O or network bottlenecks can be responsible for the issue? - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
On 8/23/07, Angel Mieres [EMAIL PROTECTED] wrote: Hi all, Im testing bacula with two jobs. One of them, backup over 70.000 files and have 2 Gb. Second one have over 100 files and have 1Gb. Why the first job is getting speed of 3.000 KB/sec and the second one 25.000 KB/sec?(the backup is to a file on both cases) Have bacula less performance with small files? Is everything on one single server and on the same harddrive / raid array? The first thought is that the biggest reason for the difference in backup rates is that in the case of the small files your harddrive/array is doing significantly more seeking than in the case with a few large files and depending on how they are on your media file readahead cache may be working against you. Have you timed the same backups using tar? It should be shorter than with bacula but does it have a similar ratio? John - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
I suspect it's some database problem. Im using postgresql 7.4.8. Maybe with Sqlite will increase the rates? Must I try with them? i would doubt sqlite could increase performance except in some very weird corner cases... That would in a lot of cases actually decrease performance. John - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
John Drescher escribió: On 8/23/07, Angel Mieres [EMAIL PROTECTED] wrote: Hi all, Im testing bacula with two jobs. One of them, backup over 70.000 files and have 2 Gb. Second one have over 100 files and have 1Gb. Why the first job is getting speed of 3.000 KB/sec and the second one 25.000 KB/sec?(the backup is to a file on both cases) Have bacula less performance with small files? Is everything on one single server and on the same harddrive / raid array? Yes is everything on one single server on the same raid. The first thought is that the biggest reason for the difference in backup rates is that in the case of the small files your harddrive/array is doing significantly more seeking than in the case with a few large files and depending on how they are on your media file readahead cache may be working against you. Maybe, but I try with two different servers, with different hardware. First one: Xeon 3,2Ghz; 2 hdd on SCSI raid 1. 4Gb RAM. ( Open Ent. Server ) Second one: Pentium 4 2,8Ghz; 1 SATA Disk, 2 Gb RAM. ( Open Ent Server too ) The performance on both cases is similar. So i don't think the problem is on the hardware. Have you timed the same backups using tar? It should be shorter than with bacula but does it have a similar ratio? Yes, i do it, happens the same. :( John __ Información de NOD32, revisión 2478 (20070823) __ Este mensaje ha sido analizado con NOD32 antivirus system http://www.nod32.com - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
Maybe, but I try with two different servers, with different hardware. First one: Xeon 3,2Ghz; 2 hdd on SCSI raid 1. 4Gb RAM. ( Open Ent. Server ) Second one: Pentium 4 2,8Ghz; 1 SATA Disk, 2 Gb RAM. ( Open Ent Server too ) The performance on both cases is similar. So i don't think the problem is on the hardware. My point for asking this is if the source and destination are the same disk/array that this will increase the amount of seeking because the backup will also be writing to the disk. This seeking will very much hurt performance. Remember hard drives are very fast at reading large bits of data sequentially but slow when they have to read small bits of data scattered all over the disk. Have you timed the same backups using tar? It should be shorter than with bacula but does it have a similar ratio? Yes, i do it, happens the same. :( If so this is not a bacula problem. There are a few ways you can help this. Tuning the readahead cache, possibly defragmenting the drive and possibly also changing the filesystem. For the first one there are several kernel params you can adjust and also you can use the blockdev command (provided you have a 2.6 kernel). The defragging part, may be more difficult as only a few filesystems have this ability and I am not sure how much control you have over this but basically you want to force the OS to write your files sequentially in the order that they will be accessed in the backup thus reducing the seeking to a minimum. As for the filesystem choice with journaling file systems there usually is a penalty for how much protection they gaurantee. John - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
Angel Mieres wrote: Hi all, Im testing bacula with two jobs. One of them, backup over 70.000 files and have 2 Gb. Second one have over 100 files and have 1Gb. Why the first job is getting speed of 3.000 KB/sec and the second one 25.000 KB/sec?(the backup is to a file on both cases) Have bacula less performance with small files? Thx in advance. Hi Angel, We are making abstraction about filesystem performance and consider that all files reside on the same type of filesystem. First question is : did you use compression ? If yes, different factor compression of each file could explain differences. If not, I suspect a bottleness in the database insert. running a query with 70.000 insert is quite longer than just 100 :-) Perharps you could check your DB configuration and optimize certain value to get insert goes quickly. Here even my-huge.cnf wasn't suffisent to do the job nicely. (saving about 700.000 files) If you use the lastest Bacula version the ./configure --enable-batch-insert can help also. If I don't abuse the rate is calculate by ratio kb backup in whole time. -- Bruno Friedmann [EMAIL PROTECTED] Ioda-Net Sàrl - www.ioda-net.ch 2830 Vellerat - Switzerland Tél : ++41 32 435 7171 Fax : ++41 32 435 7172 gsm : ++41 78 802 6760 C'est Facile et Cool d'Évoluer en ligne : www.cfcel.com - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
On 23 Aug 2007 at 12:53, Angel Mieres wrote: Hi all, Im testing bacula with two jobs. One of them, backup over 70.000 files and have 2 Gb. Second one have over 100 files and have 1Gb. Why the first job is getting speed of 3.000 KB/sec and the second one 25.000 KB/sec?(the backup is to a file on both cases) Have bacula less performance with small files? For each file, there is a constant amount of information to be stored in the database. If you have many small files, you have a proportionally larger amount of information to store in the database. Therefore, a 1MB file will backup faster than 1000 1KB files. Are you backing up to tape? If so, you might want to look into data spooling to see if that helps. -- Dan Langille - http://www.langille.org/ Available for hire: http://www.freebsddiary.org/dan_langille.php - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
Hi, I think that's an issue for just about any program that moves files around. For the same volume of data, the overhead (amount of work) required for asking the OS to open a file, and afterwards to close it, for the filesystem to find out the physical addresses of the pieces of that file, etc. adds up more quickly in the case where there are lots of small files, which means that that much less time can be used to work on the actual transfer of data. I don't think there's anything you can do about it. An obvious workaround, in certain situations, is to group all your files into archives (.tar, .zip...). Of course, that's a valid solution only if your files don't have to be accessed by applications, and you or your users don't mind the loss of comfort that implies when working with these files. Lucas Cohen -Message d'origine- Envoyé : jeudi 23 août 2007 12:54 Have bacula less performance with small files? - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
On 2007.08.23. 14:51, Angel Mieres wrote: I will try to answer all things that you said friends. - I don't use compression method to do a backup. - The database is postgres and during backup isn't taking all CPU or memory, only a 10% CPU. Process like bacula-fd/sd/dir exactly the same. what about io-based database load ? how high are iowait values during the backup ? - Im not backing up to a tape, the two backups are from: - /usr to a volume file stored on /tmp (~7 files, 2Gb) - /root to a volume file stored on /tmp ( ~100 files, 1,5Gb) - The way to tar or gzip the files is not possible, i need stimate to need a backup from 15 diferent servers and this could be crazy. Thanks to all. ... -- Rich - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Strange backups
Rich escribió: On 2007.08.23. 14:51, Angel Mieres wrote: I will try to answer all things that you said friends. - I don't use compression method to do a backup. - The database is postgres and during backup isn't taking all CPU or memory, only a 10% CPU. Process like bacula-fd/sd/dir exactly the same. what about io-based database load ? how high are iowait values during the backup ? This is the output of an iostat -k 2 when running backup of small files: avg-cpu: %user %nice%sys %iowait %idle 4.51 0.00 1.63 22.90 70.96 Device: tpskB_read/skB_wrtn/skB_readkB_wrtn fd0 0.00 0.00 0.00 0 0 sda 248.74 478.39 4168.84952 8296 And this is the output when large files are backing up: avg-cpu: %user %nice%sys %iowait %idle 4.370.00 9.99 38.0847.57 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn fd0 0.00 0.00 0.00 0 0 sda 288.00 29820.00 272.00 59640544 - Im not backing up to a tape, the two backups are from: - /usr to a volume file stored on /tmp (~7 files, 2Gb) - /root to a volume file stored on /tmp ( ~100 files, 1,5Gb) - The way to tar or gzip the files is not possible, i need stimate to need a backup from 15 diferent servers and this could be crazy. Thanks to all. ... - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
