Re: [Bacula-users] Network timeouts while running enormous backup
Hello, 2011/5/30 S H shdashb...@gmail.com Anyhow, I'm not sure Bacula is the right solution for this type of backup: the attribute despool has been running for nearly three days straight now. Does anybody else have experience backing up tons of files? Any recommendations on making it not suck? Check your database performance and if your environment support batch inserts (HAVE_BATCH_FILE_INSERT in config.h). best regards -- Radosław Korzeniewski rados...@korzeniewski.net -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Network timeouts while running enormous backup
On Mon, May 30, 2011 at 11:33 PM, John Drescher dresche...@gmail.com wrote: I added a network interface to the backup server on the same segment as the client and the timeout disappeared, so it seems my firewall was expiring a TCP session somewhere along the line. Unsure whether it's a configuration problem on the Bacula or firewall side -- I expected the heartbeat to prevent this. Anyhow, I'm not sure Bacula is the right solution for this type of backup: the attribute despool has been running for nearly three days straight now. That is not normal. I have seen 30 minutes for 5 or so TB backup with a few million files but never days. Do you have your database on a server with at least 8GB of ram? Is your database on a raid 10 or SSD? I think you're right that database performance is the culprit here -- the database is postgresql running on the backup server. It's a RAID-10 volume on six SATA 7200RPM drives. The database itself lives on a different OS partition on the same volume as the spools. 2GB RAM and a dual-core Xeon 5110 aren't helping. An untuned postgresql has some potential for improvement without hardware changes, at least. Does anybody else have experience backing up tons of files? There are users that backup 30 TB containing millions of files in one job on this list. So 50M files is not out of the ordinary? Thanks. Sorry for the newbie oversight. -SH -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Network timeouts while running enormous backup
On Thu, May 26, 2011 at 4:58 PM, S H shdashb...@gmail.com wrote: Hello, I've got a very large backup set (55M files, 1TB of data) that's simply not backing up. This is always the failure message: 26-May 16:35 buny1-dir JobId 5223: Fatal error: Network error with FD during Backup: ERR=Connection timed out 26-May 16:35 buny1-dir JobId 5223: Fatal error: No Job status returned from FD. I've set Heartbeat Interval = 1 minute on the FD, the SD, and the Director to no avail. This was happening at random times during the backup job until I enabled spooling -- then the job ran for 27 hours, spooled up all of its data to the SD and failed with the above message when it started to despool. It was heartbreaking to watch a terabyte of data and 20+GB of attribute spool just disappear. The Director and SD both run on the same server: OpenBSD 4.6 (32-bit) with Bacula 2.4.4. The client is FreeBSD 8.1 (64-bit) with Bacula 2..4.4. I know these are outdated versions; I just inherited the environment and they're working for everything else so I haven't wanted to go through the pain of upgrading until I have some spare cycles to dedicate to it. Compression and FD encryption are off but network traffic runs over TLS everywhere. Is there anything I can possibly check that might help? -SH I added a network interface to the backup server on the same segment as the client and the timeout disappeared, so it seems my firewall was expiring a TCP session somewhere along the line. Unsure whether it's a configuration problem on the Bacula or firewall side -- I expected the heartbeat to prevent this. Anyhow, I'm not sure Bacula is the right solution for this type of backup: the attribute despool has been running for nearly three days straight now. Does anybody else have experience backing up tons of files? Any recommendations on making it not suck? -SH -- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Network timeouts while running enormous backup
I added a network interface to the backup server on the same segment as the client and the timeout disappeared, so it seems my firewall was expiring a TCP session somewhere along the line. Unsure whether it's a configuration problem on the Bacula or firewall side -- I expected the heartbeat to prevent this. Anyhow, I'm not sure Bacula is the right solution for this type of backup: the attribute despool has been running for nearly three days straight now. That is not normal. I have seen 30 minutes for 5 or so TB backup with a few million files but never days. Do you have your database on a server with at least 8GB of ram? Is your database on a raid 10 or SSD? Does anybody else have experience backing up tons of files? There are users that backup 30 TB containing millions of files in one job on this list. Any recommendations on making it not suck? Not without any system information. John -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Network timeouts while running enormous backup
Hello, I've got a very large backup set (55M files, 1TB of data) that's simply not backing up. This is always the failure message: 26-May 16:35 buny1-dir JobId 5223: Fatal error: Network error with FD during Backup: ERR=Connection timed out 26-May 16:35 buny1-dir JobId 5223: Fatal error: No Job status returned from FD. I've set Heartbeat Interval = 1 minute on the FD, the SD, and the Director to no avail. This was happening at random times during the backup job until I enabled spooling -- then the job ran for 27 hours, spooled up all of its data to the SD and failed with the above message when it started to despool. It was heartbreaking to watch a terabyte of data and 20+GB of attribute spool just disappear. The Director and SD both run on the same server: OpenBSD 4.6 (32-bit) with Bacula 2.4.4. The client is FreeBSD 8.1 (64-bit) with Bacula 2..4.4. I know these are outdated versions; I just inherited the environment and they're working for everything else so I haven't wanted to go through the pain of upgrading until I have some spare cycles to dedicate to it. Compression and FD encryption are off but network traffic runs over TLS everywhere. Is there anything I can possibly check that might help? -SH -- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
