Re: [Bacula-users] Deadlock error
On Thursday 2015-08-06 09:44:06 Craig Shiroma wrote: Hi Kern, Thank you for the info! We're using MySQL 5.6 Percona Server, Release 68.0, Revision 656. Would this setting cause the problem? innodb_lock_wait_timeout = 100 Is it too high or too low or has no bearing on the problem? Hi! http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout Documentation says: -BEGIN- The timeout in seconds an InnoDB transaction may wait for a row lock before giving up. The default value is 50 seconds. A transaction that tries to access a row that is locked by another InnoDB transaction will hang for at most this many seconds before issuing the following error: ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction When a lock wait timeout occurs, the current statement is not executed. The current transaction is not rolled back. -END- So I wouldn't say that decreasing this value would change anything in your case. -- Josip Deanovic -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deadlock error
Hi Ana, Thank you for the suggestion! I'll look into adding more CPU and memory to director, although I didn't see much of an impact on either between a non-accurate run and an accurate run. For example, there was large depletion of available memory, no swapping, or high load. I did add more memory to the catalog server and turned Accurate back on for the same hosts. I had no deadlocks. Last night was mostly Fulls, though. Not sure if that makes a difference. Would you know if Bacula would uses less resources when fulls are run because it is going to back up everything anyway and no comparison of files needs to be made (I'm guessing)? When a full is done, does Bacula still need to keep a list of the files in memory for hosts using Accurate backups? My first thought is no. Thanks again for the help! Your posts are always so helpful. -craig On Thu, Aug 6, 2015 at 3:37 PM, Ana Emília M. Arruda emiliaarr...@gmail.com wrote: Hello Craig, In one of your posts you mentioned Segmentation violation in the director host. Accurate backups requires more resources than normal ones. Have you checked if CPU and memory resources are enough in director and the clients that are configured for using accurate mode? Best regards, Ana On Thu, Aug 6, 2015 at 5:36 PM, Craig Shiroma shiroma.crai...@gmail.com wrote: Thanks Kern! I'll bring in a DBA on our side to have a look. Would you have any thoughts on this question posed earlier? 3. Why is Bacula spinning off a new job right away after it detects the deadlock for each affected job instead of waiting until the rescheduled job runs? I verified that there were no duplicate jobs in the queue before the backups started running, no jobs were running before the start of the backups, and I did not start any of these backups manually to cause a second job to appear. This happened on both nights I ran with Accurate turned On on the hosts that had failed backups because of the deadlock. Regards, -craig On Thu, Aug 6, 2015 at 9:48 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 21:44, Craig Shiroma wrote: Hi Kern, Thank you for the info! We're using MySQL 5.6 Percona Server, Release 68.0, Revision 656. Would this setting cause the problem? innodb_lock_wait_timeout = 100 Is it too high or too low or has no bearing on the problem? Sorry, I am a Bacula programmer, and I do not know much about databases -- especially MySQL since I use PostgreSQL. PostgreSQL is harder to install and a bit harder to configure than MySQL, but it performs much better. Thanks again, -craig On Thu, Aug 6, 2015 at 9:26 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 18:46, Bryn Hughes wrote: I think what Kern is getting at is that your database is what threw the error, not Bacula. Whatever DB you are using is what is having the issue. Yes. That is exactly what I was implying. The rest of this is directed to Craig: If you are using MariaDB (I have no indication that you are), please be aware that it may be a very good database, maybe even better than MySQL, but Bacula is built and tested against MySQL, and if you use binaries that were built for MySQL, you could run into problems by using MariaDB. Even if your binaries were explicitly built with MariaDB, it may not be compatible with the way Bacula works. Bacula has a tendency to push databases to the extreme, and it works well with MySQL and PostgreSQL, but possibly not with other databases. I bring up MariaDB because it has been mentioned in another posting to this list. I would be very surprised if your problem has anything to do with Accurate -- the database routines know nothing about accurate and none of the data is different. It is more likely due to the VM environment or to some build or version problem with MySQL (or MariaDB). Best regards, Kern Bryn On 2015-08-06 09:11 AM, Craig Shiroma wrote: Hi Kern, Thank you very much for the reply! Would you have any suggestions on what may be causing this problem or how I can debug it? Obviously, I'm encountering deadlocks when accurate backup runs on some of our hosts and we want to use accurate backup on all of our hosts if possible. Warmest regards, -craig On Thu, Aug 6, 2015 at 12:11 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 10:15, Craig Shiroma wrote: Hello again, I just thought I'd update this post with more information in hopes of getting some explanation for the deadlocks. I ran with Accurate backup on our test VMs (RHEL) for a couple of days and got the same errors on some VMs that were running accurate and some that were not. These hosts were running concurrently. I would say 90% of the hosts that were configured to use Accurate finished successfully. However, there were a few that failed with the deadlock error -- some that were configured to use accurate and some that were not configured to use accurate. Also, on all of these, a
[Bacula-users] Restoring directories with an underscore
While evaluating Bacula (+1!) for backing up a standard application, i stumbled upon the problem that i can't restore contents from within one directory. This directory is named alf_data. I'm cd'ing into this directory and am getting Invalid path given. In the restore command in bconsole: $ ls alf* alf_data $ cd alf_data Invalid path given. cwd is: /opt/alfresco-4.2.e/ I have tried to escape the underscore, like Windows users seem to need, but no luck. Has anyone this same problem? Oh, and here the disqualifier :) I installed Bacula from centos repos, so i'm on version bacula-dir Version: 5.0.0 (26 January 2010) x86_64-redhat-linux-gnu redhat Hoping to hear from one of you guys :) Koen +-- |This was sent by k...@contezza.nl via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restoring directories with an underscore
While evaluating Bacula (+1!) for backing up a standard application, i stumbled upon the problem that i can't restore contents from within one directory. This directory is named alf_data. I'm cd'ing into this directory and am getting Invalid path given. In the restore command in bconsole: $ ls alf* alf_data $ cd alf_data Invalid path given. cwd is: /opt/alfresco-4.2.e/ I have tried to escape the underscore, like Windows users seem to need, but no luck. Hello Koen: did you scape it with quotes? E.g: cd alf_data? Regards === Heitor Medrado de Faria - LPIC-III | ITIL-F | Bacula Systems Certified Administrator II Do you need Bacula training? https://www.udemy.com/bacula-backup-software/?couponCode=bacula-list +55 61 8268-4220 Site: http://bacula.us FB: heitor.faria === -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deadlock error
Hello Craig, In one of your posts you mentioned Segmentation violation in the director host. Accurate backups requires more resources than normal ones. Have you checked if CPU and memory resources are enough in director and the clients that are configured for using accurate mode? Best regards, Ana On Thu, Aug 6, 2015 at 5:36 PM, Craig Shiroma shiroma.crai...@gmail.com wrote: Thanks Kern! I'll bring in a DBA on our side to have a look. Would you have any thoughts on this question posed earlier? 3. Why is Bacula spinning off a new job right away after it detects the deadlock for each affected job instead of waiting until the rescheduled job runs? I verified that there were no duplicate jobs in the queue before the backups started running, no jobs were running before the start of the backups, and I did not start any of these backups manually to cause a second job to appear. This happened on both nights I ran with Accurate turned On on the hosts that had failed backups because of the deadlock. Regards, -craig On Thu, Aug 6, 2015 at 9:48 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 21:44, Craig Shiroma wrote: Hi Kern, Thank you for the info! We're using MySQL 5.6 Percona Server, Release 68.0, Revision 656. Would this setting cause the problem? innodb_lock_wait_timeout = 100 Is it too high or too low or has no bearing on the problem? Sorry, I am a Bacula programmer, and I do not know much about databases -- especially MySQL since I use PostgreSQL. PostgreSQL is harder to install and a bit harder to configure than MySQL, but it performs much better. Thanks again, -craig On Thu, Aug 6, 2015 at 9:26 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 18:46, Bryn Hughes wrote: I think what Kern is getting at is that your database is what threw the error, not Bacula. Whatever DB you are using is what is having the issue. Yes. That is exactly what I was implying. The rest of this is directed to Craig: If you are using MariaDB (I have no indication that you are), please be aware that it may be a very good database, maybe even better than MySQL, but Bacula is built and tested against MySQL, and if you use binaries that were built for MySQL, you could run into problems by using MariaDB. Even if your binaries were explicitly built with MariaDB, it may not be compatible with the way Bacula works. Bacula has a tendency to push databases to the extreme, and it works well with MySQL and PostgreSQL, but possibly not with other databases. I bring up MariaDB because it has been mentioned in another posting to this list. I would be very surprised if your problem has anything to do with Accurate -- the database routines know nothing about accurate and none of the data is different. It is more likely due to the VM environment or to some build or version problem with MySQL (or MariaDB). Best regards, Kern Bryn On 2015-08-06 09:11 AM, Craig Shiroma wrote: Hi Kern, Thank you very much for the reply! Would you have any suggestions on what may be causing this problem or how I can debug it? Obviously, I'm encountering deadlocks when accurate backup runs on some of our hosts and we want to use accurate backup on all of our hosts if possible. Warmest regards, -craig On Thu, Aug 6, 2015 at 12:11 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 10:15, Craig Shiroma wrote: Hello again, I just thought I'd update this post with more information in hopes of getting some explanation for the deadlocks. I ran with Accurate backup on our test VMs (RHEL) for a couple of days and got the same errors on some VMs that were running accurate and some that were not. These hosts were running concurrently. I would say 90% of the hosts that were configured to use Accurate finished successfully. However, there were a few that failed with the deadlock error -- some that were configured to use accurate and some that were not configured to use accurate. Also, on all of these, a second job started for each of the affected hosts right after Bacula detected the deadlock even though it said a reschedule would happen 3600 seconds later (the 3600 seconds is correct). Tonight, I disabled accurate on all hosts and the deadlocks did not happen. No errors were detected and all the backups finished successfully. Some questions... 1. Can I back up multiple hosts concurrently with some hosts configured to use accurate and some configured not to use accurate? Or, is it an all or none thing, meaning all hosts that run concurrently must either be using accurate backup or not using accurate backup (cannot mix the two)? 2. It seems like the hosts that get out of the starting gate first are the ones affected. I am configured to run 50 jobs concurrently. Again, no problems with accurate turned off on all hosts for months now. 3. Why is Bacula spinning off a new job right away after it detects the deadlock for each
Re: [Bacula-users] Deadlock error
On Thursday 2015-08-06 09:44:06 Craig Shiroma wrote: Hi Kern, Thank you for the info! We're using MySQL 5.6 Percona Server, Release 68.0, Revision 656. Would this setting cause the problem? innodb_lock_wait_timeout = 100 Is it too high or too low or has no bearing on the problem? Thanks again, -craig One more thing... MySQL is using MyISAM storage engine by default while MySQL Percona is using InnoDB by default. Maybe this could be the source of the problem you are experiencing. Unless you have a better idea I would suggest to try it with MyISAM storage engine. I know few applications that just can't work very well with InnoDB and I don't know if bacula has been thoroughly tested with InnoDB MySQL support. I am using bacula with both MyISAM and InnoDB with the Accurate option enabled but my jobs are usually not executing simultaneously because I can afford it due to the small number of jobs per bacula installation (less than 100 jobs and they are relatively small and quick). Optimized database and database engine could increase the database performance considerably but in your case it wouldn't solve the problem unless something is really really bad on the database side. -- Josip Deanovic -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bsmtp failing to connect to mail server
All, I have recently setup two separate bacula servers on two subnets. They were using one server and going through the firewall for one of the subnets. The previous configuration worked well but for security reasons and speed I separated them. Along with this change the company decided to move from an inhouse email server to gmail. Since the move to gmail I have not been able to get the bsmtp configuration to work. I have tried to use gmail but it requires tls. Jul 20 10:26:59 distress bacula-dir: 20-Jul 10:26 Message delivery ERROR: Mail prog: bsmtp: bsmtp.c:145 Fatal malformed reply from smtp.googlemail.com: 530 5.7 .0 Must issue a STARTTLS command first. b9sm14093516ioj.6 - gsmtp So, I have configured postfix/cyrus on the backup server just to send email. The problem I get from bsmtp now is that it can not connect to the mail server. [root@kilchis bin]# /usr/bacula/bin/bsmtp -d 25 -h kilchis.server.com -f \\(Bacula\) \no-re...@edt.com\\ -s \Bacula daemon message\ r...@server.com test message bsmtp: bsmtp.c:338-0 Debug level = 25 bsmtp: bsmtp.c:346-0 host=kilchis.server.com bsmtp: bsmtp.c:356-0 subject=Bacula bsmtp: bsmtp.c:432-0 My hostname is: kilchis bsmtp: bsmtp.c:456-0 From addr=(Bacula) bsmtp: bsmtp.c:514-0 Failed to connect to mailhost kilchis.server.com I can send mail from the command prompt and it works just fine. What am I missing? thanks -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deadlock error
MyISAM is terrible for transactions. If the deadlock is happening because of table locking then switching the engine to InnoDB might help your problem. MyISAM locks the entire table while InnoDB only locks the rows it is updating. Check out the link below. http://stackoverflow.com/questions/20148/myisam-versus-innodb On Thu, Aug 6, 2015 at 8:37 PM, Ana Emília M. Arruda emiliaarr...@gmail.com wrote: Hello Craig, In one of your posts you mentioned Segmentation violation in the director host. Accurate backups requires more resources than normal ones. Have you checked if CPU and memory resources are enough in director and the clients that are configured for using accurate mode? Best regards, Ana On Thu, Aug 6, 2015 at 5:36 PM, Craig Shiroma shiroma.crai...@gmail.com wrote: Thanks Kern! I'll bring in a DBA on our side to have a look. Would you have any thoughts on this question posed earlier? 3. Why is Bacula spinning off a new job right away after it detects the deadlock for each affected job instead of waiting until the rescheduled job runs? I verified that there were no duplicate jobs in the queue before the backups started running, no jobs were running before the start of the backups, and I did not start any of these backups manually to cause a second job to appear. This happened on both nights I ran with Accurate turned On on the hosts that had failed backups because of the deadlock. Regards, -craig On Thu, Aug 6, 2015 at 9:48 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 21:44, Craig Shiroma wrote: Hi Kern, Thank you for the info! We're using MySQL 5.6 Percona Server, Release 68.0, Revision 656. Would this setting cause the problem? innodb_lock_wait_timeout = 100 Is it too high or too low or has no bearing on the problem? Sorry, I am a Bacula programmer, and I do not know much about databases -- especially MySQL since I use PostgreSQL. PostgreSQL is harder to install and a bit harder to configure than MySQL, but it performs much better. Thanks again, -craig On Thu, Aug 6, 2015 at 9:26 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 18:46, Bryn Hughes wrote: I think what Kern is getting at is that your database is what threw the error, not Bacula. Whatever DB you are using is what is having the issue. Yes. That is exactly what I was implying. The rest of this is directed to Craig: If you are using MariaDB (I have no indication that you are), please be aware that it may be a very good database, maybe even better than MySQL, but Bacula is built and tested against MySQL, and if you use binaries that were built for MySQL, you could run into problems by using MariaDB. Even if your binaries were explicitly built with MariaDB, it may not be compatible with the way Bacula works. Bacula has a tendency to push databases to the extreme, and it works well with MySQL and PostgreSQL, but possibly not with other databases. I bring up MariaDB because it has been mentioned in another posting to this list. I would be very surprised if your problem has anything to do with Accurate -- the database routines know nothing about accurate and none of the data is different. It is more likely due to the VM environment or to some build or version problem with MySQL (or MariaDB). Best regards, Kern Bryn On 2015-08-06 09:11 AM, Craig Shiroma wrote: Hi Kern, Thank you very much for the reply! Would you have any suggestions on what may be causing this problem or how I can debug it? Obviously, I'm encountering deadlocks when accurate backup runs on some of our hosts and we want to use accurate backup on all of our hosts if possible. Warmest regards, -craig On Thu, Aug 6, 2015 at 12:11 AM, Kern Sibbald k...@sibbald.com wrote: On 06.08.2015 10:15, Craig Shiroma wrote: Hello again, I just thought I'd update this post with more information in hopes of getting some explanation for the deadlocks. I ran with Accurate backup on our test VMs (RHEL) for a couple of days and got the same errors on some VMs that were running accurate and some that were not. These hosts were running concurrently. I would say 90% of the hosts that were configured to use Accurate finished successfully. However, there were a few that failed with the deadlock error -- some that were configured to use accurate and some that were not configured to use accurate. Also, on all of these, a second job started for each of the affected hosts right after Bacula detected the deadlock even though it said a reschedule would happen 3600 seconds later (the 3600 seconds is correct). Tonight, I disabled accurate on all hosts and the deadlocks did not happen. No errors were detected and all the backups finished successfully. Some questions... 1. Can I back up multiple hosts concurrently with some hosts configured to use accurate and some configured not to use accurate? Or, is it an all or none thing, meaning all hosts
Re: [Bacula-users] Backup finished, but Fatal error: Network error with FD/Connection reset by peer
On 8/6/2015 5:09 AM, Raimund Sacherer wrote: And finally, many switches also have TCP timeout settings and/or EEE and power management that could potentially not work correctly with either the FreeBSD or the Windows network stacks. That sound's interesting, I saw a couple of posts talking about a keepalives, I will configure our FD's, SD's an the director for a 300 seconds timeout and we will see if we still get those errors. Maybe it has nothing to do with the switch to FreeBSD, because at nearly the same time we migrated our servers from physical servers to VMWare, maybe it's the virtual vmware switch which makes troubles. Well, in either case, i'l see how it goes with the keep alive configured, Fyi, version 7.x of the client daemon added progress data. The FD sends progress data to the Dir every 30 seconds. In version 5.x the Dir - FD connection sat idle during a backup. If your Windows FDs are 5.x then that could explain why they fail on the same network where the other FDs do not. -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copy jobs and multiple pools
Hello Brendan,
In each pool you are going to copy jobs from, you need a Next Pool
directive pointing to the pool to where you are going to copy jobs. I do
not see this configuration in your CSG Daily Copy, for example. Also,
regarding to the No Next Pool for pool File message, you need to
configure Next Pool for the pool File too, once you´re using this pool as
the default for your jobs. You can also override the pool definition for
your jobs, including a pool that has the next pool directive configured,
for example, set Pool = CSG Daily in your Job CSG Backup and then
override again in schedule.
The level for a copy job does not matter (this directive is not required in
version 7.X), because the copy job will do an exactly copy of the
original job (full, differential or incremental) to the new
job/volume(s)/pool.
Best regards,
Ana
On Wed, Aug 5, 2015 at 1:13 PM, Brendan Martin bren...@physics.tamu.edu
wrote:
I'm trying to get some copy jobs set up but so far all test runs have
failed. I'm not sure how to handle my pool setup relative to the copy
jobs. I may be trying to make it too complicated.
I have separate pools defined for Full, Differential and Incremental
backups. I have defined Copy pools for each of those, but my most
recent test returned an error message that there is No Next Pool for
pool File.
These are the current job stanzas and a typical sample of the pool stanzas:
JobDefs {
Name = CSG DefaultJob
Client = csg-fd
FileSet = CSG Full Set
Schedule = WeeklyCycle
Maximum Concurrent Jobs = 4
Messages = Standard
Pool = File
Full Backup Pool = CSG Monthly
Differential Backup Pool = CSG Weekly
Incremental Backup Pool = CSG Daily
Write Bootstrap = /var/lib/bacula/%c.bsr
}
Job {
Name = CSG Backup
JobDefs = CSG DefaultJob
Type = Backup
Level = Incremental
Priority = 10
Storage = Local-File
}
Job {
Name = CSG Copy
JobDefs = CSG DefaultJob
Type = Copy
Level = Full
Priority = 12
Storage = Remote-File
SelectionType = PoolUncopiedJobs
}
# CSG Daily Pool definition
Pool {
Name = CSG Daily
Label Format = CSG-Incr-
Maximum Volume Jobs = 6
Pool Type = Backup
Recycle = yes # Bacula can automatically
recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 1 month # one month
Maximum Volume Bytes = 50G # Limit Volume size to something
reasonable
Maximum Volumes = 8 # Limit number of Volumes in Pool
Next Pool = CSG Daily Copy# Pool definition for copy job
}
# CSG Daily Copy Pool definition
Pool {
Name = CSG Daily Copy
Label Format = CSG-Incr-
Maximum Volume Jobs = 6
Pool Type = Backup
Recycle = yes # Bacula can automatically
recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 1 month # one month
Maximum Volume Bytes = 50G # Limit Volume size to something
reasonable
Maximum Volumes = 8 # Limit number of Volumes in Pool
Storage = Remote-File
}
I do understand that Copy jobs work differently, but I am not
understanding all the details of how to adjust things for them. Also,
just to confirm, the Level for all Copy jobs should be Full, correct?
Thanks,
Brendan Martin
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
