Re: [Bacula-users] Deadlock error

[Craig Shiroma]

Thanks Kern!  I'll bring in a DBA on our side to have a look.

Would you have any thoughts on this question posed earlier?

3. Why is Bacula spinning off a new job right away after it detects the
deadlock for each affected job instead of waiting until the rescheduled job
runs?  I verified that there were no duplicate jobs in the queue before the
backups started running, no jobs were running before the start of the
backups, and I did not start any of these backups manually to cause a
second job to appear.

This happened on both nights I ran with Accurate turned On on the hosts
that had failed backups because of the deadlock.

Regards,
-craig

On Thu, Aug 6, 2015 at 9:48 AM, Kern Sibbald  wrote:

> On 06.08.2015 21:44, Craig Shiroma wrote:
>
> Hi Kern,
>
> Thank you for the info!  We're using MySQL 5.6 Percona Server, Release
> 68.0, Revision 656.
>
> Would this setting cause the problem?
> innodb_lock_wait_timeout = 100
>
> Is it too high or too low or has no bearing on the problem?
>
>
> Sorry, I am a Bacula programmer, and I do not know much about databases --
> especially MySQL since I use PostgreSQL.  PostgreSQL is harder to install
> and a bit harder to configure than MySQL, but it performs much better.
>
>
>
> Thanks again,
> -craig
>
>
> On Thu, Aug 6, 2015 at 9:26 AM, Kern Sibbald  wrote:
>
>> On 06.08.2015 18:46, Bryn Hughes wrote:
>>
>> I think what Kern is getting at is that your database is what threw the
>> error, not Bacula.  Whatever DB you are using is what is having the issue.
>>
>>
>> Yes.  That is exactly what I was implying.
>>
>> The rest of this is directed to Craig:
>> If you are using MariaDB (I have no indication that you are), please be
>> aware that it may be a very good database, maybe even better than MySQL,
>> but Bacula is built and tested against MySQL, and if you use binaries that
>> were built for MySQL, you could run into problems by using MariaDB.  Even
>> if your binaries were explicitly built with MariaDB, it may not be
>> compatible with the way Bacula works.  Bacula has a tendency to push
>> databases to the extreme, and it works well with MySQL and PostgreSQL, but
>> possibly not with other databases.  I bring up MariaDB because it has been
>> mentioned in another posting to this list.
>>
>> I would be very surprised if your problem has anything to do with
>> Accurate -- the database routines know nothing about accurate and none of
>> the data is different.  It is more likely due to the VM environment or to
>> some build or version problem with MySQL (or MariaDB).
>>
>> Best regards,
>> Kern
>>
>>
>> Bryn
>>
>> On 2015-08-06 09:11 AM, Craig Shiroma wrote:
>>
>> Hi Kern,
>>
>> Thank you very much for the reply!  Would you have any suggestions on
>> what may be causing this problem or how I can debug it?  Obviously, I'm
>> encountering deadlocks when accurate backup runs on some of our hosts and
>> we want to use accurate backup on all of our hosts if possible.
>>
>> Warmest regards,
>> -craig
>>
>> On Thu, Aug 6, 2015 at 12:11 AM, Kern Sibbald  wrote:
>>
>>> On 06.08.2015 10:15, Craig Shiroma wrote:
>>>
>>> Hello again,
>>>
>>> I just thought I'd update this post with more information in hopes of
>>> getting some explanation for the deadlocks.
>>>
>>> I ran with Accurate backup on our test VMs (RHEL) for a couple of days
>>> and got the same errors on some VMs that were running accurate and some
>>> that were not.  These hosts were running concurrently.  I would say 90% of
>>> the hosts that were configured to use Accurate finished successfully.
>>> However, there were a few that failed with the deadlock error -- some that
>>> were configured to use accurate and some that were not configured to use
>>> accurate.  Also, on all of these, a second job started for each of the
>>> affected hosts right after Bacula detected the deadlock even though it said
>>> a reschedule would happen 3600 seconds later (the 3600 seconds is correct).
>>>
>>> Tonight, I disabled accurate on all hosts and the deadlocks did not
>>> happen.  No errors were detected and all the backups finished successfully.
>>>
>>> Some questions...
>>> 1.  Can I back up multiple hosts concurrently with some hosts configured
>>> to use accurate and some configured not to use accurate?  Or, is it an all
>>> or none thing, meaning all hosts that run concurrently must either be using
>>> accurate backup or not using accurate backup (cannot mix the two)?
>>>
>>> 2. It seems like the hosts that get out of the starting gate first are
>>> the ones affected.  I am configured to run 50 jobs concurrently.  Again, no
>>> problems with accurate turned off on all hosts for months now.
>>>
>>> 3. Why is Bacula spinning off a new job right away after it detects the
>>> deadlock for each affected job instead of waiting until the rescheduled job
>>> runs?  I verified that there were no duplicate jobs in the queue before the
>>> backups started running, no jobs were running before the start of the
>>> backups, and I did not sta

Re: [Bacula-users] Waiting for appendable volume

[Gilberto Nunes]

Hello Ana..

For now, I split the file to make the backups...
And it's works...
Later, I will see the results and ask for same others instrutions...

Thanks

2015-08-06 17:26 GMT-03:00 Ana Emília M. Arruda :

> Hélio Gilberto,
>
> Yes. You can have full, differentiall and incremental backups in the same
> volume/pool. Bacula will mark your volume full of it reaches the maximum
> volume bytes for this volume. Have you had configured maximum volume bytes
> for the volume being marked full?
>
> Could you send the results for list media and show pool ?
>
> Best regards,
> Ana
> Em qui, 6 de ago de 2015 às 11:49, Gilberto Nunes <
> gilberto.nune...@gmail.com> escreveu:
>
>> Hi folks...
>>
>> That's my first post in this mailing list and I think I have a very
>> unrare problem...
>>
>> I have this in bacula-sd.conf
>>
>> Device {
>>   Name = bacula
>>   Archive Device = /media/2c0bc75b-f54e-4f31-8d8c-5568edc0e765/bacula
>>   Media Type = File
>>   LabelMedia = yes
>>   Random Access = yes
>>   AutomaticMount = yes
>>   RemovableMedia = no
>>   AlwaysOpen = yes
>> }
>>
>> And this as Pool definition:
>>
>> Pool {
>>   Name = Servidor
>>   Pool Type = Backup
>>   Recycle = yes
>>   AutoPrune = yes
>>   Volume Retention = 365 days
>>   Maximum Volumes = 100   # Limit number of Volumes in Pool
>> }
>>
>> I have labeled the volume as Bkp-Servidor
>>
>> I am also using the schedule and job above:
>>
>> Schedule {
>>   Name = Servidor
>>   Run = Level=Full on 5 at 10:00
>>   Run = Level=Differential at 22:30
>> }
>> Job {
>>   Name = Servidor-Job
>>   Type = Backup
>>   Level = Full
>>   Client = Servidor
>>   FileSet = Servidor
>>   Schedule = Servidor
>>   Storage = File
>>   Pool = Servidor
>>   Messages = Standard
>> }
>>
>> What I need is make Full and Differential backup use a single file disk...
>>
>> Is that possible?
>>
>> Why the file Bkp-Servidor (which is the file where baculs writes the
>> backups), changes to Full and ask me to appendable volume?
>>
>> Perhaps I missconfigure somethink...
>>
>> Can any one assist me??
>>
>>
>>
>>
>>
>> --
>>
>> Gilberto Ferreira
>> +55 (47) 9676-7530
>> Skype: gilberto.nunes36
>>
>>
>> --
>> ___
>> Bacula-users mailing list
>> Bacula-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>


-- 

Gilberto Ferreira
+55 (47) 9676-7530
Skype: gilberto.nunes36
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Waiting for appendable volume

[Ana Emília M . Arruda]

Hélio Gilberto,

Yes. You can have full, differentiall and incremental backups in the same
volume/pool. Bacula will mark your volume full of it reaches the maximum
volume bytes for this volume. Have you had configured maximum volume bytes
for the volume being marked full?

Could you send the results for list media and show pool ?

Best regards,
Ana
Em qui, 6 de ago de 2015 às 11:49, Gilberto Nunes <
gilberto.nune...@gmail.com> escreveu:

> Hi folks...
>
> That's my first post in this mailing list and I think I have a very unrare
> problem...
>
> I have this in bacula-sd.conf
>
> Device {
>   Name = bacula
>   Archive Device = /media/2c0bc75b-f54e-4f31-8d8c-5568edc0e765/bacula
>   Media Type = File
>   LabelMedia = yes
>   Random Access = yes
>   AutomaticMount = yes
>   RemovableMedia = no
>   AlwaysOpen = yes
> }
>
> And this as Pool definition:
>
> Pool {
>   Name = Servidor
>   Pool Type = Backup
>   Recycle = yes
>   AutoPrune = yes
>   Volume Retention = 365 days
>   Maximum Volumes = 100   # Limit number of Volumes in Pool
> }
>
> I have labeled the volume as Bkp-Servidor
>
> I am also using the schedule and job above:
>
> Schedule {
>   Name = Servidor
>   Run = Level=Full on 5 at 10:00
>   Run = Level=Differential at 22:30
> }
> Job {
>   Name = Servidor-Job
>   Type = Backup
>   Level = Full
>   Client = Servidor
>   FileSet = Servidor
>   Schedule = Servidor
>   Storage = File
>   Pool = Servidor
>   Messages = Standard
> }
>
> What I need is make Full and Differential backup use a single file disk...
>
> Is that possible?
>
> Why the file Bkp-Servidor (which is the file where baculs writes the
> backups), changes to Full and ask me to appendable volume?
>
> Perhaps I missconfigure somethink...
>
> Can any one assist me??
>
>
>
>
>
> --
>
> Gilberto Ferreira
> +55 (47) 9676-7530
> Skype: gilberto.nunes36
>
>
> --
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Deadlock error

[Kern Sibbald]

  
  
On 06.08.2015 21:44, Craig Shiroma
  wrote:


  Hi Kern,


Thank you for the info!  We're using MySQL 5.6 Percona
  Server, Release 68.0, Revision 656.


Would this setting cause the problem?
innodb_lock_wait_timeout
= 100



Is it too high or too low or has no bearing on the problem?
  


Sorry, I am a Bacula programmer, and I do not know much about
databases -- especially MySQL since I use PostgreSQL.  PostgreSQL is
harder to install and a bit harder to configure than MySQL, but it
performs much better.


  


Thanks again,
-craig


  
  
On Thu, Aug 6, 2015 at 9:26 AM, Kern
  Sibbald 
  wrote:
  

On 06.08.2015 18:46, Bryn Hughes wrote:


  I think what Kern is getting at is that your
database is what threw the error, not Bacula. 
Whatever DB you are using is what is having the
issue.
  


   Yes.  That is exactly what I was implying.  
  
  The rest of this is directed to Craig:
  If you are using MariaDB (I have no indication that you
  are), please be aware that it may be a very good database,
  maybe even better than MySQL, but Bacula is built and
  tested against MySQL, and if you use binaries that were
  built for MySQL, you could run into problems by using
  MariaDB.  Even if your binaries were explicitly built with
  MariaDB, it may not be compatible with the way Bacula
  works.  Bacula has a tendency to push databases to the
  extreme, and it works well with MySQL and PostgreSQL, but
  possibly not with other databases.  I bring up MariaDB
  because it has been mentioned in another posting to this
  list.
  
  I would be very surprised if your problem has anything to
  do with Accurate -- the database routines know nothing
  about accurate and none of the data is different.  It is
  more likely due to the VM environment or to some build or
  version problem with MySQL (or MariaDB).
  
  Best regards,
  Kern
  

  
  
 Bryn
  
  On 2015-08-06 09:11 AM, Craig Shiroma wrote:


  Hi Kern,


Thank you very much for the reply!  Would
  you have any suggestions on what may be
  causing this problem or how I can debug it? 
  Obviously, I'm encountering deadlocks when
  accurate backup runs on some of our hosts and
  we want to use accurate backup on all of our
  hosts if possible.


Warmest regards,
-craig
  
  
On Thu, Aug 6, 2015 at
  12:11 AM, Kern Sibbald 
  wrote:
  

On 06.08.2015 10:15, Craig Shiroma
  wrote:


  Hello again,


I just thought I'd update this
  post with more information in
  hopes of getting some explanation
  for the deadlocks.  


I ran with Accurate backup on
  our test VMs (RHEL) for a couple
  of days and got the same errors on
  some VMs that were running
  accurate and some that were not. 
  These hosts were running
  concurrently.  I would say 90% of
  the hosts that were configured to
  use Accurate finished
 

Re: [Bacula-users] Deadlock error

[Kern Sibbald]

  
  
On 06.08.2015 21:36, Craig Shiroma
  wrote:


  Hi Bryn,


Thank you for the translation!  :-)  Much appreciated. 
  I'll ask our DBA to take a look at the DB (mysql).  Maybe it
  needs some tuning for Accurate.  Do you know of any
  documentation for this?
  


Type: "tuning mysql for bacula" into your browser ...




  
 I only saw a couple of small sections for Accurate in the
  manual, mainly how to turn it on and that it uses more
  resources.  I haven't had a chance to read the whole manual
  yet so I might have missed the section.


-craig
  
  
On Thu, Aug 6, 2015 at 6:46 AM, Bryn
  Hughes 
  wrote:
  

  I think what Kern is getting at is that your database
is what threw the error, not Bacula.  Whatever DB you
are using is what is having the issue.

Bryn

  

On 2015-08-06 09:11 AM, Craig Shiroma wrote:
  

  
  

  
Hi Kern,
  
  
  Thank you very much for the reply!  Would you
have any suggestions on what may be causing this
problem or how I can debug it?  Obviously, I'm
encountering deadlocks when accurate backup runs
on some of our hosts and we want to use accurate
backup on all of our hosts if possible.
  
  
  Warmest regards,
  -craig


  On Thu, Aug 6, 2015 at
12:11 AM, Kern Sibbald 
wrote:

  
  On 06.08.2015 10:15, Craig Shiroma
wrote:
  
  
Hello again,
  
  
  I just thought I'd update this
post with more information in hopes
of getting some explanation for the
deadlocks.  
  
  
  I ran with Accurate backup on our
test VMs (RHEL) for a couple of days
and got the same errors on some VMs
that were running accurate and some
that were not.  These hosts were
running concurrently.  I would say
90% of the hosts that were
configured to use Accurate finished
successfully.  However, there were a
few that failed with the deadlock
error -- some that were configured
to use accurate and some that were
not configured to use accurate. 
Also, on all of these, a second job
started for each of the affected
hosts right after Bacula detected
the deadlock even though it said a
reschedule would happen 3600 seconds
later (the 3600 seconds is correct).
  
  
  Tonight, I disabled accurate on
all hosts and the deadlocks did not
happen.  No errors were detected and
all the backups finished
successfully.
  
  
  Some questions...
  1.  Can I back up multiple hosts
concurrently with some hosts
configured to use accurate and some
configured not to use accurate?  Or,
is it an all or none thing, meaning
   

Re: [Bacula-users] Deadlock error

[Craig Shiroma]

Hi Kern,

Thank you for the info!  We're using MySQL 5.6 Percona Server, Release
68.0, Revision 656.

Would this setting cause the problem?
innodb_lock_wait_timeout = 100

Is it too high or too low or has no bearing on the problem?

Thanks again,
-craig


On Thu, Aug 6, 2015 at 9:26 AM, Kern Sibbald  wrote:

> On 06.08.2015 18:46, Bryn Hughes wrote:
>
> I think what Kern is getting at is that your database is what threw the
> error, not Bacula.  Whatever DB you are using is what is having the issue.
>
>
> Yes.  That is exactly what I was implying.
>
> The rest of this is directed to Craig:
> If you are using MariaDB (I have no indication that you are), please be
> aware that it may be a very good database, maybe even better than MySQL,
> but Bacula is built and tested against MySQL, and if you use binaries that
> were built for MySQL, you could run into problems by using MariaDB.  Even
> if your binaries were explicitly built with MariaDB, it may not be
> compatible with the way Bacula works.  Bacula has a tendency to push
> databases to the extreme, and it works well with MySQL and PostgreSQL, but
> possibly not with other databases.  I bring up MariaDB because it has been
> mentioned in another posting to this list.
>
> I would be very surprised if your problem has anything to do with Accurate
> -- the database routines know nothing about accurate and none of the data
> is different.  It is more likely due to the VM environment or to some build
> or version problem with MySQL (or MariaDB).
>
> Best regards,
> Kern
>
>
> Bryn
>
> On 2015-08-06 09:11 AM, Craig Shiroma wrote:
>
> Hi Kern,
>
> Thank you very much for the reply!  Would you have any suggestions on what
> may be causing this problem or how I can debug it?  Obviously, I'm
> encountering deadlocks when accurate backup runs on some of our hosts and
> we want to use accurate backup on all of our hosts if possible.
>
> Warmest regards,
> -craig
>
> On Thu, Aug 6, 2015 at 12:11 AM, Kern Sibbald  wrote:
>
>> On 06.08.2015 10:15, Craig Shiroma wrote:
>>
>> Hello again,
>>
>> I just thought I'd update this post with more information in hopes of
>> getting some explanation for the deadlocks.
>>
>> I ran with Accurate backup on our test VMs (RHEL) for a couple of days
>> and got the same errors on some VMs that were running accurate and some
>> that were not.  These hosts were running concurrently.  I would say 90% of
>> the hosts that were configured to use Accurate finished successfully.
>> However, there were a few that failed with the deadlock error -- some that
>> were configured to use accurate and some that were not configured to use
>> accurate.  Also, on all of these, a second job started for each of the
>> affected hosts right after Bacula detected the deadlock even though it said
>> a reschedule would happen 3600 seconds later (the 3600 seconds is correct).
>>
>> Tonight, I disabled accurate on all hosts and the deadlocks did not
>> happen.  No errors were detected and all the backups finished successfully.
>>
>> Some questions...
>> 1.  Can I back up multiple hosts concurrently with some hosts configured
>> to use accurate and some configured not to use accurate?  Or, is it an all
>> or none thing, meaning all hosts that run concurrently must either be using
>> accurate backup or not using accurate backup (cannot mix the two)?
>>
>> 2. It seems like the hosts that get out of the starting gate first are
>> the ones affected.  I am configured to run 50 jobs concurrently.  Again, no
>> problems with accurate turned off on all hosts for months now.
>>
>> 3. Why is Bacula spinning off a new job right away after it detects the
>> deadlock for each affected job instead of waiting until the rescheduled job
>> runs?  I verified that there were no duplicate jobs in the queue before the
>> backups started running, no jobs were running before the start of the
>> backups, and I did not start any of these backups manually to cause a
>> second job to appear.
>>
>>
>> Bacula is not aware of any SQL internal deadlocks.
>>
>>
>> From the INNODB Monitor output:
>>
>> TRANSACTION:
>> TRANSACTION 208788977, ACTIVE 1 sec setting auto-inc lock
>> mysql tables in use 4, locked 4
>> 9 lock struct(s), heap size 1184, 5 row lock(s)
>> MySQL thread id 50808, OS thread handle 0x7f8f2c3b4700, query id 29558637
>>  192.168.10.99 bacula Sending data
>> INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5,
>> DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId,
>> Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN
>> Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name =
>> Filename.Name)
>> WAITING FOR THIS LOCK TO BE GRANTED:
>> TABLE LOCK table `bacula`.`File` trx id 208788977 lock mode AUTO-INC
>> waiting
>> WE ROLL BACK TRANSACTION (2)
>>
>> I am running Bacula 7.0.5 on RHEL 6.6 x64 with Director, Storage and
>> Catalog running on separate RHEL 6.6 hosts.  Our clients are RHEL 6's, 5's
>> and Windows Servers 2008 and 

Re: [Bacula-users] Deadlock error

[Craig Shiroma]

Hi Bryn,

Thank you for the translation!  :-)  Much appreciated.  I'll ask our DBA to
take a look at the DB (mysql).  Maybe it needs some tuning for Accurate.
Do you know of any documentation for this?  I only saw a couple of small
sections for Accurate in the manual, mainly how to turn it on and that it
uses more resources.  I haven't had a chance to read the whole manual yet
so I might have missed the section.

-craig

On Thu, Aug 6, 2015 at 6:46 AM, Bryn Hughes  wrote:

> I think what Kern is getting at is that your database is what threw the
> error, not Bacula.  Whatever DB you are using is what is having the issue.
>
> Bryn
>
>
> On 2015-08-06 09:11 AM, Craig Shiroma wrote:
>
> Hi Kern,
>
> Thank you very much for the reply!  Would you have any suggestions on what
> may be causing this problem or how I can debug it?  Obviously, I'm
> encountering deadlocks when accurate backup runs on some of our hosts and
> we want to use accurate backup on all of our hosts if possible.
>
> Warmest regards,
> -craig
>
> On Thu, Aug 6, 2015 at 12:11 AM, Kern Sibbald  wrote:
>
>> On 06.08.2015 10:15, Craig Shiroma wrote:
>>
>> Hello again,
>>
>> I just thought I'd update this post with more information in hopes of
>> getting some explanation for the deadlocks.
>>
>> I ran with Accurate backup on our test VMs (RHEL) for a couple of days
>> and got the same errors on some VMs that were running accurate and some
>> that were not.  These hosts were running concurrently.  I would say 90% of
>> the hosts that were configured to use Accurate finished successfully.
>> However, there were a few that failed with the deadlock error -- some that
>> were configured to use accurate and some that were not configured to use
>> accurate.  Also, on all of these, a second job started for each of the
>> affected hosts right after Bacula detected the deadlock even though it said
>> a reschedule would happen 3600 seconds later (the 3600 seconds is correct).
>>
>> Tonight, I disabled accurate on all hosts and the deadlocks did not
>> happen.  No errors were detected and all the backups finished successfully.
>>
>> Some questions...
>> 1.  Can I back up multiple hosts concurrently with some hosts configured
>> to use accurate and some configured not to use accurate?  Or, is it an all
>> or none thing, meaning all hosts that run concurrently must either be using
>> accurate backup or not using accurate backup (cannot mix the two)?
>>
>> 2. It seems like the hosts that get out of the starting gate first are
>> the ones affected.  I am configured to run 50 jobs concurrently.  Again, no
>> problems with accurate turned off on all hosts for months now.
>>
>> 3. Why is Bacula spinning off a new job right away after it detects the
>> deadlock for each affected job instead of waiting until the rescheduled job
>> runs?  I verified that there were no duplicate jobs in the queue before the
>> backups started running, no jobs were running before the start of the
>> backups, and I did not start any of these backups manually to cause a
>> second job to appear.
>>
>>
>> Bacula is not aware of any SQL internal deadlocks.
>>
>>
>> From the INNODB Monitor output:
>>
>> TRANSACTION:
>> TRANSACTION 208788977, ACTIVE 1 sec setting auto-inc lock
>> mysql tables in use 4, locked 4
>> 9 lock struct(s), heap size 1184, 5 row lock(s)
>> MySQL thread id 50808, OS thread handle 0x7f8f2c3b4700, query id 29558637
>>  192.168.10.99 bacula Sending data
>> INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5,
>> DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId,
>> Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN
>> Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name =
>> Filename.Name)
>> WAITING FOR THIS LOCK TO BE GRANTED:
>> TABLE LOCK table `bacula`.`File` trx id 208788977 lock mode AUTO-INC
>> waiting
>> WE ROLL BACK TRANSACTION (2)
>>
>> I am running Bacula 7.0.5 on RHEL 6.6 x64 with Director, Storage and
>> Catalog running on separate RHEL 6.6 hosts.  Our clients are RHEL 6's, 5's
>> and Windows Servers 2008 and 2012R2.
>>
>> Any help would be much appreciated.
>>
>> Warmest regards,
>> -craig
>>
>> On Tue, Aug 4, 2015 at 1:56 PM, Craig Shiroma 
>> wrote:
>>
>>> BTW, I suppose there could've been two jobs for the host(s) in
>>> scheduling queue.  If this was the case, is there a way to find out after
>>> the fact?  If this did actually happen, what could cause duplicate jobs to
>>> be scheduled on the same day at the same time?  I know no one manually ran
>>> the jobs in question.  Again, this only was a problem for a few of the jobs
>>> that ran last night, not all of them and some to do accurate backup and
>>> some not.
>>>
>>> Regards,
>>> -craig
>>>
>>> On Tue, Aug 4, 2015 at 9:27 AM, Craig Shiroma >> > wrote:
>>>
 Hello,

 I had a few backups fail last night with the following error:

 2015-08-03 18:02:46bacula-dir JobId 123984: b INTO File (FileIndex,
 JobId, PathId, FilenameId,

Re: [Bacula-users] Deadlock error

[Kern Sibbald]

  
  
On 06.08.2015 18:46, Bryn Hughes wrote:


  
  I think what Kern is getting at is
that your database is what threw the error, not Bacula. 
Whatever DB you are using is what is having the issue.
  


Yes.  That is exactly what I was implying.  

The rest of this is directed to Craig:
If you are using MariaDB (I have no indication that you are), please
be aware that it may be a very good database, maybe even better than
MySQL, but Bacula is built and tested against MySQL, and if you use
binaries that were built for MySQL, you could run into problems by
using MariaDB.  Even if your binaries were explicitly built with
MariaDB, it may not be compatible with the way Bacula works.  Bacula
has a tendency to push databases to the extreme, and it works well
with MySQL and PostgreSQL, but possibly not with other databases.  I
bring up MariaDB because it has been mentioned in another posting to
this list.

I would be very surprised if your problem has anything to do with
Accurate -- the database routines know nothing about accurate and
none of the data is different.  It is more likely due to the VM
environment or to some build or version problem with MySQL (or
MariaDB).

Best regards,
Kern


   Bryn

On 2015-08-06 09:11 AM, Craig Shiroma wrote:
  
  
Hi Kern,
  
  
  Thank you very much for the reply!  Would you have any
suggestions on what may be causing this problem or how I can
debug it?  Obviously, I'm encountering deadlocks when
accurate backup runs on some of our hosts and we want to use
accurate backup on all of our hosts if possible.
  
  
  Warmest regards,
  -craig


  On Thu, Aug 6, 2015 at 12:11 AM, Kern
Sibbald 
wrote:

  
  On 06.08.2015 10:15, Craig Shiroma wrote:
  
  
Hello again,
  
  
  I just thought I'd update this post with more
information in hopes of getting some explanation
for the deadlocks.  
  
  
  I ran with Accurate backup on our test VMs
(RHEL) for a couple of days and got the same
errors on some VMs that were running accurate
and some that were not.  These hosts were
running concurrently.  I would say 90% of the
hosts that were configured to use Accurate
finished successfully.  However, there were a
few that failed with the deadlock error -- some
that were configured to use accurate and some
that were not configured to use accurate.  Also,
on all of these, a second job started for each
of the affected hosts right after Bacula
detected the deadlock even though it said a
reschedule would happen 3600 seconds later (the
3600 seconds is correct).
  
  
  Tonight, I disabled accurate on all hosts and
the deadlocks did not happen.  No errors were
detected and all the backups finished
successfully.
  
  
  Some questions...
  1.  Can I back up multiple hosts concurrently
with some hosts configured to use accurate and
some configured not to use accurate?  Or, is it
an all or none thing, meaning all hosts that run
concurrently must either be using accurate
backup or not using accurate backup (cannot mix
the two)?
  
  
  2. It seems like the hosts that get out of
the starting gate first are the ones affected. 
I am configured to run 50 jobs concurrently. 
Again, no problems with accurate turned off on
all hosts for months now.
  
  
  3. Why is Bacula spinning off a new job right
away after it detects the deadlock for each
affected job instead of waiting until the
rescheduled job runs?  I verified that there
 

Re: [Bacula-users] Deadlock error

[Bryn Hughes]

I think what Kern is getting at is that your database is what threw the 
error, not Bacula.  Whatever DB you are using is what is having the issue.


Bryn

On 2015-08-06 09:11 AM, Craig Shiroma wrote:

Hi Kern,

Thank you very much for the reply!  Would you have any suggestions on 
what may be causing this problem or how I can debug it?  Obviously, 
I'm encountering deadlocks when accurate backup runs on some of our 
hosts and we want to use accurate backup on all of our hosts if possible.


Warmest regards,
-craig

On Thu, Aug 6, 2015 at 12:11 AM, Kern Sibbald > wrote:


On 06.08.2015 10:15, Craig Shiroma wrote:

Hello again,

I just thought I'd update this post with more information in
hopes of getting some explanation for the deadlocks.

I ran with Accurate backup on our test VMs (RHEL) for a couple of
days and got the same errors on some VMs that were running
accurate and some that were not.  These hosts were running
concurrently.  I would say 90% of the hosts that were configured
to use Accurate finished successfully.  However, there were a few
that failed with the deadlock error -- some that were configured
to use accurate and some that were not configured to use
accurate.  Also, on all of these, a second job started for each
of the affected hosts right after Bacula detected the deadlock
even though it said a reschedule would happen 3600 seconds later
(the 3600 seconds is correct).

Tonight, I disabled accurate on all hosts and the deadlocks did
not happen.  No errors were detected and all the backups finished
successfully.

Some questions...
1.  Can I back up multiple hosts concurrently with some hosts
configured to use accurate and some configured not to use
accurate?  Or, is it an all or none thing, meaning all hosts that
run concurrently must either be using accurate backup or not
using accurate backup (cannot mix the two)?

2. It seems like the hosts that get out of the starting gate
first are the ones affected.  I am configured to run 50 jobs
concurrently.  Again, no problems with accurate turned off on all
hosts for months now.

3. Why is Bacula spinning off a new job right away after it
detects the deadlock for each affected job instead of waiting
until the rescheduled job runs?  I verified that there were no
duplicate jobs in the queue before the backups started running,
no jobs were running before the start of the backups, and I did
not start any of these backups manually to cause a second job to
appear.


Bacula is not aware of any SQL internal deadlocks.



From the INNODB Monitor output:

TRANSACTION:
TRANSACTION 208788977, ACTIVE 1 sec setting auto-inc lock
mysql tables in use 4, locked 4
9 lock struct(s), heap size 1184, 5 row lock(s)
MySQL thread id 50808, OS thread handle 0x7f8f2c3b4700, query id
29558637  192.168.10.99 bacula Sending data
INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat,
MD5, DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId,
Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM
batch JOIN Path ON (batch.Path = Path.Path) JOIN Filename ON
(batch.Name = Filename.Name)
WAITING FOR THIS LOCK TO BE GRANTED:
TABLE LOCK table `bacula`.`File` trx id 208788977 lock mode
AUTO-INC waiting
WE ROLL BACK TRANSACTION (2)

I am running Bacula 7.0.5 on RHEL 6.6 x64 with Director, Storage
and Catalog running on separate RHEL 6.6 hosts.  Our clients are
RHEL 6's, 5's and Windows Servers 2008 and 2012R2.

Any help would be much appreciated.

Warmest regards,
-craig

On Tue, Aug 4, 2015 at 1:56 PM, Craig Shiroma
mailto:shiroma.crai...@gmail.com>> wrote:

BTW, I suppose there could've been two jobs for the host(s)
in scheduling queue.  If this was the case, is there a way to
find out after the fact?  If this did actually happen, what
could cause duplicate jobs to be scheduled on the same day at
the same time?  I know no one manually ran the jobs in
question.  Again, this only was a problem for a few of the
jobs that ran last night, not all of them and some to do
accurate backup and some not.

Regards,
-craig

On Tue, Aug 4, 2015 at 9:27 AM, Craig Shiroma
mailto:shiroma.crai...@gmail.com>> wrote:

Hello,

I had a few backups fail last night with the following error:

2015-08-03 18:02:46bacula-dir JobId 123984: b INTO File
(FileIndex, JobId, PathId, FilenameId, LStat, MD5,
DeltaSeq) SELECT batch.FileIndex, batch.JobId,
Path.PathId, Filename.FilenameId,batch.LStat, batch.MD5,
batch.DeltaSeq FROM batch JOIN Path ON (batch.Path =
Path.Path) JOIN Filename ON (batch.Name = Filename.Name):
ERR=Deadlock 

Re: [Bacula-users] Deadlock error

[Craig Shiroma]

Hi Kern,

Thank you very much for the reply!  Would you have any suggestions on what
may be causing this problem or how I can debug it?  Obviously, I'm
encountering deadlocks when accurate backup runs on some of our hosts and
we want to use accurate backup on all of our hosts if possible.

Warmest regards,
-craig

On Thu, Aug 6, 2015 at 12:11 AM, Kern Sibbald  wrote:

> On 06.08.2015 10:15, Craig Shiroma wrote:
>
> Hello again,
>
> I just thought I'd update this post with more information in hopes of
> getting some explanation for the deadlocks.
>
> I ran with Accurate backup on our test VMs (RHEL) for a couple of days and
> got the same errors on some VMs that were running accurate and some that
> were not.  These hosts were running concurrently.  I would say 90% of the
> hosts that were configured to use Accurate finished successfully.  However,
> there were a few that failed with the deadlock error -- some that were
> configured to use accurate and some that were not configured to use
> accurate.  Also, on all of these, a second job started for each of the
> affected hosts right after Bacula detected the deadlock even though it said
> a reschedule would happen 3600 seconds later (the 3600 seconds is correct).
>
> Tonight, I disabled accurate on all hosts and the deadlocks did not
> happen.  No errors were detected and all the backups finished successfully.
>
> Some questions...
> 1.  Can I back up multiple hosts concurrently with some hosts configured
> to use accurate and some configured not to use accurate?  Or, is it an all
> or none thing, meaning all hosts that run concurrently must either be using
> accurate backup or not using accurate backup (cannot mix the two)?
>
> 2. It seems like the hosts that get out of the starting gate first are the
> ones affected.  I am configured to run 50 jobs concurrently.  Again, no
> problems with accurate turned off on all hosts for months now.
>
> 3. Why is Bacula spinning off a new job right away after it detects the
> deadlock for each affected job instead of waiting until the rescheduled job
> runs?  I verified that there were no duplicate jobs in the queue before the
> backups started running, no jobs were running before the start of the
> backups, and I did not start any of these backups manually to cause a
> second job to appear.
>
>
> Bacula is not aware of any SQL internal deadlocks.
>
>
> From the INNODB Monitor output:
>
> TRANSACTION:
> TRANSACTION 208788977, ACTIVE 1 sec setting auto-inc lock
> mysql tables in use 4, locked 4
> 9 lock struct(s), heap size 1184, 5 row lock(s)
> MySQL thread id 50808, OS thread handle 0x7f8f2c3b4700, query id 29558637
>  192.168.10.99 bacula Sending data
> INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5,
> DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId,
> Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN
> Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name =
> Filename.Name)
> WAITING FOR THIS LOCK TO BE GRANTED:
> TABLE LOCK table `bacula`.`File` trx id 208788977 lock mode AUTO-INC
> waiting
> WE ROLL BACK TRANSACTION (2)
>
> I am running Bacula 7.0.5 on RHEL 6.6 x64 with Director, Storage and
> Catalog running on separate RHEL 6.6 hosts.  Our clients are RHEL 6's, 5's
> and Windows Servers 2008 and 2012R2.
>
> Any help would be much appreciated.
>
> Warmest regards,
> -craig
>
> On Tue, Aug 4, 2015 at 1:56 PM, Craig Shiroma 
> wrote:
>
>> BTW, I suppose there could've been two jobs for the host(s) in scheduling
>> queue.  If this was the case, is there a way to find out after the fact?
>> If this did actually happen, what could cause duplicate jobs to be
>> scheduled on the same day at the same time?  I know no one manually ran the
>> jobs in question.  Again, this only was a problem for a few of the jobs
>> that ran last night, not all of them and some to do accurate backup and
>> some not.
>>
>> Regards,
>> -craig
>>
>> On Tue, Aug 4, 2015 at 9:27 AM, Craig Shiroma 
>> wrote:
>>
>>> Hello,
>>>
>>> I had a few backups fail last night with the following error:
>>>
>>> 2015-08-03 18:02:46bacula-dir JobId 123984: b INTO File (FileIndex,
>>> JobId, PathId, FilenameId, LStat, MD5, DeltaSeq) SELECT batch.FileIndex,
>>> batch.JobId, Path.PathId, Filename.FilenameId,batch.LStat, batch.MD5,
>>> batch.DeltaSeq FROM batch JOIN Path ON (batch.Path = Path.Path) JOIN
>>> Filename ON (batch.Name = Filename.Name): ERR=Deadlock found when trying to
>>> get lock; try restarting transaction
>>>
>>> The only thing I did yesterday was switch a bunch of backups to use
>>> Accurate backup and restart bacula-dir and bacula-sd after that.  However,
>>> the above problem also occurred on some hosts that was not set to use
>>> Accurate backup.  From the log, it seems like two jobs for this host was
>>> scheduled to run at 18:00 because the second job started and found a
>>> duplicate job (job 123984) and canceled the backup.  I know there were no
>>> jobs running before 18:00 

[Bacula-users] Waiting for appendable volume

[Gilberto Nunes]

Hi folks...

That's my first post in this mailing list and I think I have a very unrare
problem...

I have this in bacula-sd.conf

Device {
  Name = bacula
  Archive Device = /media/2c0bc75b-f54e-4f31-8d8c-5568edc0e765/bacula
  Media Type = File
  LabelMedia = yes
  Random Access = yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = yes
}

And this as Pool definition:

Pool {
  Name = Servidor
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 365 days
  Maximum Volumes = 100   # Limit number of Volumes in Pool
}

I have labeled the volume as Bkp-Servidor

I am also using the schedule and job above:

Schedule {
  Name = Servidor
  Run = Level=Full on 5 at 10:00
  Run = Level=Differential at 22:30
}
Job {
  Name = Servidor-Job
  Type = Backup
  Level = Full
  Client = Servidor
  FileSet = Servidor
  Schedule = Servidor
  Storage = File
  Pool = Servidor
  Messages = Standard
}

What I need is make Full and Differential backup use a single file disk...

Is that possible?

Why the file Bkp-Servidor (which is the file where baculs writes the
backups), changes to Full and ask me to appendable volume?

Perhaps I missconfigure somethink...

Can any one assist me??





-- 

Gilberto Ferreira
+55 (47) 9676-7530
Skype: gilberto.nunes36
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Backup finished, but "Fatal error: Network error with FD"/"Connection reset by peer"

[Josh Fisher]

On 8/6/2015 5:09 AM, Raimund Sacherer wrote:
> Hello Josh, Bacula-users,
>
>
>> I have seen this before as well, although not with FreeBSD. Bacula-dir
>> expects the TCP connection with the client to remain up throughout the
>> entire job. In my case I concluded that it was aggressive Windows power
>> management shutting down the Ethernet interface PHY. I continue to have
>> problems with Mac OSX clients power management shutting down the
>> wireless PHY, but have not had time to investigate. With Windows 7 it is
>> possible to disable the "Allow the computer to turn off this device to
>> save power" setting in the Power Management tab of the network adapter's
>> Properties. It depends on the NIC driver as to whether or not this is
>> needed. Some drivers report that they handle various sleep states when
>> they in fact do not, or at least they do not return to D0 state in a
>> timely manner.
> We do not backup client computers, only servers. I really sincerely hope that 
> a Windows Server does not do ethernet shutdowns or power management :-). But 
> I am a Unix guy ...

Yes, but server NICs supporting 802.3az are green independently of the 
OS, other than the NIC driver sending a Low Power Idle request. The 
firmware shuts down the PHY transmitter after a period of sending LPI 
symbols, and the Dir-FD connection is idle for an extended time. What do 
pre-2010 switches without 802.3az support do when those NICs shut down 
their transmitters? If they are green enough to shutdown the port then 
it may well look like a dropped connection on one end or the other.

>> And finally, many switches also have TCP timeout settings and/or EEE and
>> power management that could potentially not work correctly with either
>> the FreeBSD or the Windows network stacks.
> That sound's interesting, I saw a couple of posts talking about a keepalives, 
> I will configure our FD's, SD's an the director for a 300 seconds timeout and 
> we will see if we still get those errors.
>
> Maybe it has nothing to do with the switch to FreeBSD, because at nearly the 
> same time we migrated our servers from physical servers to VMWare, maybe it's 
> the virtual vmware switch which makes troubles.
>
> Well, in either case, i'l see how it goes with the keep alive configured,

Let us know, please. I had mixed results with the keep alive, while 
replacing an old switch seemed to magically fix Windows clients.

>
> Thank you
> Best
> Ray
>


--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] error connecting to database

[Alex Domoradov]

You could find out with which version of mysql client has been compiled
your bacula with the following command

# ldd /usr/sbin/bacula-dir | grep mysql
libmysqlclient.so.18 => /lib64/libmysqlclient.so.18
(0x7f07abe3d000)


# rpm -qf /lib64/libmysqlclient.so.18
Percona-Server-shared-55-5.5.43-rel37.2.el7.x86_64

On Thu, Aug 6, 2015 at 3:47 PM, Heitor Faria  wrote:

> Hey Heitor,
>
>  Actually to 1, no they are not. I have mariadb-5.5.41 on the bacula
> server (client side) and mariadb 10 on the db server. I might try upgrading
> the client on the bacula server tomorrow. I don't have SELinux enabled
> anywhere currently. I probably will enable that tho once I get everything
> working.
>
> Most important of all is to know what MySQL / MariaDB development
> libraries were used to build you Bacula binaries. You may want / need to
> update Bacula with binaries built from source:
> http://bacula.us/compilation/
>
> Regards,
> ===
> Heitor Medrado de Faria - LPIC-III | ITIL-F |  Bacula Systems Certified
> Administrator II
> Do you need Bacula training?
> https://www.udemy.com/bacula-backup-software/?couponCode=bacula-list
> +55 61 <%2B55%2061%202021-8260>8268-4220 <%2B55%2061%208268-4220>
> Site: http://bacula.us FB: heitor.faria
> 
> ===
>
>
> I'll try to update you guys tomorrow.
>
> Thanks for all your input!
>
> Tim
>
> On Wed, Aug 5, 2015 at 8:45 AM, Heitor Faria  wrote:
>
>>
>>> Em ter, 4 de ago de 2015 às 23:01, Tim Dunphy 
>>> escreveu:
>>>
 Hey Ana,
  Nice to hear from you!

 Tried that:


 Catalog {
   Name = MyCatalog
 # Uncomment the following line if you want the dbi driver
   #dbdriver = "dbi:mysql"; dbaddress = "db.example.com"; dbport = 3306
   dbname = "bacula";  dbuser = "admin_ssl"; dbpassword = "secret";
 dbaddress = "db.example.com"; dbport = 3306
 }

 And restarted. Same result unfortunately! :(

 [root@ops:~] #tail -f /var/log/bacula/bacula.log
 Database=bacula User=admin_ssl
 MySQL connect failed either server not running or your authorization is
 incorrect.
 05-Aug 01:59 bacula-dir ERROR TERMINATION
 Please correct configuration file: /etc/bacula/bacula-dir.conf
 05-Aug 01:59 bacula-dir JobId 0: Fatal error: Could not open Catalog
 "MyCatalog", database "bacula".
 05-Aug 01:59 bacula-dir JobId 0: Fatal error: mysql.c:210 Unable to
 connect to MySQL server.
 Database=bacula User=admin_ssl
 MySQL connect failed either server not running or your authorization is
 incorrect.
 05-Aug 01:59 bacula-dir ERROR TERMINATION
 Please correct configuration file: /etc/bacula/bacula-dir.conf

>>> 1. Is your remote MySQL server version the same installed in your Bacula
>> Server?
>> 2. From your Bacula server can you "telnet ip_address 3306" your MySQL
>> server?
>> 3. Do you have selinux or iptables enabled at MySQL Server? Someone
>> wrote that never had problems with selinux. Neither do I, since I always
>> disable it. =)
>>
>> Just ignore 2 and 3. I forgot you can connect with calling the client
>> directly.
>>
>>
 Any more ideas?

 Thanks,
 Tim



>
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>
>
>
>
> --
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
>
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] error connecting to database

[Heitor Faria]

> Hey Heitor,

> Actually to 1, no they are not. I have mariadb-5.5.41 on the bacula server
> (client side) and mariadb 10 on the db server. I might try upgrading the 
> client
> on the bacula server tomorrow. I don't have SELinux enabled anywhere 
> currently.
> I probably will enable that tho once I get everything working.

Most important of all is to know what MySQL / MariaDB development libraries 
were used to build you Bacula binaries. You may want / need to update Bacula 
with binaries built from source: http://bacula.us/compilation/ 

Regards, 
=== 
Heitor Medrado de Faria - LPIC-III | ITIL-F | Bacula Systems Certified 
Administrator II 
Do you need Bacula training? 
https://www.udemy.com/bacula-backup-software/?couponCode=bacula-list 
+55 61 8268-4220 
Site: http://bacula.us FB: heitor.faria 
=== 

> I'll try to update you guys tomorrow.

> Thanks for all your input!

> Tim

> On Wed, Aug 5, 2015 at 8:45 AM, Heitor Faria < hei...@bacula.com.br > wrote:

> Em ter, 4 de ago de 2015 às 23:01, Tim Dunphy < bluethu...@gmail.com > 
> escreveu:

>> Hey Ana,
>> Nice to hear from you!

>> Tried that:

>> Catalog {
>> Name = MyCatalog
>> # Uncomment the following line if you want the dbi driver
>> #dbdriver = "dbi:mysql"; dbaddress = " db.example.com "; dbport = 3306
>> dbname = "bacula"; dbuser = "admin_ssl"; dbpassword = "secret"; 
>> dbaddress = "
>> db.example.com "; dbport = 3306
>> }

>> And restarted. Same result unfortunately! :(

>> [root@ops:~] #tail -f /var/log/bacula/bacula.log
>> Database=bacula User=admin_ssl
>> MySQL connect failed either server not running or your authorization is
>> incorrect.
>> 05-Aug 01:59 bacula-dir ERROR TERMINATION
>> Please correct configuration file: /etc/bacula/bacula-dir.conf
>> 05-Aug 01:59 bacula-dir JobId 0: Fatal error: Could not open Catalog
>> "MyCatalog", database "bacula".
>> 05-Aug 01:59 bacula-dir JobId 0: Fatal error: mysql.c:210 Unable to 
>> connect to
>> MySQL server.
>> Database=bacula User=admin_ssl
>> MySQL connect failed either server not running or your authorization is
>> incorrect.
>> 05-Aug 01:59 bacula-dir ERROR TERMINATION
>> Please correct configuration file: /etc/bacula/bacula-dir.conf

>>> 1. Is your remote MySQL server version the same installed in your Bacula 
>>> Server?
>>> 2. From your Bacula server can you "telnet ip_address 3306" your MySQL 
>>> server?
>>> 3. Do you have selinux or iptables enabled at MySQL Server? Someone wrote 
>>> that
>>> never had problems with selinux. Neither do I, since I always disable it. =)

>> Just ignore 2 and 3. I forgot you can connect with calling the client 
>> directly.

>> Any more ideas?

>> Thanks,
>> Tim

> --
> GPG me!!

> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Deadlock error

[Kern Sibbald]

  
  
On 06.08.2015 10:15, Craig Shiroma
  wrote:


  Hello again,


I just thought I'd update this post with more information
  in hopes of getting some explanation for the deadlocks.  


I ran with Accurate backup on our test VMs (RHEL) for a
  couple of days and got the same errors on some VMs that were
  running accurate and some that were not.  These hosts were
  running concurrently.  I would say 90% of the hosts that were
  configured to use Accurate finished successfully.  However,
  there were a few that failed with the deadlock error -- some
  that were configured to use accurate and some that were not
  configured to use accurate.  Also, on all of these, a second
  job started for each of the affected hosts right after Bacula
  detected the deadlock even though it said a reschedule would
  happen 3600 seconds later (the 3600 seconds is correct).


Tonight, I disabled accurate on all hosts and the deadlocks
  did not happen.  No errors were detected and all the backups
  finished successfully.


Some questions...
1.  Can I back up multiple hosts concurrently with some
  hosts configured to use accurate and some configured not to
  use accurate?  Or, is it an all or none thing, meaning all
  hosts that run concurrently must either be using accurate
  backup or not using accurate backup (cannot mix the two)?


2. It seems like the hosts that get out of the starting
  gate first are the ones affected.  I am configured to run 50
  jobs concurrently.  Again, no problems with accurate turned
  off on all hosts for months now.


3. Why is Bacula spinning off a new job right away after it
  detects the deadlock for each affected job instead of waiting
  until the rescheduled job runs?  I verified that there were no
  duplicate jobs in the queue before the backups started
  running, no jobs were running before the start of the backups,
  and I did not start any of these backups manually to cause a
  second job to appear.
  


Bacula is not aware of any SQL internal deadlocks.


  


From the INNODB Monitor output:



  TRANSACTION:
  TRANSACTION 208788977, ACTIVE 1 sec setting auto-inc lock
  mysql tables in use 4, locked 4
  9 lock struct(s), heap size 1184, 5 row lock(s)
  MySQL thread id 50808, OS thread handle 0x7f8f2c3b4700,
query id 29558637  192.168.10.99 bacula Sending
data
  INSERT INTO File (FileIndex, JobId, PathId, FilenameId,
LStat, MD5, DeltaSeq) SELECT batch.FileIndex, batch.JobId,
Path.PathId, Filename.FilenameId,batch.LStat, batch.MD5,
batch.DeltaSeq FROM batch JOIN Path ON (batch.Path =
Path.Path) JOIN Filename ON (batch.Name = Filename.Name)
  WAITING FOR THIS LOCK TO BE GRANTED:
  TABLE LOCK table `bacula`.`File` trx id 208788977 lock
mode AUTO-INC waiting
  WE ROLL BACK TRANSACTION (2)



I am running Bacula 7.0.5 on RHEL 6.6 x64 with Director,
  Storage and Catalog running on separate RHEL 6.6 hosts.  Our
  clients are RHEL 6's, 5's and Windows Servers 2008 and 2012R2.


Any help would be much appreciated.


Warmest regards,
-craig
  
  
On Tue, Aug 4, 2015 at 1:56 PM, Craig
  Shiroma 
  wrote:
  
BTW, I suppose there could've been two jobs
  for the host(s) in scheduling queue.  If this was the
  case, is there a way to find out after the fact?  If this
  did actually happen, what could cause duplicate jobs to be
  scheduled on the same day at the same time?  I know no one
  manually ran the jobs in question.  Again, this only was a
  problem for a few of the jobs that ran last night, not all
  of them and some to do accurate backup and some not.
  
  
  Regards,
  -craig


  

  On Tue, Aug 4, 2015 at 9:27
AM, Craig Shiroma 
wrote:

  Hello,


I had a few backups fail last night with
  the following error:


2015-08-03

Re: [Bacula-users] Bacula BLOCKED waiting for mount

[Kern Sibbald]

  
  
On 04.08.2015 18:41, Michael Schwager
  wrote:


  
Actually,
  the restore jobs were in the 900-range. 687 was indeed the
  backup job, as I can see that the numbers before and closely
  following it were all backup jobs in the same data range (I
  didn't do my restore until months later).
  

It
  was that restore process that taught me how to manage my
  Bacula Directory so as to avoid bscan's in the future. But
  it's good to know that I can restore without restoring the
  info in the catalog.
  

(also
  I learned that Backups
are useless. It's the restore I care about. At work we
  no longer talk about our "backup infrastructure", we talk
  about our "restore infrastructure").

  


Most people take a very long time to learn what you arew writing
above.  However, without a backup, you cannot do a restore. 
Consequently, I prefer to say that  "Restore is the most important
part of backup/restore"

Best regards,
Kern

  

  

  

  

  

  - Mike Schwager

  Linux
Network Engineer, Mocho Trading LLC
  
   
  312-646-4783 Phone    312-637-0011 Cell   
  312-957-9804 Fax
  
  

  
  

  

  

  


On Tue, Aug 4, 2015 at 7:30 AM, Ana
  Emília M. Arruda 
  wrote:
  

  Hello Michael,
  
  
  When you use bscan
to restore jobs and files information from a volume into
catalog, a new jobid is created for the original jobid
(the jobid that used the volume for its backup). This is
why you have the JobId 687 and not the original one.
This is the JobId that you will find in your Job table.
And this is the jobId that Bacula will tie to your media
in JobMedia table.
  
  
  If the bscan
operation is usual for you, I would recommend you to
have an Admin Job doing prune operations of your jobs
scheduled in an adequate date/time for your environment.
  
  
  Also, I would
remember you that you can use bls/bextrac/bscan to
restore the directories/files without restoring the
volume/job info into catalog.
  
  
  Best regards,
  Ana 
  

  
On Mon, Aug 3, 2015 at 7:15 PM,
  Michael Schwager 
  wrote:

  
  

  

  
  On Sun, Aug 2,
2015 at 5:43 PM, Michael Schwager 
wrote:

  Starngely,
we have another tape that SHOULD be
available. Tapes in this pool are
set to recycle after 90 days, and 90
days ago from today is May 4th.
​
  We have a tape that was last
  written to on March 28. So why
  would Bacula block on the
  unavailable tape?
  

  
  

​
  To answer my own question (with help from
  the other denizens of the Bacula-cave): ​
  

A
  couple of months ago I needed to do a
  restore, but I had set some of the
  retention times too shor

Re: [Bacula-users] Backup finished, but "Fatal error: Network error with FD"/"Connection reset by peer"

[Raimund Sacherer]

Hello Josh, Bacula-users, 


> I have seen this before as well, although not with FreeBSD. Bacula-dir
> expects the TCP connection with the client to remain up throughout the
> entire job. In my case I concluded that it was aggressive Windows power
> management shutting down the Ethernet interface PHY. I continue to have
> problems with Mac OSX clients power management shutting down the
> wireless PHY, but have not had time to investigate. With Windows 7 it is
> possible to disable the "Allow the computer to turn off this device to
> save power" setting in the Power Management tab of the network adapter's
> Properties. It depends on the NIC driver as to whether or not this is
> needed. Some drivers report that they handle various sleep states when
> they in fact do not, or at least they do not return to D0 state in a
> timely manner.
We do not backup client computers, only servers. I really sincerely hope that a 
Windows Server does not do ethernet shutdowns or power management :-). But I am 
a Unix guy ...


> And finally, many switches also have TCP timeout settings and/or EEE and
> power management that could potentially not work correctly with either
> the FreeBSD or the Windows network stacks.
That sound's interesting, I saw a couple of posts talking about a keepalives, I 
will configure our FD's, SD's an the director for a 300 seconds timeout and we 
will see if we still get those errors. 

Maybe it has nothing to do with the switch to FreeBSD, because at nearly the 
same time we migrated our servers from physical servers to VMWare, maybe it's 
the virtual vmware switch which makes troubles. 

Well, in either case, i'l see how it goes with the keep alive configured, 

Thank you
Best
Ray


--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Deadlock error

[Craig Shiroma]

One thing I missed mentioning was on the second night accurate backup was
used, the director died about 45 minutes into the backup with the following
error.  This did not happen on the first run with accurate enabled.

Aug  4 19:02:27  bacula-dir: Bacula interrupted by signal 11:
Segmentation violation

Also, as many of you know, since a duplicate job was spinned off for each
job with a deadlock, the backup was cancelled.

On Wed, Aug 5, 2015 at 10:15 PM, Craig Shiroma 
wrote:

> Hello again,
>
> I just thought I'd update this post with more information in hopes of
> getting some explanation for the deadlocks.
>
> I ran with Accurate backup on our test VMs (RHEL) for a couple of days and
> got the same errors on some VMs that were running accurate and some that
> were not.  These hosts were running concurrently.  I would say 90% of the
> hosts that were configured to use Accurate finished successfully.  However,
> there were a few that failed with the deadlock error -- some that were
> configured to use accurate and some that were not configured to use
> accurate.  Also, on all of these, a second job started for each of the
> affected hosts right after Bacula detected the deadlock even though it said
> a reschedule would happen 3600 seconds later (the 3600 seconds is correct).
>
> Tonight, I disabled accurate on all hosts and the deadlocks did not
> happen.  No errors were detected and all the backups finished successfully.
>
> Some questions...
> 1.  Can I back up multiple hosts concurrently with some hosts configured
> to use accurate and some configured not to use accurate?  Or, is it an all
> or none thing, meaning all hosts that run concurrently must either be using
> accurate backup or not using accurate backup (cannot mix the two)?
>
> 2. It seems like the hosts that get out of the starting gate first are the
> ones affected.  I am configured to run 50 jobs concurrently.  Again, no
> problems with accurate turned off on all hosts for months now.
>
> 3. Why is Bacula spinning off a new job right away after it detects the
> deadlock for each affected job instead of waiting until the rescheduled job
> runs?  I verified that there were no duplicate jobs in the queue before the
> backups started running, no jobs were running before the start of the
> backups, and I did not start any of these backups manually to cause a
> second job to appear.
>
> From the INNODB Monitor output:
>
> TRANSACTION:
> TRANSACTION 208788977, ACTIVE 1 sec setting auto-inc lock
> mysql tables in use 4, locked 4
> 9 lock struct(s), heap size 1184, 5 row lock(s)
> MySQL thread id 50808, OS thread handle 0x7f8f2c3b4700, query id 29558637
>  192.168.10.99 bacula Sending data
> INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5,
> DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId,
> Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN
> Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name =
> Filename.Name)
> WAITING FOR THIS LOCK TO BE GRANTED:
> TABLE LOCK table `bacula`.`File` trx id 208788977 lock mode AUTO-INC
> waiting
> WE ROLL BACK TRANSACTION (2)
>
> I am running Bacula 7.0.5 on RHEL 6.6 x64 with Director, Storage and
> Catalog running on separate RHEL 6.6 hosts.  Our clients are RHEL 6's, 5's
> and Windows Servers 2008 and 2012R2.
>
> Any help would be much appreciated.
>
> Warmest regards,
> -craig
>
> On Tue, Aug 4, 2015 at 1:56 PM, Craig Shiroma 
> wrote:
>
>> BTW, I suppose there could've been two jobs for the host(s) in scheduling
>> queue.  If this was the case, is there a way to find out after the fact?
>> If this did actually happen, what could cause duplicate jobs to be
>> scheduled on the same day at the same time?  I know no one manually ran the
>> jobs in question.  Again, this only was a problem for a few of the jobs
>> that ran last night, not all of them and some to do accurate backup and
>> some not.
>>
>> Regards,
>> -craig
>>
>> On Tue, Aug 4, 2015 at 9:27 AM, Craig Shiroma 
>> wrote:
>>
>>> Hello,
>>>
>>> I had a few backups fail last night with the following error:
>>>
>>> 2015-08-03 18:02:46bacula-dir JobId 123984: b INTO File (FileIndex,
>>> JobId, PathId, FilenameId, LStat, MD5, DeltaSeq) SELECT batch.FileIndex,
>>> batch.JobId, Path.PathId, Filename.FilenameId,batch.LStat, batch.MD5,
>>> batch.DeltaSeq FROM batch JOIN Path ON (batch.Path = Path.Path) JOIN
>>> Filename ON (batch.Name = Filename.Name): ERR=Deadlock found when trying to
>>> get lock; try restarting transaction
>>>
>>> The only thing I did yesterday was switch a bunch of backups to use
>>> Accurate backup and restart bacula-dir and bacula-sd after that.  However,
>>> the above problem also occurred on some hosts that was not set to use
>>> Accurate backup.  From the log, it seems like two jobs for this host was
>>> scheduled to run at 18:00 because the second job started and found a
>>> duplicate job (job 123984) and canceled the backup.  I know there were no
>>> jobs running before

Re: [Bacula-users] Deadlock error

[Craig Shiroma]

Hello again,

I just thought I'd update this post with more information in hopes of
getting some explanation for the deadlocks.

I ran with Accurate backup on our test VMs (RHEL) for a couple of days and
got the same errors on some VMs that were running accurate and some that
were not.  These hosts were running concurrently.  I would say 90% of the
hosts that were configured to use Accurate finished successfully.  However,
there were a few that failed with the deadlock error -- some that were
configured to use accurate and some that were not configured to use
accurate.  Also, on all of these, a second job started for each of the
affected hosts right after Bacula detected the deadlock even though it said
a reschedule would happen 3600 seconds later (the 3600 seconds is correct).

Tonight, I disabled accurate on all hosts and the deadlocks did not
happen.  No errors were detected and all the backups finished successfully.

Some questions...
1.  Can I back up multiple hosts concurrently with some hosts configured to
use accurate and some configured not to use accurate?  Or, is it an all or
none thing, meaning all hosts that run concurrently must either be using
accurate backup or not using accurate backup (cannot mix the two)?

2. It seems like the hosts that get out of the starting gate first are the
ones affected.  I am configured to run 50 jobs concurrently.  Again, no
problems with accurate turned off on all hosts for months now.

3. Why is Bacula spinning off a new job right away after it detects the
deadlock for each affected job instead of waiting until the rescheduled job
runs?  I verified that there were no duplicate jobs in the queue before the
backups started running, no jobs were running before the start of the
backups, and I did not start any of these backups manually to cause a
second job to appear.

>From the INNODB Monitor output:

TRANSACTION:
TRANSACTION 208788977, ACTIVE 1 sec setting auto-inc lock
mysql tables in use 4, locked 4
9 lock struct(s), heap size 1184, 5 row lock(s)
MySQL thread id 50808, OS thread handle 0x7f8f2c3b4700, query id 29558637
 192.168.10.99 bacula Sending data
INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5,
DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId,
Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN
Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name =
Filename.Name)
WAITING FOR THIS LOCK TO BE GRANTED:
TABLE LOCK table `bacula`.`File` trx id 208788977 lock mode AUTO-INC waiting
WE ROLL BACK TRANSACTION (2)

I am running Bacula 7.0.5 on RHEL 6.6 x64 with Director, Storage and
Catalog running on separate RHEL 6.6 hosts.  Our clients are RHEL 6's, 5's
and Windows Servers 2008 and 2012R2.

Any help would be much appreciated.

Warmest regards,
-craig

On Tue, Aug 4, 2015 at 1:56 PM, Craig Shiroma 
wrote:

> BTW, I suppose there could've been two jobs for the host(s) in scheduling
> queue.  If this was the case, is there a way to find out after the fact?
> If this did actually happen, what could cause duplicate jobs to be
> scheduled on the same day at the same time?  I know no one manually ran the
> jobs in question.  Again, this only was a problem for a few of the jobs
> that ran last night, not all of them and some to do accurate backup and
> some not.
>
> Regards,
> -craig
>
> On Tue, Aug 4, 2015 at 9:27 AM, Craig Shiroma 
> wrote:
>
>> Hello,
>>
>> I had a few backups fail last night with the following error:
>>
>> 2015-08-03 18:02:46bacula-dir JobId 123984: b INTO File (FileIndex,
>> JobId, PathId, FilenameId, LStat, MD5, DeltaSeq) SELECT batch.FileIndex,
>> batch.JobId, Path.PathId, Filename.FilenameId,batch.LStat, batch.MD5,
>> batch.DeltaSeq FROM batch JOIN Path ON (batch.Path = Path.Path) JOIN
>> Filename ON (batch.Name = Filename.Name): ERR=Deadlock found when trying to
>> get lock; try restarting transaction
>>
>> The only thing I did yesterday was switch a bunch of backups to use
>> Accurate backup and restart bacula-dir and bacula-sd after that.  However,
>> the above problem also occurred on some hosts that was not set to use
>> Accurate backup.  From the log, it seems like two jobs for this host was
>> scheduled to run at 18:00 because the second job started and found a
>> duplicate job (job 123984) and canceled the backup.  I know there were no
>> jobs running before 18:00 so 123984 was not an old job still running.  Same
>> with the other jobs that were canceled because of the above situation.
>>
>> Anyway, does anyone have an idea what would cause this, especially how
>> the second job got shot into the system.  After the deadlock error, Bacula
>> said it would reschedule the job.  However the second job started right
>> after the deadlock error instead of one hour later which makes me think
>> that there were two jobs for this host scheduled to run at 18:00.
>>
>> Thank you in advance,
>> -craig
>>
>
>
--
__