Re: [Bacula-users] Volume max jobs exceeded on drive - bacula 5.2.12-3.1

2017-10-08 Thread Gi Dot 
Think I'll go ahead with setting up a director on the administrative
district. Thanks a lot for your tips and advice. You have been a great help.

On Mon, Oct 9, 2017 at 12:37 PM, Phil Stracchino 
wrote:

> On 10/09/17 00:13, Gi Dot wrote:
> > Well I have the freedom to test out any kind of setup to make this work,
> > so yes it is possible. But we're talking about a hundred of districts,
> > do you think that this approach could be an overkill? Sorry, I should've
> > mentioned that there are more than 7 clients involved for the real
> > implementation.
>
> Yeah, i would say that going with the multiple-SDs solution probably
> isn't going to scale to hundreds of districts.  Performance would be
> abysmal.  But then, neither will the shared-SD approach.
>
> > Thanks for the explanation on bacula with cluster. I've never setup
> > bacula with cluster before, it's something to look at.
>
> If you are going to try running Bacula against a Galera synchronous
> cluster, you must build the Director with attribute spooling disabled.
> MySQL attribute spooling as currently implemented does not work on
> Galera clusters and will cause jobs to fail.  Bacula on Galera *without*
> attribute spooling works just fine.
>
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   ph...@caerllewys.net
>   p...@co.ordinate.org
>   Landline: +1.603.293.8485
>   Mobile:   +1.603.998.6958
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Volume max jobs exceeded on drive - bacula 5.2.12-3.1

2017-10-08 Thread Gi Dot 
Well I have the freedom to test out any kind of setup to make this work, so
yes it is possible. But we're talking about a hundred of districts, do you
think that this approach could be an overkill? Sorry, I should've mentioned
that there are more than 7 clients involved for the real implementation.

Thanks for the explanation on bacula with cluster. I've never setup bacula
with cluster before, it's something to look at.


On Mon, Oct 9, 2017 at 11:54 AM, Phil Stracchino 
wrote:

> On 10/08/17 23:41, Gi Dot wrote:
> > Ah, I didn't think of that. Make sense. The 7 clients are from 7
> > districts across the state. Currently each client backs up to a tape
> > locally. Recently a request was made to make another copy of backup to
> > be sent to a NAS at the administrative district. My first thought was
> > just to treat it as a secondary storage for bacula which I've done
> > before, but with a single director.
>
>
> Ah, I understand more about why it's done that way now.
>
> It would probably perform poorly, but ...  is it possible to give each
> Director its OWN storage daemon on the FreeNAS, all running on different
> ports?
>
>
>
> > In an event that one of the server goes down, it takes the records of
> > its backups with it <-- Isn't this is the case even in a normal setup?
> > Unless you have a backup of the bacula catalog resides somewhere else,
> > then you wouldn't need to bscan the backup media for restoration.
>
>
> There are multiple ways around that.  But at the first level, if you
> have a separate master Director, and it goes down, none of the client
> servers are down, so the urgency of getting the master back up is less.
> And if one of the client servers goes down, then the master is still
> online and can begin restores immediately.
>
> I have a single Director, but my Catalog database is on a three-node
> synchronous cluster.  If my master Director were to go down, I have two
> other working nodes in the cluster, and I could fairly quickly install a
> substitute Director on another host to restore it.
>
> Of course if something takes down the client servers AND the master
> director, then you're still looking at bscans before you can get
> anything back up.  If all three nodes of my cluster were down, I would
> have to do the same thing.  I might conceivably have to do a minimal
> reinstall on one machine, enough to get a minimal Bacula install up,
> bscan media, and restore the other two servers, then use those to
> properly restore the first.  That is always the worst-case scenario,
> restoring everything from bare metal.
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   ph...@caerllewys.net
>   p...@co.ordinate.org
>   Landline: +1.603.293.8485
>   Mobile:   +1.603.998.6958
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Volume max jobs exceeded on drive - bacula 5.2.12-3.1

2017-10-08 Thread Phil Stracchino 
On 10/08/17 23:41, Gi Dot wrote:
> Ah, I didn't think of that. Make sense. The 7 clients are from 7
> districts across the state. Currently each client backs up to a tape
> locally. Recently a request was made to make another copy of backup to
> be sent to a NAS at the administrative district. My first thought was
> just to treat it as a secondary storage for bacula which I've done
> before, but with a single director.


Ah, I understand more about why it's done that way now.

It would probably perform poorly, but ...  is it possible to give each
Director its OWN storage daemon on the FreeNAS, all running on different
ports?


> In an event that one of the server goes down, it takes the records of
> its backups with it <-- Isn't this is the case even in a normal setup?
> Unless you have a backup of the bacula catalog resides somewhere else,
> then you wouldn't need to bscan the backup media for restoration.


There are multiple ways around that.  But at the first level, if you
have a separate master Director, and it goes down, none of the client
servers are down, so the urgency of getting the master back up is less.
And if one of the client servers goes down, then the master is still
online and can begin restores immediately.

I have a single Director, but my Catalog database is on a three-node
synchronous cluster.  If my master Director were to go down, I have two
other working nodes in the cluster, and I could fairly quickly install a
substitute Director on another host to restore it.

Of course if something takes down the client servers AND the master
director, then you're still looking at bscans before you can get
anything back up.  If all three nodes of my cluster were down, I would
have to do the same thing.  I might conceivably have to do a minimal
reinstall on one machine, enough to get a minimal Bacula install up,
bscan media, and restore the other two servers, then use those to
properly restore the first.  That is always the worst-case scenario,
restoring everything from bare metal.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Volume max jobs exceeded on drive - bacula 5.2.12-3.1

2017-10-08 Thread Gi Dot 
Ah, I didn't think of that. Make sense. The 7 clients are from 7 districts
across the state. Currently each client backs up to a tape locally.
Recently a request was made to make another copy of backup to be sent to a
NAS at the administrative district. My first thought was just to treat it
as a secondary storage for bacula which I've done before, but with a single
director.

In an event that one of the server goes down, it takes the records of its
backups with it <-- Isn't this is the case even in a normal setup? Unless
you have a backup of the bacula catalog resides somewhere else, then you
wouldn't need to bscan the backup media for restoration.

On Mon, Oct 9, 2017 at 11:07 AM, Phil Stracchino 
wrote:

> On 10/08/17 22:39, Gi Dot wrote:
> > Your understanding is correct. I am not aware that this setup is not
> > supported/possible. I supposed the only way would be to configure a
> > bacula director to handle the jobs that go to FreeNAS.
> >
> > Is there a bacula-dir/fd plugin for FreeNAS? I have only found bacula-sd.
>
> Well, the problem there, I think, is the one Storage shared between
> seven Directors.  That's the part that's not supported.  Inevitably
> you're going to have multiple Directors trying to ask the same Storage
> to mount different volumes at the same time.
>
> Is there a reason you're running a Director on each client instead of
> just a file daemon and a single coordinating Director?  With the
> configuration you have, if one of those seven servers goes down, it
> takes the records of its own backups with it.  That means you've got to
> bscan backup media before you can do any disaster-recovery restores.
>
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   ph...@caerllewys.net
>   p...@co.ordinate.org
>   Landline: +1.603.293.8485
>   Mobile:   +1.603.998.6958
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Volume max jobs exceeded on drive - bacula 5.2.12-3.1

2017-10-08 Thread Phil Stracchino 
On 10/08/17 22:39, Gi Dot wrote:
> Your understanding is correct. I am not aware that this setup is not
> supported/possible. I supposed the only way would be to configure a
> bacula director to handle the jobs that go to FreeNAS.
> 
> Is there a bacula-dir/fd plugin for FreeNAS? I have only found bacula-sd.

Well, the problem there, I think, is the one Storage shared between
seven Directors.  That's the part that's not supported.  Inevitably
you're going to have multiple Directors trying to ask the same Storage
to mount different volumes at the same time.

Is there a reason you're running a Director on each client instead of
just a file daemon and a single coordinating Director?  With the
configuration you have, if one of those seven servers goes down, it
takes the records of its own backups with it.  That means you've got to
bscan backup media before you can do any disaster-recovery restores.



-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Volume max jobs exceeded on drive - bacula 5.2.12-3.1

2017-10-08 Thread Gi Dot 
Your understanding is correct. I am not aware that this setup is not
supported/possible. I supposed the only way would be to configure a bacula
director to handle the jobs that go to FreeNAS.

Is there a bacula-dir/fd plugin for FreeNAS? I have only found bacula-sd.

Thanks.

On Mon, Oct 9, 2017 at 10:28 AM, Phil Stracchino 
wrote:

> On 10/08/17 22:17, Gi Dot wrote:
> > Hi,
> >
> > Does bacula storage has a limit to accept jobs at one time?
> >
> > I have configured 7 servers to backup on a FreeNAS. Each server runs
> > bacula-dir pointing to its own bacula database locally. Each server has
> > 2 storage configured, local and FreeNAS which is geographically remote.
> > Jobs were configured to run at the same time on all servers.
> >
> > Out of these 7 servers, only 2 backup jobs were successful on the first
> > night of running according to schedule. When I checked the FreeNAS
> > storage status, it says:
> ...
>
>
> OK, on a second re-reading:  If I understand this correctly, you have
> seven Bacula directors, each with its own Catalog database and with
> itself as its only Client, all pointing at the same Storage daemon
> shared between all seven Directors.
>
>
> If that is a correct understanding, it is NOT a supported configuration,
> and I'm highly surprised it's working at all.
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   ph...@caerllewys.net
>   p...@co.ordinate.org
>   Landline: +1.603.293.8485
>   Mobile:   +1.603.998.6958
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Volume max jobs exceeded on drive - bacula 5.2.12-3.1

2017-10-08 Thread Phil Stracchino 
On 10/08/17 22:17, Gi Dot wrote:
> Hi,
> 
> Does bacula storage has a limit to accept jobs at one time?
> 
> I have configured 7 servers to backup on a FreeNAS. Each server runs
> bacula-dir pointing to its own bacula database locally. Each server has
> 2 storage configured, local and FreeNAS which is geographically remote.
> Jobs were configured to run at the same time on all servers.
> 
> Out of these 7 servers, only 2 backup jobs were successful on the first
> night of running according to schedule. When I checked the FreeNAS
> storage status, it says:
...


OK, on a second re-reading:  If I understand this correctly, you have
seven Bacula directors, each with its own Catalog database and with
itself as its only Client, all pointing at the same Storage daemon
shared between all seven Directors.


If that is a correct understanding, it is NOT a supported configuration,
and I'm highly surprised it's working at all.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Volume max jobs exceeded on drive - bacula 5.2.12-3.1

2017-10-08 Thread Gi Dot 
Hi,

Does bacula storage has a limit to accept jobs at one time?

I have configured 7 servers to backup on a FreeNAS. Each server runs
bacula-dir pointing to its own bacula database locally. Each server has 2
storage configured, local and FreeNAS which is geographically remote. Jobs
were configured to run at the same time on all servers.

Out of these 7 servers, only 2 backup jobs were successful on the first
night of running according to schedule. When I checked the FreeNAS storage
status, it says:

Jobs waiting to reserve a drive:
   3610 JobId=759 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=821 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=825 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=799 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=768 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=816 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=802 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=772 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=1490 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=270 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=828 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).
   3610 JobId=832 Volume max jobs exceeded on drive "FreeNAS" (/mnt/files).

For the second run, I scheduled the job to run 15 minutes apart among the
servers. Only 3 jobs were successful while the rest were still in Running
mode.

Relevant configurations below.

job:
Job {
  Name = "TestBackup"
  JobDefs = "DefaultJob"
  Pool = client2
  FileSet = "Test Set"
  Storage = FreeNAS
  Schedule = "clientBackup"
  Priority = 11
  Enabled = no
}

pool:
Pool {
  Name = client2
  Pool Type = Backup
  Recycle = yes
  Recycle Oldest Volume = Yes
  AutoPrune = yes
  Volume Retention = 5 days
  Maximum Volume Jobs = 1
  Maximum Volumes = 7
  Label Format = "${Client:s/-fd//t}-${Level}-${NumVols:p/3/0/r}"
}

bacula-sd.conf:
Device {
  Name = FreeNAS
  Media Type = File
  Archive Device = /mnt/files
  LabelMedia = yes;
  Random Access = Yes;
  AutomaticMount = yes;
  RemovableMedia = no;
  AlwaysOpen = no;
  Maximum Network Buffer Size = 65536
}

Have I misconfigured the pool? Would appreciate any help.

Thanks.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Not Receiving Messages for Read-Only file system

2017-10-08 Thread Bill Arlofski 
Jim, I think I have (accidentally) reproduced the same issue but in a
different way.

http://bugs.bacula.org/view.php?id=2332

In my case, I had a zero byte file volume. Bacula noticed that there was no
label on it, and kept looping and reloading this volume and did not abandon it
until I marked it "Disabled"

Best regards,

Bill


-- 
Bill Arlofski
http://www.revpol.com/bacula
-- Not responsible for anything below this line --

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] A device per pool

2017-10-08 Thread Kern Sibbald 

  
  
The idea of putting the PoolName and
  MediaType into the volume label at the time the code was written
  seemed reasonable, but it makes changing the volume rather
  inflexible.  As a consequence, though those fields are written, no
  one ever actually uses them.
  
  Best regards,
  Kern
  
  On 08/09/2017 16:22, Dan Langille wrote:


  
  

  On Sep 8, 2017, at 6:47 AM, Martin Simmons  wrote:
  
  
That's a useful article.

  



Thank you.


  
There is a minor caveat when copying volume
  files: the volume label contains
  the PoolName and MediaType.  This is minor because it only
  matters if you need
  to bscan the volume before it is recycled.

  



I hope nobody ever has to use bscan. (using bscan means
  something has gone wrong;
this is not a criticism of the tool).


I don't think we can easily change that label.  For those
  reading, here is an older document
I found via Google. http://www.bacula.org/2.4.x-manuals/en/developers/Storage_Media_Output_Format.html


As you say, it won't be relevant until bscan time.



  -- 
Dan Langille - BSDCan / PGCon
d...@langille.org

  






  
__Martin
  
  
  
  

  

  On Thu, 7 Sep
2017 22:04:35 -0400, Dan Langille said:
  

  


I have recently moved to one Storage device per pool.

Why?

So each pool is on a different ZFS dataset.

Once I figured out the process for the first one, the
rest was mostly waiting for data to be copied.

I documented it here: https://dan.langille.org/2017/09/06/bacula-ran-out-of-space-moved-some-volumes-to-another-zpool/


The final zfs layout looks like this: https://gist.github.com/dlangille/78811ba69b2a14f6c1e63c2b4bafc99e


Here is the rough overview:


# in bacula-dir.conf

Pool {
 Name = DiffFile
 Pool Type    = Backup
 Recycle  = yes
 AutoPrune    = yes
 Volume Retention = 6 weeks
 Storage  = bacula-sd-01-DiffFile
 Next Pool    = Differentials

 Maximum Volume Bytes = 10G
 Maximum Volume Jobs  = 1
 Maximum Volumes  = 300

 LabelFormat = "DiffAuto-"
}

# Definiton of file storage device
Storage {
 Name   = bacula-sd-01-DiffFile
 Address    = bacula-sd-01.int.unixathome.org
   # N.B. Use a fully qualified name here
 SDPort = 9103
 Password   = "[redacted]"

 AutoChanger = Yes

 Device = VirtualDisk-DiffFile
 Media Type = DiffFile

 Maximum Concurrent Jobs = 27
}

# in bacula-sd.conf

# why do I do an autochanger? Some AutoChangers have
multiple drives.
# It's easier to make them all similar.
# an example with 10 virtual drives is here: https://gist.github.com/dlangille/ba787f6e6d838d2fbcc6ad9aad9afd55


Autochanger {
 Name = VirtualDisk-DiffFile

 Changer Device  = /dev/null
 Changer Command = /dev/null

 Device  = vDrive-DiffFile-0
}

Device {
 Name   = vDrive-DiffFile-0
 Media Type = DiffFile
 

Re: [Bacula-users] qt-console Compile Error

2017-10-08 Thread Kern Sibbald 

  
  
Hello,
  
  It appears that you are trying to build bat without building
  "bacula" (or at least the Bacula libraries).  Bat depends on
  Bacula libraries,
  so the recommended way to build it is with:
  
  ./configure 
  make
  cd src/qt-console
  make
  
  If you really know Bacula and what you are doing with it, this
  could possibly be shortened a bit, but the above is the simplest.
  
  Best regards,
  Kern
  
  On 15/09/2017 10:34, Can Şirin wrote:


  
  Hi,

I have problem with compiling qt-console v9.0.4 on RHEL 7. I
have previously succesfully compiled it in v7.2.0. There was no
problem with that version. I am trying to upgrade from v7.2.0 to
v9.0.4. For this component I have installed qt qt-devel packages
from the RHEL base repo with the version 4.8.5-13.

You can see the error output of compiling progress and the
configure parameter in the attachment.

Is there anyone encountering such kind of problem?
  
  
  
  --
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
  
  
  
  ___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users




  


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users