[Bacula-users] best practices in separating servers support Bacula processes

[Robert M. Candey]

I've been using Bacula to back up many servers and desktops to a tape library 
since early on, but always had one server running all of the Bacula processes 
except for the individual file servers.


I'm setting up a new tape library and have new data servers, so I'm wondering if 
there is a more efficient architecture for backing up 1PB, mostly stored on one 
server and NFS-mounted to the other data servers.


Does it make sense to run the PostgreSQL database server and storage servers on 
their own servers dedicated to Bacula?


Is there value in running the Director on one or the other?

Should I continue to run the storage daemon on the server that hosts the large 
data?

I'm thinking that the NFS server might be more efficient if run on its own, and 
transfer its data over the network (100GbE) to the Bacula storage server 
attached to the tape library.  And perhaps PostgreSQL could have dedicated 
memory and CPU. I don't know what if anything is slowing down our backups.  Full 
backups take 4-6 weeks for 500 TB now.


Ideas?  Thoughts?  Suggestions?

Thank you

Robert Candey


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] 350TB backup

[Robert M. Candey]

  
  
We have 80TB of heliophysics data that we mirror
  with rsync daily to another storage server for fast switchover. We
  use Bacula to make quarterly full backups to LTO-5 tapes that we
  send to another building (and annually to an Iron Mountain
  facility with 10 year retention), and incremental and differential
  backups in between to another tape pool.  We split the full
  backups into 5 jobs by parts of the directory hierarchy in order
  to keep the backups under a week long, with the 5th job being
  everything not included in the specific directories of the first 4
  jobs, and the PostgreSQL catalog.  It made a difference to run the
  backups from a separate server with dedicated spool RAID array,
  48GB RAM, and Fibre-Channel to the tape library, with the servers
  connected through a 10GbE Ethernet switch. We'll soon be getting
  80TB more data a year and so are getting a LTO-7 library and
  putting the mirrored storage in separate buildings (GlusterFS on
  top of ZFS).  I'm also thinking of using SSDs for the spool area;
  does anyone have recommendations on that?
  
  Robert Candey



 Original Message 
  Subject: Re: [Bacula-users] 350TB backup
  From: Thing 
  To: Bacula-users@lists.sourceforge.net
  
  Cc: "Bacula-users@lists.sourceforge.net"
  
  Date: Mon Oct 19 2015 17:49:23 GMT-0400 (EDT)



  
  Multiple NFS file systems on a NAS array.  500TB total, 350TB used.
Research data, much of it rarely accessed, after 1 year things like climate
data up to 30 years old, probably highly compressible.   Suspect multiple
bacula backup instances to distribute the load? Growth about 30tb a year.

On 20 October 2015 at 10:08, Dimitri Maziuk  wrote:


  
On 10/19/2015 03:53 PM, Thing wrote:


  Hi,

Is anyone backing total volumes of this order?  and if so, what sort of
scaling, design, hardware?



I take it, that's the size of your filesystems? Not the estimated size
of the backup set (i.e. all cycles in retention period)?

--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



--

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users







Multiple NFS file systems on a NAS array.  500TB
  total, 350TB used. Research data, much of it rarely accessed,
  after 1 year things like climate data up to 30 years old,
  probably highly compressible.   Suspect multiple bacula backup
  instances to distribute the load? Growth about 30tb a year.


  On 20 October 2015 at 10:08, Dimitri
Maziuk 
wrote:
On 10/19/2015 03:53 PM, Thing wrote:
> Hi,
>
> Is anyone backing total volumes of this order?  and
if so, what sort of
> scaling, design, hardware?

  I take it, that's the size of your filesystems? Not
  the estimated size
  of the backup set (i.e. all cycles in retention period)?
  
  --
  Dimitri Maziuk
  Programmer/sysadmin
  BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
  

--
  
  ___
  Bacula-users mailing list
  Bacula-users@lists.sourceforge.net
  https://lists.sourceforge.net/lists/listinfo/bacula-users
  

  
  




--




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

  


  


--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] how to debug a job

[Robert M. Candey]

We modify the Bacula source to remove this limit since we regularly run 10-14 
day backups. As an archive, we do full backups of each volume annually to go to 
Iron Mountain and quarterly to go to another building (data are mirrored online 
as well), with volumes on the order of 50-100 TB.  Splitting these up further 
runs the risk of missing some directories, and complicates restore.  I do not 
understand the need for a hard-coded limit; perhaps kill a job with no activity 
after some period instead.  We are pleased with Bacula and have found it very 
reliable.

Robert Candey

>  Original Message 
> Subject: Re: [Bacula-users] how to debug a job
> From: Roberts, Ben 
> To: Kern Sibbald 
> Cc: bacula-users 
> Date: Mon Jan 26 2015 02:51:10 GMT-0500 (EST)
>
>
> Hi Kern,
>
> >> Hard-coded, huh? Nobody's tried backing up that big data I keep hearing 
> >> about?
>
> > No, there is no change in the hard coded 6 day limit, and at the moment, I 
> personally
>
> > am not planning to implement anything, for two reasons: 1. I would like to 
> limit the
>
> > number of new Directives to what is absolutely necessary because there are 
> already
>
> > more than I can remember.  2. In my opinion, any Job that runs more than 6 
> days is
>
> > virtually destined to have problems during restore -- i.e. you will likely 
> have backup
>
> > dates that span 6 days of time in a single job.  That appears to me to be 
> something very undesirable.
>
> Just to add my views to the 6-day limit conversation I frequently have issues 
> running into this limit. As I write I am crossing my fingers hoping that a 
> 44TB backup job will complete in time. If it maintains a constant speed to 
> 100MB/sec it should in theory take 5.6 days however my other weekend jobs 
> have 
> slowed this average down to just 71MB/sec so I suspect it’s going to fail L. 
> If it does, I think I’ll be out of options other than to patch and recompile 
> Bacula to remove the limit myself.
>
> This is a ZFS snapshot being written via bpipe so regardless of how long it 
> takes to write out on the storage daemon there’s no risk of internal 
> inconsistency. While I understand the reason behind the limit when backups 
> are 
> taken at the file level, in this case the six-day limit serves no technical 
> purpose, and only causes me more work. An additional directive to configure 
> this, even if not used by 99% of Bacula users would be much appreciated if 
> only to save the maintenance overhead of patching each Bacula release.
>
> Regards,
>
> Ben Roberts
>
> *From:*Kern Sibbald [mailto:k...@sibbald.com]
> *Sent:* 25 January 2015 10:56
> *To:* Radosław Korzeniewski; Dimitri Maziuk
> *Cc:* bacula-users
> *Subject:* Re: [Bacula-users] how to debug a job
>
> On 23.01.2015 19:22, Radosław Korzeniewski wrote:
>
> Hello,
>
> 2015-01-22 3:42 GMT+01:00 Dimitri Maziuk  >:
>
> On 01/21/2015 06:41 PM, Bill Arlofski wrote:
>
> > Bacula has a hard-coded 6 day limit on a job's run time.   518401 
> seconds =
> > 6.1157 days, so it appears that is the cause for the watchdog
> killing the job.
>
> Hard-coded, huh? Nobody's tried backing up that big data I keep hearing
> about?
>
> Yes, but nobody was interested in changing it to the config parameter. It
> is possible that someone did that in 7.x, I need to check.
>
>
> No, there is no change in the hard coded 6 day limit, and at the moment, I 
> personally am not planning to implement anything, for two reasons: 1. I would 
> like to limit the number of new Directives to what is absolutely necessary 
> because there are already more than I can remember.  2. In my opinion, any 
> Job 
> that runs more than 6 days is virtually destined to have problems during 
> restore -- i.e. you will likely have backup dates that span 6 days of time in 
> a single job.  That appears to me to be something very undesirable.
>
> Best regards,
> Kern
>


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula, chocolatey and homebrew

[Robert M. Candey]

For Macs, I find it easier to make an installer package than a brew install:

For the latest MacOS X, I added to postflight.in:
mkdir -p -m 0755 /usr/local/var

add to autoconf/confdefs.h (or crc32.c):
#define HAVE_LITTLE_ENDIAN 1
since it wasn't getting defined for some reason.

Then do:
./configure --enable-client-only --without-tcp-wrappers --disable-conio
make -C platforms/osx

and it makes a .dmg file that you can copy around to various clients.

Robert Candey

The contents of this message are mine personally and do not reflect 
any position of the US Government or NASA.


Date: Thu, 28 Nov 2013 17:09:50 +0100
From: Julien Cochennec 
To: 
Subject: [Bacula-users] Bacula, chocolatey and homebrew

Hi,

1) Homebrew
I'd like to install bacula-fd on Mac computers easily, therefore I had a
look at Macports, Fink and Homebrew, and Homebrew is definitely the
fastest tool here, so I'd like to install Bacula with Homebrew. Is there
anyone who has tried this yet?
Bacula is in homebrew formulas but only in version 5.2.13 as you can see
in this script :

require  'formula'

class  BaculaFd  <  Formula
url 
'http://downloads.sourceforge.net/project/bacula/bacula/5.2.12/bacula-5.2.12.tar.gz'
homepage  'http://www.bacula.org/'
md5  'b04c22b128b73359e4bbc9de06652c38'

def  install
  system  "./configure",  "--prefix=#{prefix}",  "--sbindir=#{bin}",
"--with-working-dir=#{prefix}/working",
"--with-pid-dir=#{HOMEBREW_PREFIX}/var/run",
"--enable-client-only",
"--disable-conio"
  system  "make"
  system  "make install"

  # Ensure var/run exists:
  (var  +  'run').mkpath
end

end

Is there a way to install bacula 5.2.6 by modifying this script? Do I
have to change the MD5 parameter if I replace the 5.2.12 parameter?

2) Chocolatey
Same question for chocolatey, has anyone succeeded in installing
Bacula-fd on windows (>=7) with chocolatey?

3) Is installing 5.2.13 version of bacula on the Debian server a good
idea? Or is it to early?

--
Rapidly troubleshoot problems before they affect your business. Most IT 
organizations don't have a clear picture of how application performance 
affects their revenue. With AppDynamics, you get 100% visibility into your 
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] backing up clients on two separate networks

[Robert M. Candey]

Ana and Radoslaw,


Very useful questions.  DIR and SD are on same backup server.  We had 
checked port access between systems but not within the backup server 
itself.  Solution turned out to be adding the second IP (private) 
address to hosts.allow at the top, along with the other ALLOW ALL on 
127.0.0.1 and its default address (in addition to defining two 
Storage and JobDefs resources).  We removed the 0.0.0.0 addresses we 
had added, as not needed.  Thank you very much.


Robert Candey


>Date: Thu, 19 Sep 2013 10:57:24 -0300
>From: Ana Emília M. Arruda 
>Subject: Re: [Bacula-users] backing up clients on two separate networks
>
>Hi!
>
>So, this is not the problem :)
>
>Maybe you should check for network communication problems with some 
>telnet commands:
>
>- from client (192.168.0.X) to bacula-sd (192.168.0.5): telnet on port 9103
>
>- from client (192.168.0.X) to bacula-sd (1G network ip address): 
>telnet on port 9103
>
>- from client (192.168.0.X) to bacula-dir (192.168.0.Y): telnet on port 9101
>
>- from director (192.168.0.Y) to client (192.168.0.X): telnet on port 9102
>
>- from client (192.168.0.X) to bacula-dir (1G network ip address): 
>telnet on port 9101
>
>- from director (1G network ip address) to client (192.168.0.X): 
>telnet on port 9102
>
>
>- from director (192.168.0.Y) to bacula-sd (192.168.0.5): telnet on port 9103
>
>
>- from bacula-sd (192.168.0.5) to director (192.168.0.Y): telnet on port 9101
>
>Don´t know if your director and bacula-sd are on the same host. If 
>so, they should be able to talk to each other.
>
>Regards,
>Ana

--
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. 
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] backing up clients on two separate networks

[Robert M. Candey]

[RESEND: many apologies for sending as HTML]

As a follow up on Uthra Rao's question ("client connect to storage 
daemon problem"), here's a different explanation of what we are 
trying to accomplish, plus the requested config files.

We have been successfully using Bacula for many years and now need to 
add clients on a separate high speed network (due to the volume of 
data).  We added a 10 GbE switch and network interface cards to the 
large data clients (but not all) and the backup server, to form a 
private network (192.168.0.*).

Since the Storage resource is defined with one address (on the 
regular network) and not its address on the private network, we get 
errors:

Fatal error: Authorization key rejected by Storage daemon.
Fatal error: Failed to authenticate Storage daemon.
Fatal error: Bad response to Storage command: wanted 2000 OK storage 
, got 2902 Bad storage

The "Dealing with Firewalls" section of the manual talks about 
clients on two networks, although notes that it hasn't been tested.

Early versions of the manual refer to defining two Storage resources 
and two corresponding Job resources, differing in Name and Address 
.  This 
gives the same errors.

The latest manual instead defines a generic Storage-server name and 
depends on the hosts file on each client to refer to the correct name 
for that network.  The two Job resources are there, probably left 
over from the earlier version, and I assume are not needed.  [It also 
may be difficult to maintain the appropriate changes to the hosts 
files on dozens of clients.]


Below are the beginning of our config files.  The "*Address = 
0.0.0.0" was added based on an old comment on the list, to ensure the 
services listen to all network interfaces (which they seem to do by 
default anyway).

I assume the requirement to match the storage server name/address is 
for added security over the passwords, but it makes this situation 
much more difficult.

Has anyone succeeded at running one backup server with clients on two 
networks and network cards?  Any suggestions for this?  Thanx.

Robert Candey


Director {
   Name = backup-dir
   DirAddress = 0.0.0.0
   DIRport = 9101# where we listen for UA connections
   QueryFile = "/usr/local/bacula/etc/query.sql"
   WorkingDirectory = "/var/bacula/working"
   PidDirectory = "/var/run"
   ScriptsDirectory = /usr/local/etc/bacula-clients
   Maximum Concurrent Jobs = 4
   Password =  # Console password
   Messages = Daemon
   fd connect timeout = 60sec
   Heartbeat Interval = 60
}

JobDefs {
   Name = standard-job
   Type = Backup
   Level = Incremental
   FileSet = standard-set
   Schedule = standard-sched
   Storage = jukebox
   Messages = Standard
   Pool = server-partial
   Priority = 10
   Write Bootstrap = "/var/bacula/working/BootStrap/%c.bsr"
   Spool data = yes
}

JobDefs {
   Name = standard-job-p
   Type = Backup
   Level = Incremental
   FileSet = standard-set
   Schedule = standard-sched
   Storage = jukebox-p
   Messages = Standard
   Pool = server-partial
   Priority = 10
   Write Bootstrap = "/var/bacula/working/BootStrap/%c.bsr"
   Spool data = yes
}


Storage {
   Name = jukebox
   Address = backup.gsfc.nasa.gov
   SDPort = 9103
   Password = 
   Device = Autochanger
   Media Type = LTO-5
   autochanger = yes
   maximum concurrent jobs = 20
}

Storage {
   Name = jukebox-p
   Address = 192.168.0.5
   SDPort = 9103
   Password = 
   Device = Autochanger
   Media Type = LTO-5
   autochanger = yes
   maximum concurrent jobs = 20
}


bacula-sd.conf:
Storage {
   Name = backup-sd
   SDAddress = 0.0.0.0
   SDPort = 9103  # Director's port 
   WorkingDirectory = "/var/bacula/working"
   Pid Directory = "/var/run"
   Maximum Concurrent Jobs = 20
   Heartbeat Interval = 60
}

#
# List Directors who are permitted to contact Storage daemon
#
Director {
   Name = backup-dir
   Password = 
}


bacula-fd.conf:
Director {
Name = backup-dir
Password = 
}

FileDaemon {
   Name = backup-fd
   FDAddress = 0.0.0.0
   FDport = 9102  # where we listen for the director
   WorkingDirectory = /var/bacula/working
   Pid Directory = /var/run
   Maximum Concurrent Jobs = 20
   Heartbeat Interval = 60
}

--
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. 
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] backing up clients on two separate networks

[Robert M. Candey]

Title: backing up clients on two separate
networks


As a follow up on Uthra Rao's question ("client connect to
storage daemon problem"), here's a different explanation of what
we are trying to accomplish, plus the requested config files.

We have been successfully using Bacula for many years and now
need to add clients on a separate high speed network (due to the
volume of data).  We added a 10 GbE switch and network interface
cards to the large data clients (but not all) and the backup server,
to form a private network (192.168.0.*). 

Since the Storage resource is defined with one address (on the
regular network) and not its address on the private network, we get
errors:

Fatal error: Authorization key rejected by Storage daemon.
Fatal error: Failed to authenticate Storage daemon.
Fatal error: Bad response to Storage command: wanted 2000 OK storage ,
got 2902 Bad storage

The "Dealing with Firewalls" section of the manual
talks about clients on two networks, although notes that it hasn't
been tested.

Early versions of the manual refer to defining two Storage
resources and two corresponding Job resources, differing in Name and
Address
. 
This gives the same errors.

The latest manual instead defines a generic Storage-server name
and depends on the hosts file on each client to refer to the correct
name for that network.  The two Job resources are there, probably
left over from the earlier version, and I assume are not needed. 
[It also may be difficult to maintain the appropriate changes to the
hosts files on dozens of clients.]
h_Firewalls.html>

Below are the beginning of our config files.  The
"*Address = 0.0.0.0" was added based on an old comment on
the list, to ensure the services listen to all network interfaces
(which they seem to do by default anyway).

I assume the requirement to match the storage server name/address
is for added security over the passwords, but it makes this situation
much more difficult.

Has anyone succeeded at running one backup server with clients on
two networks and network cards?  Any suggestions for this? 
Thanx.

Robert Candey


Director { 
  Name = backup-dir
  DirAddress = 0.0.0.0
  DIRport =
9101    # where we listen for UA
connections
  QueryFile =
"/usr/local/bacula/etc/query.sql"
  WorkingDirectory = "/var/bacula/working"
  PidDirectory = "/var/run"
  ScriptsDirectory = /usr/local/etc/bacula-clients
  Maximum Concurrent Jobs = 4
  Password =
 #
Console password
  Messages = Daemon
  fd connect timeout = 60sec
  Heartbeat Interval = 60 
}

JobDefs {
  Name = standard-job
  Type = Backup
  Level = Incremental
  FileSet = standard-set
  Schedule = standard-sched
  Storage = jukebox
  Messages = Standard
  Pool = server-partial
  Priority = 10
  Write Bootstrap =
"/var/bacula/working/BootStrap/%c.bsr"
  Spool data = "">
}

JobDefs {
  Name = standard-job-p
  Type = Backup
  Level = Incremental
  FileSet = standard-set
  Schedule = standard-sched
  Storage = jukebox-p
  Messages = Standard
  Pool = server-partial
  Priority = 10
  Write Bootstrap =
"/var/bacula/working/BootStrap/%c.bsr"
  Spool data = "">
}


Storage {
  Name = jukebox
  Address = backup.gsfc.nasa.gov
  SDPort = 9103
  Password = 
  Device = Autochanger
  Media Type = LTO-5
  autochanger = yes
  maximum concurrent jobs = 20
}

Storage {
  Name = jukebox-p
  Address = 192.168.0.5
  SDPort = 9103
  Password = 
  Device = Autochanger
  Media Type = LTO-5
  autochanger = yes
  maximum concurrent jobs = 20
}


bacula-sd.conf:
Storage {  
  Name = backup-sd
  SDAddress = 0.0.0.0
  SDPort =
9103  # Director's
port 
  WorkingDirectory = "/var/bacula/working"
  Pid Directory = "/var/run"
  Maximum Concurrent Jobs = 20
  Heartbeat Interval = 60
}

#
# List Directors who are permitted to contact Storage daemon
#
Director {
  Name = backup-dir
  Password = 
}


bacula-fd.conf:
Director {
   Name = backup-dir
   Password = 
}

FileDaemon { 
  Name = backup-fd
  FDAddress = 0.0.0.0
  FDport =
9102  # where we listen
for the director
  WorkingDirectory = /var/bacula/working
  Pid Directory = /var/run
  Maximum Concurrent Jobs = 20
  Heartbeat Interval = 60
}



--
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. 
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users