Re: [Bacula-users] Pruning is taking too long; downside of force update status to recycle.

Kern Sibbald Wed, 07 Dec 2016 01:27:58 -0800

Hello,

Here are a few tips for you.

- You absolutely need to understand what Pruning is. The best way to understand it is to read the manual.

- If you are running MySQL as that catalog and you are having catalog performance issues, you should consider switching to Postgresql.

- Usually performance issues with the catalog are related to several points
   1. You may have added additional indexes. Doing so sometimes speeds up a particular operation,
       but much more often totally destroys the performance elsewhere. If you have added any, start by
       removing them.
   2. Every database and particularly Postgresql needs some tuning for your system and your workload.
       Generally the database vendor has programs that will help you find problems.

- If you are running more than 1000 jobs per day, then you clearly need Postgresql and you need it to be properly tuned. You may also need to turn off automatic pruning and specifically schedule a prune.

- Pruning of a single job should never take an excessive amount of time, but if as noted above you have 1000 jobs, it is not always ideal to do it while backups are running.

- You should *never* be manually (with SQL commands) be changing the Bacula catalog.

- You should almost never need to use a built in command to modify the catalog Media records. This does not include running prune or purge commands.

- If you are running more than 500 jobs a day, you need to take the Bacula Admin I course.

- If you are running more than 500 jobs and not a Bacula expert, you most likely need a professional service contract.

Best regards,
Kern

On 12/06/2016 09:55 PM, Gi Dot wrote:

Weird thing is, after the long pruning, I decided to change the volume status to recycle. Done that, the backup worked fine. After that I changed the status of all volumes to recycle. But last night, the scheduled backup didn't run because of pruning vol-F1. I thought when the volume is recycled it will skip all the pruning part.

On Wed, Dec 7, 2016 at 4:41 AM, Gi Dot <gadi...@gmail.com> wrote:

All of the volumes were full at the time, and have past the retention period. Bacula prunes the oldest volume, and stuck at it.

Sorry if this question is trivial, but I don't really understand purging. If I were to purge a volume manually, I will definitely lose all of the backups in the volume. And all the jobs associated with the volume will be removed from the catalog. So how does it make me lose other backups that I wanted to keep? Isn't it pretty much the same if I were to recycle the volume?

On Tue, Dec 6, 2016 at 3:18 AM, Martin Simmons <mar...@lispworks.com> wrote:

>>>>> On Mon, 5 Dec 2016 17:40:08 +0800, Gi Dot said:
>
> > What exactly do you mean by "too long"? Does bacula encounter a
> > timeout during the pruning from a database error?
> Like it runs overnight and still not done with it. Sample log:
> https://dpaste.de/wetb/raw

This doesn't look like a database indexing problem to me.

It looks like it needs an extra volume and fails to find one older than the
retention periods.

Did you expect vol-D1 to fill up? If so, which volume should it use next?
You need to look at the retention period of that volume to see why it was not
recycled.

BTW, forcing a volume from Used to Recycle using the update command is a bad
idea because the database will bloat with unwanted file records. You could
use the purge command but beware that you might lose backups that you wanted
to keep.

__Martin

>
> Thanks.
>
>
>
> On Mon, Dec 5, 2016 at 5:02 PM, Uwe Schuerkamp <uwe.schuerk...@nionex.net>
> wrote:
>
> > On Mon, Dec 05, 2016 at 03:18:45PM +0800, Gi Dot wrote:
> > > Hello,
> > >
> > > I have this problem with one of my client experiencing pruning of a
> > volume
> > > that is taking too long (and in the end I ended up recycling it manuall=
> y
> > by
> > > updating the volume status). I have googled up on this and from what I
> > > understand it is mostly due to the database indexing (to be honest I
> > don't
> > > entirely understand this part).
> > >
> > > My question is, is there any downside or side effect if I were to
> > include a
> > > script that looks up for Used volume and update it to Recycle before th=
> e
> > > backup runs for the day. I am using this script on another client and
> > > things are going fine over there, but I'm just worried if there is any
> > > impact in a long run.
> > >
> > > If anyone would be so kind to explain to me what exactly it means by
> > > pruning; as in what bacula does when it runs pruning on a volume, it is
> > > much appreciated as well. I have read somewhere that bacula removes the
> > > jobs associated with the volume from the catalog.
> > >
> >
> > Hello Gidot,
> >
> > some more info could be useful to help you in analyzing your setup
> > further.
> >
> > - Hardware specs of the director (assuming all components run on a
> > single machine)
> >
> > - Which database are you using for the catalog?
> >
> > - Amount of RAM available to the DB / backend storage (disks, ssds?)
> >
> > - Catalog size (file table rows)
> >
> > - Bacula version
> >
> > What exactly do you mean by "too long"? Does bacula encounter a
> > timeout during the pruning from a database error?
> >
> > All the best,
> >
> > Uwe
> >
> >
> > --
> > Uwe Sch=C3=BCrkamp | email: <uwe.schuerk...@nionex.net>
> >
> >
> >
> >
> >
> >
> >
> >
>
> --001a113fc4b4cef4d20542e612a1
> Content-Type: text/html; charset=UTF-8
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr"><div>- Hardware specs of the director (assuming all compon=
> ents run on a<br>
> single machine)<br>
> </div>>> Backup runs on a database server. There's also one remot=
> e client being backed up once a week.<br><div><br>
> - Which database are you using for the catalog?<br></div><div>>> Post=
> greSQL<br></div><div>
> <br>
> - Amount of RAM available to the DB / backend storage (disks, ssds?)<br></d=
> iv><div>>> 32G RAM on host, 8GB for effective cache size, 8MB for wor=
> k mem, and 8GB for shared buffers / SAS disks<br></div><div>
> <br>
> - Catalog size (file table rows)<br></div><div>>> The size of the cat=
> alog is close to 5MB. Nothing huge.<br></div><div>
> <br>
> - Bacula version<br></div><div>>> 5.2.12-3.1<br></div><div>
> <br>
> What exactly do you mean by "too long"? Does bacula encounter a<b=
> r>
> timeout during the pruning from a database error?<br></div><div>>> Li=
> ke it runs overnight and still not done with it. Sample log:<br><a href=""> > https://dpaste.de/wetb/raw">https://dpaste.de/wetb/raw</a><br><br></div><di=
> v>Thanks.<br><br><br></div><div><div class=3D"gmail_extra"><br><div class=
> =3D"gmail_quote">On Mon, Dec 5, 2016 at 5:02 PM, Uwe Schuerkamp <span dir=
> =3D"ltr"><<a href="" moz-do-not-send="true" href="mailto:uwe.schuerk...@nionex.net" target="_blank">uwe.schuerkamp@nionex.net" target=3D"_blank"=
> >uwe.schuerk...@nionex.net</a>></span> wrote:<br><blockquote class=3D"gm=
> ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,=
> 204,204);padding-left:1ex"><span class=3D"gmail-">On Mon, Dec 05, 2016 at 0=
> 3:18:45PM +0800, Gi Dot wrote:<br>
> > Hello,<br>
> ><br>
> > I have this problem with one of my client experiencing pruning of a vo=
> lume<br>
> > that is taking too long (and in the end I ended up recycling it manual=
> ly by<br>
> > updating the volume status). I have googled up on this and from what I=
> <br>
> > understand it is mostly due to the database indexing (to be honest I d=
> on't<br>
> > entirely understand this part).<br>
> ><br>
> > My question is, is there any downside or side effect if I were to incl=
> ude a<br>
> > script that looks up for Used volume and update it to Recycle before t=
> he<br>
> > backup runs for the day. I am using this script on another client and<=
> br>
> > things are going fine over there, but I'm just worried if there is=
> any<br>
> > impact in a long run.<br>
> ><br>
> > If anyone would be so kind to explain to me what exactly it means by<b=
> r>
> > pruning; as in what bacula does when it runs pruning on a volume, it i=
> s<br>
> > much appreciated as well. I have read somewhere that bacula removes th=
> e<br>
> > jobs associated=C2=A0 with the volume from the catalog.<br>
> ><br>
> <br>
> </span>Hello Gidot,<br>
> <br>
> some more info could be useful to help you in analyzing your setup<br>
> further.<br>
> <br>
> - Hardware specs of the director (assuming all components run on a<br>
> single machine)<br>
> <br>
> - Which database are you using for the catalog?<br>
> <br>
> - Amount of RAM available to the DB / backend storage (disks, ssds?)<br>
> <br>
> - Catalog size (file table rows)<br>
> <br>
> - Bacula version<br>
> <br>
> What exactly do you mean by "too long"? Does bacula encounter a<b=
> r>
> timeout during the pruning from a database error?<br>
> <br>
> All the best,<br>
> <br>
> Uwe<br>
> <br>
> <br>
> --<br>
> Uwe Sch=C3=BCrkamp | email: <<a href="" moz-do-not-send="true" href="mailto:uwe.schuerk...@nionex.net" target="_blank">uwe.schuerkamp@nionex.net=
> ">uwe.schuerk...@nionex.net</a>><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> </blockquote></div><br></div></div></div>
>
> --001a113fc4b4cef4d20542e612a1--
>
>
> --===============8049073509691406397==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> ------------------------------------------------------------------------------
>
> --===============8049073509691406397==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> _______________________________________________
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
> --===============8049073509691406397==--

>

------------------------------------------------------------------------------
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/xeonphi
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/xeonphi

_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Pruning is taking too long; downside of force update status to recycle.

Reply via email to