Re: [Bacula-users] Backup from client with deduplication
Hello, śr., 7 paź 2020 o 06:37 Рукавцов Дмитрий Геннадьевич napisał(a): > So, there is no solution? Bacula can't backup such clients? > I know nothing about windows filesystem deduplication capabilities, but I assume to properly read contents of these files a standard win32 BackupRead() dedicated to backup function is not sufficient and requires special handling. I do not know if this special handling is available in Bacula, but as this is not my area of interests in Bacula my answer could be incomplete. best regards -- Radosław Korzeniewski rados...@korzeniewski.net ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Discrepancy between backup/restore size and files
Hello, śr., 30 wrz 2020 o 12:33 Ben Laurie napisał(a): > > > On Wed, 30 Sep 2020 at 09:42, Radosław Korzeniewski < > rados...@korzeniewski.net> wrote: > >> Hello, >> >> pon., 28 wrz 2020 o 14:34 Ben Laurie napisał(a): >> >>> >>> >>> On Sun, 13 Sep 2020 at 18:58, Josip Deanovic >>> wrote: >>> On Sunday 2020-09-13 15:43:31 Ben Laurie wrote: > > Not sure if IgnoreCase option could caused it somehow. > > According to the documentation it just makes it ignore case for pattern > matching, so seems unlikely? I can't say because I didn't use Bacula with Windows a lot. It worked fine back in 2010-2012. > Yes, funny I somehow had not noticed that before - I've added it. I > presume it won't fix this problem, tho. No, but it will make the difference for future backups. >>> >>> I tried using accurate but it just caused errors (I forget what now, but >>> some problem with database searches?). >>> >>> In other news, I did a full restore as a test recently and I got quite a >>> few errors along the lines of: >>> >>> 25-Sep 14:46 xxx-fd JobId 35: Error: File /tmp/bacula-restores/k/xxx >>> already exists and could not be replaced. ERR=Permission denied. >>> >> >> First of all to properly restore Windows files on any Linux or Unix >> machine you have to set a Portable flag for your backup. >> > > According to > https://www.bacula.org/5.1.x-manuals/fr/main/main/Windows_Version_Bacula.html > that is no longer true... > > "Note: with Bacula versions 1.39.x and later, non-portable Windows data > can be restore to any machine." > Not exactly. You can restore some data because non windows file daemon accept windows stream. And this is what the above statement means. But this stream is what win32 BackupRead() generates. It could be the same bit2bit what standard read() can generate but there is no guarantee. The Portable flag replaces win32 BackupRead() for a simple read(). best regards -- Radosław Korzeniewski rados...@korzeniewski.net ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup from client with deduplication
So, there is no solution? Bacula can't backup such clients?
Maybe i explained something wrong. I have bacula server on centos
(server), i don't have any deduplication on the bacula server and i
don't need it there. I have windows 2019 server (client), it has it's
own deduplication on it's own volume which is enabled in windows
settings. From this windows volume i need to backup files.
Full backup size before i turn on deduplication (which is windows
deduplication) on windows server (client) was like 200GB, full backup
after i turn on deduplication with same files on the same server was
6.7GB, files count same, when i restoring it, most files is
unreadable. I'm thinking cause bacula makes snapshot of the files when
making backup and it's not working with deduplication on windows
server (client), but maybe i'm wrong.
I just want normal (recoverable) backup from this windows server
(which is 200gb), i don't need any deduplication on my backups.
This is FileSet
FileSet {
Name = "winsrv"
Enable VSS = no
Include {
Options {
compression = GZIP
signature = MD5
aclsupport = yes
}
File = D:/FS/
}
}
this is log from backup:
Sep 29 19:20:36 bacula bacula-dir: Build OS:
x86_64-redhat-linux-gnu-bacula redhat (Core)
Sep 29 19:20:36 bacula bacula-dir: JobId: 3814
Sep 29 19:20:36 bacula bacula-dir: Job: gc-main.2020-09-29_10.58.43_03
Sep 29 19:20:36 bacula bacula-dir: Backup Level: Full
Sep 29 19:20:36 bacula bacula-dir: Client: "winsrv" 9.6.5 (11Jun20)
Microsoft Standard Edition (build 9200), 64-bit,Cross-compile,Win64
Sep 29 19:20:36 bacula bacula-dir: FileSet: "winsrv" 2020-09-15 22:00:00
Sep 29 19:20:36 bacula bacula-dir: Pool: "main-pool" (From Job resource)
Sep 29 19:20:36 bacula bacula-dir: Catalog: "default" (From Client
resource)
Sep 29 19:20:36 bacula bacula-dir: Storage: "backup-sd" (From Job
resource)
Sep 29 19:20:36 bacula bacula-dir: Scheduled time: 29-сен-2020 19:00:00
Sep 29 19:20:36 bacula bacula-dir: Start time: 29-сен-2020 19:00:02
Sep 29 19:20:36 bacula bacula-dir: End time: 29-сен-2020 19:20:36
Sep 29 19:20:36 bacula bacula-dir: Elapsed time: 20 mins 34
secs
Sep 29 19:20:36 bacula bacula-dir: Priority: 1
Sep 29 19:20:36 bacula bacula-dir: FD Files Written: 905,999
Sep 29 19:20:36 bacula bacula-dir: SD Files Written: 905,999
Sep 29 19:20:36 bacula bacula-dir: FD Bytes Written: 6,500,269,762
(6.500 GB)
Sep 29 19:20:36 bacula bacula-dir: SD Bytes Written: 6,708,284,663
(6.708 GB)
Sep 29 19:20:36 bacula bacula-dir: Rate: 5267.6 KB/s
Sep 29 19:20:36 bacula bacula-dir: Software Compression: 15.7% 1.2:1
Sep 29 19:20:36 bacula bacula-dir: Comm Line Compression: 0.6% 1.0:1
Sep 29 19:20:36 bacula bacula-dir: Snapshot/VSS: no
Sep 29 19:20:36 bacula bacula-dir: Encryption: no
Sep 29 19:20:36 bacula bacula-dir: Accurate: no
Sep 29 19:20:36 bacula bacula-dir: Volume name(s): All0121
Sep 29 19:20:36 bacula bacula-dir: Volume Session Id: 1
Sep 29 19:20:36 bacula bacula-dir: Volume Session Time: 1601366166
Sep 29 19:20:36 bacula bacula-dir: Last Volume Bytes: 6,743,816,269
(6.743 GB)
Sep 29 19:20:36 bacula bacula-dir: Non-fatal FD errors: 2
Sep 29 19:20:36 bacula bacula-dir: SD Errors: 0
Sep 29 19:20:36 bacula bacula-dir: FD termination status: OK
Sep 29 19:20:36 bacula bacula-dir: SD termination status: OK
Sep 29 19:20:36 bacula bacula-dir: Termination: Backup OK -- with
warnings
30.09.2020 13:22, Radosław Korzeniewski пишет:
Hello,
wt., 29 wrz 2020 o 09:50 Рукавцов Дмитрий Геннадьевич
mailto:m...@santel-navi.ru>> napisał(a):
Hello, i have bacula server 9.6.5 on Centos and i need to backup
folders
from Windows 2019 server client with deduplication on (optimized
files).
Problem if i'm making Full backup, the archive is incomplete
Could you show the job terminate log?
(files
count match, but size is wrong)
I do not understand how you judge that a size is wrong?
Is there any way to tell Bacula server
to ignore deduplication (optimized files) on client or any other way?
I do not understand your statement. Could you elaborate what you want
to achieve and what configuration you use?
Deduplication in Bacula Enterprise Edition is called GED (Global
Endpoint Deduplication) and it is performed on client side (endpoint)
and on storage daemon. You can enable/disable deduplication for a
single FileSet/Include resource.
A deduplication optimized volumes functionality (aligned plugin)
available on Bacula Community requires an external deduplication
solution, hardware or software level to perform the required
computation and management. This solution does not involve a backup
client in any way.
Tried to set "Enable VSS = no" at FileSet but it gives nothing.
VSS is a Microsoft Windo
[Bacula-users] Baculum 9.6.6.1
Hello, We are pleased to announce a new Baculum 9.6.6.1 version. This is a minor new feature and bug fix release. It introduces new functions to the Security page which help creating user accounts with restricted Bacula resources access. Binary packages for popular distributions are available already. New changes and fixes since version 9.6.6: - New features and improvements to multi-user interface and restricted access - Add default days and limit values to status schedules API endpoint - Update Portuguese translations - Fix displaying multitextbox and multicombobox directive controls - Fix date filter on status schedule page - Fix #2570 fileset excludes settings with PHP 7.4 Sources: https://www.bacula.org/downloads/baculum/releases/9.6.6.1/baculum-9.6.6.1.tar.gz https://www.bacula.org/downloads/baculum/releases/9.6.6.1/baculum-9.6.6.1.tar.gz.sig Useful links: Documentation: https://www.bacula.org/9.6.x-manuals/en/console/Baculum_API_Web_GUI_Tools.html New features 9.6.6.1 video: https://www.youtube.com/watch?v=cvKtvZCvjLQ Online Demo: https://baculum.app Thank you to everybody involved in preparing this release. Best regards, Marcin Haba (gani) -- "Greater love hath no man than this, that a man lay down his life for his friends." Jesus Christ "Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie za przyjaciół swoich." Jezus Chrystus ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] performance&design&configuration challenges
Hi,
I have done some testing:
a) testing storage with dd command (eg: dd if=/dev/zero
of=/storage/test1.img bs=1G count=1 oflag=dsync). The results are:
-writing to IBM storage (with cloud enabled) shows 300 MB/sec
-writing to local SSD storage shows 600 MB/sec.
I guess storage is not a bottleneck.
b) testing file copy from linux centos 6 server to bacula server with
rsync (eg. rsync --info=progress2 source destination)
-writing to local storage: 82 MB/sec
-writing to IBM storage: 85 MB/sec
I guess this is ok for a 1 GB network link.
c) using bacula:
-linux centos 6 file server: 13 MB/sec on IBM storage, 16 MB/sec on
local SSD storage (version of client 5.2.13).
-windows file server: around 18 MB/sec - there could be some additional
problem, because I perform a backup from a deduplicated drive (version
of client 9.6.5)
d) I have tried to manipulate encryption/compression settings, but I
believe there is no significant difference
I think that bacula rate (15 MB/sec) in quite slow comparing to file
copy results (85 MB/sec) from the same client/server. It should be
better... Do you agree?
I have implemented autochanger in order to perform backup from both
servers at the same time. We shall see the results tomorrow.
I have not changed the version of the client on linux server yet. My
windows server uses new client version, so that was not my first idea...
Will try this tomorrow if needed.
What about retention?
I would like to:
- create incremental daily backup (retention 1 week)
- create weekly full backup (retention 1 month)
- create monthly full backup (retention 1 year)
At the moment I use different job/schedule for monthly backup, but that
triggers full backup also on Monday after monthly backup (I would like
to run incremental then). Is there a better way? Relevant parts of conf
below...
Regards,
Ziga
JobDefs {
Name = "bazar2-job"
Schedule = "WeeklyCycle"
...
}
Job {
Name = "bazar2-backup"
JobDefs = "bazar2-job"
Full Backup Pool = bazar2-weekly-pool
Incremental Backup Pool = bazar2-daily-pool
}
Job {
Name = "bazar2-monthly-backup"
Level = Full
JobDefs = "bazar2-job"
Pool = bazar2-monthly-pool
Schedule = "MonthlyFull" #schedule : see in bacula-dir.conf (monthly
pool with longer retention)
}
Example output:
06-Oct 12:19 bacula-dir JobId 714: Bacula bacula-dir 9.6.5 (11Jun20):
Build OS: x86_64-redhat-linux-gnu-bacula redhat (Core)
JobId: 714
Job:bazar2-monthly-backup.2020-10-06_09.33.25_03
Backup Level: Full
Client: "bazar2.kranj.cetrtapot.si-fd" 5.2.13 (19Jan13)
x86_64-redhat-linux-gnu,redhat,(Core)
FileSet:"bazar2-fileset" 2020-09-30 15:40:26
Pool: "bazar2-monthly-pool" (From Job resource)
Catalog:"MyCatalog" (From Client resource)
Storage:"FSTestBackup" (From Job resource)
Scheduled time: 06-Oct-2020 09:33:15
Start time: 06-Oct-2020 09:33:28
End time: 06-Oct-2020 12:19:19
Elapsed time: 2 hours 45 mins 51 secs
Priority: 10
FD Files Written: 53,682
SD Files Written: 53,682
FD Bytes Written: 168,149,175,433 (168.1 GB)
SD Bytes Written: 168,158,044,149 (168.1 GB)
Rate: 16897.7 KB/s
Software Compression: 36.6% 1.6:1
Comm Line Compression: None
Snapshot/VSS: no
Encryption: no
Accurate: no
Volume name(s): bazar2-monthly-vol-0300
Volume Session Id: 11
Volume Session Time:1601893281
Last Volume Bytes: 337,370,601,852 (337.3 GB)
Non-fatal FD errors:0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination:Backup OK
On 06.10.2020 14:28, Josh Fisher wrote:
On 10/6/20 3:45 AM, Žiga Žvan wrote:
I believe that I have my spooling attributes set correctly on jobdefs
(see bellow). Spool attributes = yes; Spool data defaults to no. Any
other idea for performance problems?
Regard,
Ziga
The client version is very old. First try updating the client to 9.6.x.
For testing purposes, create another storage device on local disk and
write a full backup to that. If it is much faster to local disk
storage than it is to the s3 driver, then there may be an issue with
how the s3 driver is compiled, version of s3 driver, etc.
Otherwise, with attribute spooling enabled, the status of the job as
given by the status dir command in bconsole will change to "despooling
attributes" or something like that when the client has finished
sending data. That is the period at the end of the job when the
spooled attributes are being written to the catalog database. If
despooling is taking a long time, then database performance might be
the bottleneck.
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.n
Re: [Bacula-users] performance&design&configuration challenges
On 10/6/20 3:45 AM, Žiga Žvan wrote: I believe that I have my spooling attributes set correctly on jobdefs (see bellow). Spool attributes = yes; Spool data defaults to no. Any other idea for performance problems? Regard, Ziga The client version is very old. First try updating the client to 9.6.x. For testing purposes, create another storage device on local disk and write a full backup to that. If it is much faster to local disk storage than it is to the s3 driver, then there may be an issue with how the s3 driver is compiled, version of s3 driver, etc. Otherwise, with attribute spooling enabled, the status of the job as given by the status dir command in bconsole will change to "despooling attributes" or something like that when the client has finished sending data. That is the period at the end of the job when the spooled attributes are being written to the catalog database. If despooling is taking a long time, then database performance might be the bottleneck. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] performance&design&configuration challenges
Hello Ziga,
Your client is probably too old for the 9.2.x Director.
Even CentOS 6 is old, most likely in the end of life.
Other than that you can try some tuning:
http://www.bacula.lat/tuning-better-performance-and-treatment-of-backup-bottlenecks/?lang=en
Rgds.
--
MSc Heitor Faria
CEO Bacula LatAm
mobile1: + 1 909 655-8971
mobile2: + 55 61 98268-4220
América Latina
[ http://bacula.lat/]
Original Message
From: Žiga Žvan
Sent: Tuesday, October 6, 2020 03:11 AM
To: bacula-users@lists.sourceforge.net
Subject: [Bacula-users] performance&design&configuration challenges
>Hi,
>
>I'm having some performance challenges. I would appreciate some educated
>guess from an experienced bacula user.
>
>I'm changing old backup sw that writes to tape drive with bacula
>writing to disk. The results are:
>a) windows file server backup from a deduplicated drive (1.700.000
>files, 900 GB data, deduplicated space used 600 GB). *Bacula: 12 hours,
>old software: 2.5 hours*
>b) linux file server backup (50.000 files, 166 GB data).*Bacula 3.5
>hours, old software: 1 hour*.
>
>I have tried to:
>a) turn off compression&encryption. The result is the same: backup speed
>around 13 MB/sec.
>b) change destination storage (from a new ibm storage attached over nfs,
>to a local SSD disk attached on bacula server virtual machine). It took
>2 hours 50 minutes to backup linux file server (instead of 3.5 hours).
>Sequential write test tested with linux dd command shows write speed 300
>MB/sec for IBM storage and 600 MB/sec for local SSD storage (far better
>than actual throughput).
>
>The network bandwidth is 1 GB (1 GB on client, 10 GB on server) so I
>guess this is not a problem; however I have noticed that bacula-fd on
>client side uses 100% of CPU.
>
>I'm using:
>-bacula server version 9.6.5
>-bacula client version 5.2.13 (original from centos 6 repo).
>
>Any idea what is wrong and/or what performance should I expect?
>I would also appreciate some answers on the questions bellow.
>
>Kind regards,
>Ziga Zvan
>
>
>
>
>On 05.08.2020 10:52, Žiga Žvan wrote:
>>
>> Dear all,
>> I have tested bacula sw (9.6.5) and I must say I'm quite happy with
>> the results (eg. compression, encryption, configureability). However I
>> have some configuration/design questions I hope, you can help me with.
>>
>> Regarding job schedule, I would like to:
>> - create incremental daily backup (retention 1 week)
>> - create weekly full backup (retention 1 month)
>> - create monthly full backup (retention 1 year)
>>
>> I am using dummy cloud driver that writes to local file storage.
>> Volume is a directory with fileparts. I would like to have seperate
>> volumes/pools for each client. I would like to delete the data on disk
>> after retention period expires. If possible, I would like to delete
>> just the fileparts with expired backup.
>>
>> Questions:
>> a) At the moment, I'm using two backup job definitions per client and
>> central schedule definition for all my clients. I have noticed that my
>> incremental job gets promoted to full after monthly backup ("No prior
>> Full backup Job record found"; because monthly backup is a seperate
>> job, but bacula searches for full backups inside the same job). Could
>> you please suggest a better configuration. If possible, I would like
>> to keep central schedule definition (If I manipulate pools in a
>> schedule resource, I would need to define them per client).
>>
>> b) I would like to delete expired backups on disk (and in the catalog
>> as well). At the moment I'm using one volume in a daily/weekly/monthly
>> pool per client. In a volume, there are fileparts belonging to expired
>> backups (eg. part1-23 in the output bellow). I have tried to solve
>> this with purge/prune scripts in my BackupCatalog job (as suggested in
>> the whitepapers) but the data does not get deleted. Is there any way
>> to delete fileparts? Should I create separate volumes after retention
>> period? Please suggest a better configuration.
>>
>> c) Do I need a restore job for each client? I would just like to
>> restore backup on the same client, default to /restore folder... When
>> I use bconsole restore all command, the wizard asks me all the
>> questions (eg. 5- last backup for a client, which client,fileset...)
>> but at the end it asks for a restore job which changes all previously
>> defined things (eg. client).
>>
>> d) At the moment, I have not implemented autochanger functionality.
>> Clients compress/encrypt the data and send them to bacula server,
>> which writes them on one central storage system. Jobs are processed in
>> sequential order (one at a time). Do you expect any significant
>> performance gain if i implement autochanger in order to have jobs run
>> simultaneously?
>>
>> Relevant part of configuration attached bellow.
>>
>> Looking forward to move in the production...
>> Kind regards,
>> Ziga Zvan
>>
>>
>> *Volume example *(fileparts 1-23 should be deleted)*:*
Re: [Bacula-users] Bacula 9.6.5 TLS issue - solved in 9.6.6
On 2020-10-06 02:07, Phil Stracchino wrote: On 9/28/20 12:33 PM, Phil Stracchino wrote: test phase 1: All clients and Storage on 9.6.6, Director still on 9.6.3 No hung jobs so far. I plan to leave it this way for at least a week before upgrading the Director to 9.6.6. as well. OK, a week of no issues and monthly full backups just ran. I am now updating the Director from 9.6.3 to 9.6.6. No other changes. Fingers crossed. I am eager to hear about your conclusions. Good luck. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] performance&design&configuration challenges
I believe that I have my spooling attributes set correctly on jobdefs
(see bellow). Spool attributes = yes; Spool data defaults to no. Any
other idea for performance problems?
Regard,
Ziga
JobDefs {
Name = "bazar2-job"
Type = Backup
Level = Incremental
Client = bazar2.kranj.cetrtapot.si-fd #Client names: will be match on
bacula-fd.conf on client side
FileSet = "bazar2-fileset"
Schedule = "WeeklyCycle" #schedule : see in bacula-dir.conf
Storage = FSTestBackup
# Storage = FSOciCloudStandard
Messages = Standard
Pool = bazar2-daily-pool
Spool Attributes = yes # Better for backup to disk
Max Full Interval = 15 days # Ensure that full backup exist
Priority = 10
Write Bootstrap = "/opt/bacula/working/%c.bsr"
}
status dir not showing files transfered:
JobId Type Level Files Bytes Name Status
==
714 Back Full 0 0 bazar2-monthly-backup is running
On 06.10.2020 09:14, Žiga Žvan wrote:
Thanks Josh for your reply and sorry for my previous duplicate email.
I will try to disable data spooling and report back the results.
What about manipulating retention? Currently I have different jobs for
weekly full and monthly full backup (see bellow), but that triggers
full backup instead of incremental on Monday (because I use different
job resource). Is there a better way to have monthly backup with
longer retention?
Kind regards,
Ziga
#For all clients
Schedule {
Name = "MonthlyFull"
Run = Full 1st fri at 23:05
}
# This schedule does the catalog. It starts after the WeeklyCycle
Schedule {
Name = "WeeklyCycleAfterBackup"
Run = Full sun-sat at 23:10
}
#Job for each client
Job {
Name = "oradev02-backup"
JobDefs = "oradev02-job"
Full Backup Pool = oradev02-weekly-pool
Incremental Backup Pool = oradev02-daily-pool
}
Job {
Name = "oradev02-monthly-backup"
JobDefs = "oradev02-job"
Pool = oradev02-monthly-pool
Schedule = "MonthlyFull" #schedule : see in bacula-dir.conf
(monthly pool with longer retention)
}
On 05.10.2020 16:30, Josh Fisher wrote:
On 10/5/20 9:20 AM, Žiga Žvan wrote:
Hi,
I'm having some performance challenges. I would appreciate some
educated guess from an experienced bacula user.
I'm changing old backup sw that writes to tape drive with bacula
writing to disk. The results are:
a) windows file server backup from a deduplicated drive (1.700.000
files, 900 GB data, deduplicated space used 600 GB). *Bacula: 12
hours, old software: 2.5 hours*
b) linux file server backup (50.000 files, 166 GB data).*Bacula 3.5
hours, old software: 1 hour*.
I have tried to:
a) turn off compression&encryption. The result is the same: backup
speed around 13 MB/sec.
b) change destination storage (from a new ibm storage attached over
nfs, to a local SSD disk attached on bacula server virtual machine).
It took 2 hours 50 minutes to backup linux file server (instead of
3.5 hours). Sequential write test tested with linux dd command shows
write speed 300 MB/sec for IBM storage and 600 MB/sec for local SSD
storage (far better than actual throughput).
There are directives to enable/disable spooling of both data and the
attributes (metadata) being written to the catalog database. When
using disk volumes, you probably want to disable data spooling and
enable attribute spooling. The attribute spooling will prevent a
database write after each file backed up and instead do the database
writes as a batch at the end of the job. Data spooling would rarely
if ever be needed when writing to dick media.
With attribute spooling enabled, you can make a rough guess as to
whether DB performance is the problem by judging how long the job is
in the 'attribute despooling' state, The status dir command in
bconsole shows the job state.
The network bandwidth is 1 GB (1 GB on client, 10 GB on server) so I
guess this is not a problem; however I have noticed that bacula-fd
on client side uses 100% of CPU.
I'm using:
-bacula server version 9.6.5
-bacula client version 5.2.13 (original from centos 6 repo).
Any idea what is wrong and/or what performance should I expect?
I would also appreciate some answers on the questions bellow (I
think this email went unanswered).
Kind regards,
Ziga Zvan
On 05.08.2020 10:52, Žiga Žvan wrote:
Dear all,
I have tested bacula sw (9.6.5) and I must say I'm quite happy with
the results (eg. compression, encryption, configureability).
However I have some configuration/design questions I hope, you can
help me with.
Regarding job schedule, I would like to:
- create incremental daily backup (retention 1 week)
- create weekly full backup (retention 1 month)
- create monthly full backup (retention 1 year)
I am using dummy cloud driver that writes to local file storage.
Volume is a directory with fileparts. I would like to have seperate
volumes/pools for each client. I wou
[Bacula-users] performance&design&configuration challenges
Hi,
I'm having some performance challenges. I would appreciate some educated
guess from an experienced bacula user.
I'm changing old backup sw that writes to tape drive with bacula
writing to disk. The results are:
a) windows file server backup from a deduplicated drive (1.700.000
files, 900 GB data, deduplicated space used 600 GB). *Bacula: 12 hours,
old software: 2.5 hours*
b) linux file server backup (50.000 files, 166 GB data).*Bacula 3.5
hours, old software: 1 hour*.
I have tried to:
a) turn off compression&encryption. The result is the same: backup speed
around 13 MB/sec.
b) change destination storage (from a new ibm storage attached over nfs,
to a local SSD disk attached on bacula server virtual machine). It took
2 hours 50 minutes to backup linux file server (instead of 3.5 hours).
Sequential write test tested with linux dd command shows write speed 300
MB/sec for IBM storage and 600 MB/sec for local SSD storage (far better
than actual throughput).
The network bandwidth is 1 GB (1 GB on client, 10 GB on server) so I
guess this is not a problem; however I have noticed that bacula-fd on
client side uses 100% of CPU.
I'm using:
-bacula server version 9.6.5
-bacula client version 5.2.13 (original from centos 6 repo).
Any idea what is wrong and/or what performance should I expect?
I would also appreciate some answers on the questions bellow.
Kind regards,
Ziga Zvan
On 05.08.2020 10:52, Žiga Žvan wrote:
Dear all,
I have tested bacula sw (9.6.5) and I must say I'm quite happy with
the results (eg. compression, encryption, configureability). However I
have some configuration/design questions I hope, you can help me with.
Regarding job schedule, I would like to:
- create incremental daily backup (retention 1 week)
- create weekly full backup (retention 1 month)
- create monthly full backup (retention 1 year)
I am using dummy cloud driver that writes to local file storage.
Volume is a directory with fileparts. I would like to have seperate
volumes/pools for each client. I would like to delete the data on disk
after retention period expires. If possible, I would like to delete
just the fileparts with expired backup.
Questions:
a) At the moment, I'm using two backup job definitions per client and
central schedule definition for all my clients. I have noticed that my
incremental job gets promoted to full after monthly backup ("No prior
Full backup Job record found"; because monthly backup is a seperate
job, but bacula searches for full backups inside the same job). Could
you please suggest a better configuration. If possible, I would like
to keep central schedule definition (If I manipulate pools in a
schedule resource, I would need to define them per client).
b) I would like to delete expired backups on disk (and in the catalog
as well). At the moment I'm using one volume in a daily/weekly/monthly
pool per client. In a volume, there are fileparts belonging to expired
backups (eg. part1-23 in the output bellow). I have tried to solve
this with purge/prune scripts in my BackupCatalog job (as suggested in
the whitepapers) but the data does not get deleted. Is there any way
to delete fileparts? Should I create separate volumes after retention
period? Please suggest a better configuration.
c) Do I need a restore job for each client? I would just like to
restore backup on the same client, default to /restore folder... When
I use bconsole restore all command, the wizard asks me all the
questions (eg. 5- last backup for a client, which client,fileset...)
but at the end it asks for a restore job which changes all previously
defined things (eg. client).
d) At the moment, I have not implemented autochanger functionality.
Clients compress/encrypt the data and send them to bacula server,
which writes them on one central storage system. Jobs are processed in
sequential order (one at a time). Do you expect any significant
performance gain if i implement autochanger in order to have jobs run
simultaneously?
Relevant part of configuration attached bellow.
Looking forward to move in the production...
Kind regards,
Ziga Zvan
*Volume example *(fileparts 1-23 should be deleted)*:*
[root@bacula cetrtapot-daily-vol-0022]# ls -ltr
total 0
-rw-r--r--. 1 bacula disk 262 Jul 28 23:05 part.1
-rw-r--r--. 1 bacula disk 35988 Jul 28 23:06 part.2
-rw-r--r--. 1 bacula disk 35992 Jul 28 23:07 part.3
-rw-r--r--. 1 bacula disk 36000 Jul 28 23:08 part.4
-rw-r--r--. 1 bacula disk 35981 Jul 28 23:09 part.5
-rw-r--r--. 1 bacula disk 328795126 Jul 28 23:10 part.6
-rw-r--r--. 1 bacula disk 35988 Jul 29 23:09 part.7
-rw-r--r--. 1 bacula disk 35995 Jul 29 23:10 part.8
-rw-r--r--. 1 bacula disk 35981 Jul 29 23:11 part.9
-rw-r--r--. 1 bacula disk 35992 Jul 29 23:12 part.10
-rw-r--r--. 1 bacula disk 453070890 Jul 29 23:12 part.11
-rw-r--r--. 1 bacula disk 35995 Jul 30 23:09 part.12
-rw-r--r--. 1 bacula disk 35993 Jul 30 23:10 part.13
