Re: [Bacula-users] SPEED!
Le Mer 6 juillet 2011 18:43, John Drescher a écrit : 2011/7/6 Jake Debord jake.deb...@gmail.com: I have a machine I back up that when done averages: Elapsed time: 41 mins 47 secs Priority: 1 FD Files Written: 6,948 SD Files Written: 6,948 FD Bytes Written: 14,587,852,350 (14.58 GB) SD Bytes Written: 14,589,273,339 (14.58 GB) Rate: 5818.8 KB/s Software Compression: 11.7 % Is this acceptable??? 6Mbps seems slow. I backup my machine and achieve a little better results Elapsed time: 3 mins 51 secs Priority: 1 FD Files Written: 665 SD Files Written: 665 FD Bytes Written: 2,192,593,865 (2.192 GB) SD Bytes Written: 2,192,728,783 (2.192 GB) Rate: 9491.7 KB/s Software Compression: 9.8 % Both of our Machines are almost identical in specs. I'm just wondering if this is typical or if there are tweeks to speeding things up. My setup is basically out of the box so not much extra done to it. I also use mysql for the database. Are you using disk based volumes? If so try this with compression turned off. Also A Full backup will have a much higher rate than an Incremental or Differential because more of the time will be spent looking for the files to backup instead of backing up every file. Fragmentation of the client disk also plays a large part in backup rates. John you could try an iperf for each machine o be sure all is ok with network -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Wednesday 30 March 2011 15:41:29 Alan Brown, vous avez écrit : Laurent HENRY wrote: I let bacula a few days with attribute spooling. Things are not really different but it is a nice feature anyway. I am still not beyond 100Mb/s on my 1Gb/s network. You should be able to hit 300-400Mb/s assuming you are spooling to a mechanical hard drive or cheap SSD. Faster than that requires an array of disks setup for striping. Well, i am spooling to a server with Nearline SAS disks (10k) on RAID1/RAID 5 and a huge iSCSI disk area. I already thought the bottleneck could be the client, which i don't know how to improve. What makes me trying to do something on the server is, with other backups with differents jobs are not above 100Mb/s either On this 2nd simple configuration: Client with a dedicated Gb interface on a private VLAN Server with a dedicated Gb interface on the same private vlan. Both directly connected through a Cisco catalyst 6500 Backups on a LTO-4 tape drive directly connected via SAS on the bacula server. Trying a iperf between both machines gives me a Gb speed as expected: # /usr/bin/iperf -s -i 2 Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) [ ID] Interval Transfer Bandwidth [ 4] 0.0- 2.0 sec 224 MBytes 942 Mbits/sec [ 4] 2.0- 4.0 sec 224 MBytes 941 Mbits/sec [ 4] 4.0- 6.0 sec 224 MBytes 941 Mbits/sec [ 4] 6.0- 8.0 sec 224 MBytes 941 Mbits/sec [ 4] 8.0-10.0 sec 224 MBytes 941 Mbits/sec [ 4] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec During last backup i had few peaks at 110 Mb/s, nothing more. Average trafic is more about 30Mb/s Did someone tried to tweak some network parameters (On linux Debian) ? I think about Jumbo frames, but because it is a vlan-wide parameter and bacula is not the same on this network, it is a little complicated to deploy. I have some settings but they won't gain much until other areas are addressed. Before you attempt to do anything, you will need to benchmarking network speeds and speeds of simple file transfers, etc. You need to get a handle on how fast your network and disk susbsystems are before you can see how fast Bacula can run. In my experience throughputs are far more limited by client disk speed/loading (busy clients can be very slow to provide data), server disk speeds/loading and finally by network limitations - in that order. Tuning networking when the bottlenecks are elsewhere will give few noticeable improvements until those other areas are quantified and addressed if needed. I can easily sustain 1Gb/s on terabyte-scale backups without jumbo packets, but it took a bit of tuning of the client's disk handling to achieve it and I have hardware arrays with 96 drives onboard apiece. The tuning was mainly intended to improve its performance as a fileserver and came at the expense of its usefulness as an interactive machine (it's a dedicated fileserver) - there's no such thing as a free lunch. -- Laurent HENRY Administrateur Systèmes Réseaux Responsable du CRI RSSI EHESS - CRI 190 Av de France 75013 Paris Secrétariat du CRI: 01 49 54 23 08 Tel: 01 49 54 23 61 -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Tuesday 15 March 2011 13:40:10 John Drescher, vous avez écrit : Fine, i understand it now. Is there a way to just spool attributes and not backuped files themselwes ? SpoolAttributes=yes in the Job resource. http://www.bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html# SECTION00183 Thank you. I let bacula a few days with attribute spooling. Things are not really different but it is a nice feature anyway. I am still not beyond 100Mb/s on my 1Gb/s network. Did someone tried to tweak some network parameters (On linux Debian) ? I think about Jumbo frames, but because it is a vlan-wide parameter and bacula is not the same on this network, it is a little complicated to deploy. -- Enable your software for Intel(R) Active Management Technology to meet the growing manageability and security demands of your customers. Businesses are taking advantage of Intel(R) vPro (TM) technology - will your software be a part of the solution? Download the Intel(R) Manageability Checker today! http://p.sf.net/sfu/intel-dev2devmar ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Friday 11 March 2011 15:08:45 John Drescher, vous avez écrit : Attribute spooling makes sense for either though. Attribute spooling ? That enables spooling if the database entries at the end of the job in a batch instead of 1 at a time as the files are being processed. Its quite a bit faster to enable attribute spooling unless you have your database on an array of SSDs. Fine, i understand it now. Is there a way to just spool attributes and not backuped files themselwes ? -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Wednesday 09 March 2011 16:28:52, vous avez écrit : I tried to enable spooling (500 Gb spool) for the tape backup. The perf were worse. BTW, I use 5GB spool. I assume then you do not run concurrent jobs? For a single job the time will be longer because bacula does not concurrently spool and despool but with concurrent jobs and spooling with a few GB spool size bacula will maximize the performance of the tape drive when spooling is enabled. I get despool rates of 20 to 50MB/s for LTO2 which is about what the tape drive can handle. The rates vary because of compression. Yes, i use just one job at a time. For disk backup, it seems to me spooling does not make sense. Attribute spooling makes sense for either though. Attribute spooling ? -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Thursday 10 March 2011 01:32:20 Mike Hobbs, vous avez écrit : On 3/1/2011 8:40 AM, Laurent HENRY wrote: Is anyone know if there is a way to improve performances of network usage ? Bacula - Turn off software compression. already done Switch and Network Cards - If you are fully gigabit and your equipment supports it, turn on jumbo frames. i think about it but still thinking about a bacula sofware solution -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Tuesday 08 March 2011 14:58:18 John Drescher, vous avez écrit : On Tue, Mar 8, 2011 at 8:24 AM, Laurent HENRY laurent.he...@ehess.fr wrote: Le Wednesday 02 March 2011 20:50:18 Mark, vous avez écrit : btw, i actually experience it with disk backups. Does this mean software compression limits deeply backup speed ? In my case, software compression was a _massive_ slowdown, to the point that my preference was to throw more disk space to bacula to keep my jobs completing in a reasonable time. This was backing up a server with a couple hundred gigs of large files, not tons of small files. Try your job without compression enabled at least once, to get yourself a good comparison. Mark Turning compression off on disk backup definitely improve the performance. I was at 17MB/s and now i am at 30 MB/s. On my network graphs i still see i don't often go beyond the 100Mb/s i don't find any explanation on the network side. Do you have attribute spooling enabled? If not enable that. John I tried to enable spooling (500 Gb spool) for the tape backup. The perf were worse. For disk backup, it seems to me spooling does not make sense. -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Wednesday 02 March 2011 20:50:18 Mark, vous avez écrit : btw, i actually experience it with disk backups. Does this mean software compression limits deeply backup speed ? In my case, software compression was a _massive_ slowdown, to the point that my preference was to throw more disk space to bacula to keep my jobs completing in a reasonable time. This was backing up a server with a couple hundred gigs of large files, not tons of small files. Try your job without compression enabled at least once, to get yourself a good comparison. Mark Turning compression off on disk backup definitely improve the performance. I was at 17MB/s and now i am at 30 MB/s. On my network graphs i still see i don't often go beyond the 100Mb/s i don't find any explanation on the network side. -- What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Wednesday 02 March 2011 02:24:16 John Drescher, vous avez écrit : On Tue, Mar 1, 2011 at 8:40 AM, Laurent HENRY laurent.he...@ehess.fr wrote: Hi all, Maybe not a direct question about bacula but i am looking for some feedback of bacula users. My Debian 5 bacula server is directly connected to a Cisco 6500 Switch. Either the bacula server, the switch, the servers backuped are using Gb interfaces. Looking a bandwith usage, i barely use 100Mb. i see bursts at 110 or 130 Mb/s and nothing more. I know speed depends a lot of files backuped (big ones, a lot of short ones, etc) but this really less that i need to backup huge volumes. Is anyone know if there is a way to improve performances of network usage ? Turn software compression off. Backup to tape with software compression off. John btw, i actually experience it with disk backups. Does this mean software compression limits deeply backup speed ? I experience this low network usage behavior with tape backup without software compression and with disk backups but with a very low compression ratio (less than 5%) -- Free Software Download: Index, Search Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula bandwith usage
Le Wednesday 02 March 2011 11:59:06 Alan Brown, vous avez écrit : On 02/03/11 01:24, John Drescher wrote: Is anyone know if there is a way to improve performances of network usage ? Turn software compression off. Backup to tape with software compression off. Turn spooling on. Spooling works on the bacula server side (and seems to be really usefull only with tape backups not with disks backups). Do think a local cause on the end server could slow down de backuped client sending datas ? -- Free Software Download: Index, Search Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] how to access a file backup manually
Le Wednesday 02 February 2011 02:09:24 Dan Langille, vous avez écrit : On 2/1/2011 11:34 AM, Laurent HENRY wrote: Hello For some reason i don't know yet, there are files in my catalog which are not restoring. How have you established this? How do you know the files are in your Catalog? Well, i was in a hurry to explain this yesterday. I still possess the backup file (i did not recycled volumes), but the catalog have been pruned, so i can't use the usual bconsole procedure. I read the link about volume utility tool but i feared doing something with my catalog doing what is proposed importing file in my catalog. My backup was a file backup and i have some old backups files still in my possession but purged from the catalog. Hmmm, perhaps you mean 'there are files in my backups'. I read somewhere, but i don't remember where, we can access an old tape direclty without using bconsole tools. I guess a backup file is not a regular tar file. Does anyone know how to do something similar to a tar xvf with a bacula backup file ? The link Craig gave you is a good start. However, if you can give us some background information about your situation, perhaps we can help. Thank you, i will try this -- Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! Finally, a world-class log management solution at an even better price-free! Download using promo code Free_Logger_4_Dev2Dev. Offer expires February 28th, so secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsight-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] how to access a file backup manually
Hello For some reason i don't know yet, there are files in my catalog which are not restoring. My backup was a file backup and i have some old backups files still in my possession but purged from the catalog. I read somewhere, but i don't remember where, we can access an old tape direclty without using bconsole tools. I guess a backup file is not a regular tar file. Does anyone know how to do something similar to a tar xvf with a bacula backup file ? -- Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! Finally, a world-class log management solution at an even better price-free! Download using promo code Free_Logger_4_Dev2Dev. Offer expires February 28th, so secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsight-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula confused about slots on autochanger
Hi, no clever idea for me, just basic thoughts. what is the output of an update slots command ? what happen when you ask a 'mount' request from slot 3 ? Le Mer 4 août 2010 03:20, Maria McKinley a écrit : It seems that part of the problem I have been having with backup is that bacula is having communication problems with the autochanger. Bacula tells the autochanger to load slot 3, but instead slot 2 is loaded: *messages 03-Aug 18:14 billie-sd JobId 1728: 3301 Issuing autochanger loaded? drive 0 command. 03-Aug 18:14 billie-sd JobId 1728: 3302 Autochanger loaded? drive 0, result: nothing loaded. 03-Aug 18:14 billie-sd JobId 1728: 3304 Issuing autochanger load slot 3, drive 0 command. * 03-Aug 18:15 billie-sd JobId 1728: 3305 Autochanger load slot 3, drive 0, status is OK. * 03-Aug 18:15 billie-sd JobId 1728: Warning: Director wanted Volume A005. Current Volume B005 not acceptable because: 1998 Volume B005 status is Full, not in Pool. 03-Aug 18:15 billie-sd JobId 1728: Please mount Volume A005 or label a new one for: Job: NightlySave.2010-08-01_01.05.00.32 Storage: Drive-1 (/dev/nst0) Pool: Daily Media type: VXA-2 *exit billie:~# mtx -f /dev/sg2 status Storage Changer /dev/sg2:1 Drives, 10 Slots ( 0 Import/Export ) Data Transfer Element 0:Full (Storage Element 2 Loaded):VolumeTag = B005 Storage Element 1:Full :VolumeTag=A006 Storage Element 2:Empty:VolumeTag= Storage Element 3:Full :VolumeTag=A005 Storage Element 4:Full :VolumeTag=A004 Storage Element 5:Full :VolumeTag=B004 Storage Element 6:Full :VolumeTag=C005 Storage Element 7:Full :VolumeTag=A007 Storage Element 8:Full :VolumeTag=C006 Storage Element 9:Full :VolumeTag=A010 Storage Element 10:Full :VolumeTag=CLNA0001 Any ideas what is going on or how to fix it? thanks, maria -- The Palm PDK Hot Apps Program offers developers who use the Plug-In Development Kit to bring their C/C++ apps to Palm for a share of $1 Million in cash or HP Products. Visit us here for more details: http://p.sf.net/sfu/dev2dev-palm ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- The Palm PDK Hot Apps Program offers developers who use the Plug-In Development Kit to bring their C/C++ apps to Palm for a share of $1 Million in cash or HP Products. Visit us here for more details: http://p.sf.net/sfu/dev2dev-palm ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] About the hardware support : PowerVault TL2000 or TL4000
Dell TL2000 works perfectly. I operate one, with direct SAS connexion under Debian. Le Sam 24 juillet 2010 11:35, yesi a écrit : Hi, I'd like to know whether the Dell PowerVault TL2000 or TL4000 are supported by Bacula ? Thank you a lot for your return. Regards, yesi -- This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] full backups on tape and incremental on files ?
Hi, Is it possible to have a backup job doing full backups on tapes and incremental on file ? I would like to keep my full backups on tape for archiving once a week. I don't want to keep my daily incrementals there to save room on tapes but i need them. I was thinking create a pool of 6 files volumes recycling every 7 days. Thank you PS: I am actually using bacula 2.4.4-1 with Debian 5.0 -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula backuping on wrong network interface
Hello,
SD, FD, and networks are fine.
I think i am getting it !
Putting the private adresse in the client config is not enough.
You have to put the private FQDN of the bacula server in the Storage config
for the client coming back on the good NIC
Storage {
Name = Autochangeur
# Do not use localhost here
Address = bacula.private.ehess.fr# N.B. Use a fully
qualified name here
SDPort = 9103
...
Thank you for your help :)
Le Mar 16 mars 2010 16:34, Carlo Filippetto a écrit :
Enable the firewall, so only the internal IP is allowed to accede that
server on those ports
Ciao
---
Carlo Filippetto
2010/3/16 RadosÅaw Korzeniewski rados...@korzeniewski.net:
Hello,
Check if SD is listening on private network address, using telnet, for
example - check from client (FD) machine. Next, check if Bacula SD
configuration uses private address (on private network) instead of
public.
And finally check communication from FD to SD using traceroute and
private
SD address...
regards
Radek
2010/3/16 Laurent HENRY (EHESS/CRI) laurent.he...@ehess.fr
Hello all,
   I have a problem to understandhow to configure bacula to make
backups on different network interfaces.
My config:
- backuped server (server1) with 2 network cards, one for internet
traffic
(public IP), one for backups (private IP).
- bacula server (2.4 on debian) with 2 network cards. One private IP
for
backups and one public IP for other backups elsewhere.
In my bacula-dir config
Client {
 Name = server1-fd
 Address = 10.2.0.21
 FDPort = 9102
...
}
I don't know how but while doing tcpdump during backups, i am noticing
bacula network traffic is between my 2 public interfaces and there is
no
flow at all on my privates cards.
Could anyone know where i can take a look ?
I've never configured the public IP of the backuped server i don't
understand how bacula find it (and that is not what i want him to do)
Thank you for any clue.
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
RadosÅaw Korzeniewski
rados...@korzeniewski.net
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bacula backuping on wrong network interface
Hello all,
I have a problem to understandhow to configure bacula to make
backups on different network interfaces.
My config:
- backuped server (server1) with 2 network cards, one for internet traffic
(public IP), one for backups (private IP).
- bacula server (2.4 on debian) with 2 network cards. One private IP for
backups and one public IP for other backups elsewhere.
In my bacula-dir config
Client {
Name = server1-fd
Address = 10.2.0.21
FDPort = 9102
...
}
I don't know how but while doing tcpdump during backups, i am noticing
bacula network traffic is between my 2 public interfaces and there is no
flow at all on my privates cards.
Could anyone know where i can take a look ?
I've never configured the public IP of the backuped server i don't
understand how bacula find it (and that is not what i want him to do)
Thank you for any clue.
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
