[Bacula-users] Should I use data spooling when writing to nfs mounted storage?
I have my director using an external raid device which exports its filesystem with nfs protocol as storage. I just ended my bacula configuration and just launched my first remote job. I have this in the report JobId 3: Job write elapsed time = 00:14:23, Transfer rate = 1.182 M Bytes/second which seems a little poor to me (our network is currently on a 100M switch, we're planning to upgrade with something better in the near future). Currently data flows from bacula-fd on client to bacula-sd on bacula server and then the storage daemon writes data on a volume mounted in an nfs partition, so data goes on the wire at the same time. Is advisable to enable data spooling in the storage daemon in this situation? Thanks. -- Fabio Napoleoni - ZENIT fa...@zenit.org "Computer Science is no more about computers than astronomy is about telescopes" Edsger W. Dijkstra -- Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
Il giorno 03/mar/2011, alle ore 04.52, Fabio Napoleoni - ZENIT ha scritto: > I have my director using an external raid device which exports its filesystem > with nfs protocol as storage. I just ended my bacula configuration and just > launched my first remote job. I have this in the report > > JobId 3: Job write elapsed time = 00:14:23, Transfer rate = 1.182 M > Bytes/second > > which seems a little poor to me (our network is currently on a 100M switch, > we're planning to upgrade with something better in the near future). > Currently data flows from bacula-fd on client to bacula-sd on bacula server > and then the storage daemon writes data on a volume mounted in an nfs > partition, so data goes on the wire at the same time. > > Is advisable to enable data spooling in the storage daemon in this situation? I made a try with this results JobId 7: Spooling data ... JobId 7: Job write elapsed time = 00:13:07, Transfer rate = 1.295 M Bytes/second JobId 7: Committing spooled data to Volume "FullVolume-0004". Despooling 1,021,072,888 bytes ... JobId 7: Despooling elapsed time = 00:02:19, Transfer rate = 7.345 M Bytes/second JobId 7: Sending spooled attrs to the Director. Despooling 6,816,214 bytes ... There is only a little improvement of performances. So I'm asking you what is the transfer rate in your jobs? These rates are normal or are they slow compared to yours? Thanks. -- Fabio Napoleoni - ZENIT fa...@zenit.org "Computer Science is no more about computers than astronomy is about telescopes" Edsger W. Dijkstra -- Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
On Thu, Mar 3, 2011 at 9:52 AM, Fabio Napoleoni - ZENIT wrote: > > Il giorno 03/mar/2011, alle ore 04.52, Fabio Napoleoni - ZENIT ha scritto: > >> I have my director using an external raid device which exports its >> filesystem with nfs protocol as storage. I just ended my bacula >> configuration and just launched my first remote job. I have this in the >> report >> >> JobId 3: Job write elapsed time = 00:14:23, Transfer rate = 1.182 M >> Bytes/second >> >> which seems a little poor to me (our network is currently on a 100M switch, >> we're planning to upgrade with something better in the near future). >> Currently data flows from bacula-fd on client to bacula-sd on bacula server >> and then the storage daemon writes data on a volume mounted in an nfs >> partition, so data goes on the wire at the same time. >> >> Is advisable to enable data spooling in the storage daemon in this situation? > > I made a try with this results > > JobId 7: Spooling data ... > JobId 7: Job write elapsed time = 00:13:07, Transfer rate = 1.295 M > Bytes/second > JobId 7: Committing spooled data to Volume "FullVolume-0004". Despooling > 1,021,072,888 bytes ... > JobId 7: Despooling elapsed time = 00:02:19, Transfer rate = 7.345 M > Bytes/second > JobId 7: Sending spooled attrs to the Director. Despooling 6,816,214 bytes ... > > There is only a little improvement of performances. > > So I'm asking you what is the transfer rate in your jobs? These rates are > normal or are they slow compared to yours? > I use gigabit and LTO2 so I am seeing 20 to 50 MB/s per tape drive in my 2 drive LTO2 autochanger. John -- Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
John Drescher wrote: > I made a try with this results >> >> JobId 7: Spooling data ... >> JobId 7: Job write elapsed time = 00:13:07, Transfer rate = 1.295 M >> Bytes/second >> JobId 7: Committing spooled data to Volume "FullVolume-0004". Despooling >> 1,021,072,888 bytes ... >> JobId 7: Despooling elapsed time = 00:02:19, Transfer rate = 7.345 M >> Bytes/second >> JobId 7: Sending spooled attrs to the Director. Despooling 6,816,214 bytes >> ... >> >> There is only a little improvement of performances. Spooling or not spooling is very much a case of "it depends". There is little point in using spooling if the destination device is a disk drive and you are not using concurrent jobs.(*) For a NFS mounted destination network delays are your biggest problem. Make sure rsize and wsize are the maximum possible (32768 bytes on NFSv3, several Mb on NFSv4) If you have 1Gb/s ethernet, use jumbo frames. 7-8Mb/s is about right for untuned NFS on a 100Mb/s network environment. Because of NFS overheads you won't go much faster even though the network theoretical maximum is 12.5Mb/s (*) If you are using concurrent jobs and writing output to disk, spooling will usually help maintain speeds, because writing multiple files to the target disk will result in slowdowns due to head seeking. A single job despooling is more akin to streaming throughput. >> So I'm asking you what is the transfer rate in your jobs? These rates are >> normal or are they slow compared to yours? >> > > I use gigabit and LTO2 so I am seeing 20 to 50 MB/s per tape drive in > my 2 drive LTO2 autochanger. Spooling is almost always worthwhile if the target is tape. Spooling for differential/incremental jobs won't make much difference to the overall job rate but it WILL stop the tape drive "shoeshining" as it tries to write small amounts of data. That saves a lot of wear-and-tear. Spooling for full jobs is a win as long as your spool disks are fast enough to keep up with the tape drive (or at least, faster than what's coming off the backup filesystem). Spooling is _essential_ if you're running concurrent jobs, otherwise the jobs get interleaved on tape and restores will run incredibly slowly because the drive has to seek back and forth to get data in 64kB chunks. Concurrent jobs slow down the effective read rate when despooling to tape due to head seeking penalties. A single 7200rpm 320Gb HDD can _just_ despool fast enough to LTO2 to keep its buffers full whilst having another job simultaneously spooling data. A single 7200rpm 320Gb drive _cannot_ despool fast enough to LTO3 (or higher levels), even when despooling is the only activity on the disk. A 4 disk 7200rpm RAID0 stripe can just keep up with LTO4 whilst spooling another job. I use a set of Intel X25E SSDs in RAID-0 layout for the spool area. They are slightly faster in streaming output than 7200rpm drives, but the head seek penalty reduces from avg ~8ms down to under 0.1ms. I can concurrently despool to 3 LTO5 drives whilst spooling 3 more jobs. -- Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
On 03/03/11 09:52, Fabio Napoleoni - ZENIT wrote: > > Il giorno 03/mar/2011, alle ore 04.52, Fabio Napoleoni - ZENIT ha scritto: > >> I have my director using an external raid device which exports its >> filesystem with nfs protocol as storage. I just ended my bacula >> configuration and just launched my first remote job. I have this in the >> report >> >> JobId 3: Job write elapsed time = 00:14:23, Transfer rate = 1.182 M >> Bytes/second >> >> which seems a little poor to me (our network is currently on a 100M switch, >> we're planning to upgrade with something better in the near future). >> Currently data flows from bacula-fd on client to bacula-sd on bacula server >> and then the storage daemon writes data on a volume mounted in an nfs >> partition, so data goes on the wire at the same time. >> >> Is advisable to enable data spooling in the storage daemon in this situation? > > I made a try with this results > > JobId 7: Spooling data ... > JobId 7: Job write elapsed time = 00:13:07, Transfer rate = 1.295 M > Bytes/second > JobId 7: Committing spooled data to Volume "FullVolume-0004". Despooling > 1,021,072,888 bytes ... > JobId 7: Despooling elapsed time = 00:02:19, Transfer rate = 7.345 M > Bytes/second > JobId 7: Sending spooled attrs to the Director. Despooling 6,816,214 bytes ... > > There is only a little improvement of performances. > > So I'm asking you what is the transfer rate in your jobs? These rates are > normal or are they slow compared to yours? Is there any way you can run your SD directly on the storage device, instead of the SD writing to storage mounted over NFS? -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, SQL wrangler, Free Stater It's not the years, it's the mileage. -- Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
On 3/3/2011 6:52 AM, Fabio Napoleoni - ZENIT wrote: > > JobId 7: Spooling data ... > JobId 7: Job write elapsed time = 00:13:07, Transfer rate = 1.295 M > Bytes/second > JobId 7: Committing spooled data to Volume "FullVolume-0004". Despooling > 1,021,072,888 bytes ... > JobId 7: Despooling elapsed time = 00:02:19, Transfer rate = 7.345 M > Bytes/second > JobId 7: Sending spooled attrs to the Director. Despooling 6,816,214 bytes ... > > There is only a little improvement of performances. > > So I'm asking you what is the transfer rate in your jobs? These rates are > normal or are they slow compared to yours? > > Thanks. > > -- > Fabio Napoleoni - ZENIT > fa...@zenit.org > Given your environment, the despool rate is probably about what you would expect. On the other hand, the job write rate is really quite low--assuming this was a full job when it was the only job running. If you are using compression, then that can explain a lot. Also, I don't think you mentioned whether the job was on a remote client or not--if it was, then 8-10Mbytes/sec would be about the most you could expect on a 100Mbit switched network. When I run only a single job at a time, I'm able to spool the local client at 50Mbytes/sec or more, and remote clients spool at ~20-30Mbytes/sec (usually running 2-3 at once on a gigabit network)--but I don't use compression or encryption. Is the server fairly busy with other stuff, or is the server machine underpowered? Look at spooling attributes even if you don't spool data, that may help. Turn off compression (if you use it), or lower the level, unless you need it (with disk storage, you may want to have it). For a 100Mbit network, remote clients are never going to be fast enough to drive multiple spindles of spool (i.e. some kind of RAID), but you should be able to do quite a bit better than what you are seeing (unless you need compression). -se -- Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
Il giorno 03/mar/2011, alle ore 17.58, Alan Brown ha scritto:
> John Drescher wrote:
>
>> I made a try with this results
>>>
>>> JobId 7: Spooling data ...
>>> JobId 7: Job write elapsed time = 00:13:07, Transfer rate = 1.295 M
>>> Bytes/second
>>> JobId 7: Committing spooled data to Volume "FullVolume-0004". Despooling
>>> 1,021,072,888 bytes ...
>>> JobId 7: Despooling elapsed time = 00:02:19, Transfer rate = 7.345 M
>>> Bytes/second
>>> JobId 7: Sending spooled attrs to the Director. Despooling 6,816,214 bytes
>>> ...
>>>
>>> There is only a little improvement of performances.
>
> Spooling or not spooling is very much a case of "it depends".
>
> There is little point in using spooling if the destination device is a disk
> drive and you are not using concurrent jobs.(*)
>
> For a NFS mounted destination network delays are your biggest problem.
Thank you for your analysis, after that I think that the problem is not the nfs
overhead, because the despooling phase (over nfs filsystem) has a transfer rate
of 7.3 MBps, so it's fine. Instead the first phase (bacula-fd -> bacula-sd)
happens at 1.2 MBps which is very poor value I think.
So the bottleneck should be the client configuration or something similar. What
I should check to improve performances?
This is the fileset used for that backup
FileSet {
Name = "FileSystem Full"
Include {
Options {
compression=GZIP # compress backup
signature = MD5 # store md5 of saved files
onefs = no # remember to exclude nfs directory: default no
accurate = mcs # attributes to consider when examining changes
verify = pins5 # attributes to consider when verifying files
noatime = yes# don't update file access time
}
# whole filesystem
File = /
}
Exclude {
# use this syntax to exclude files coming from a client file
# The given file MUST exist on the client otherwise backup will fail
File = "\\ (*) If you are using concurrent jobs and writing output to disk, spooling
> will usually help maintain speeds, because writing multiple files to the
> target disk will result in slowdowns due to head seeking. A single job
> despooling is more akin to streaming throughput.
Ok, currently I haven't tried concurrent jobs, but I'll maintain spooling
configuration when I'll do them.
--
Fabio Napoleoni - ZENIT
fa...@zenit.org
"Computer Science is no more about computers than astronomy is
about telescopes"
Edsger W. Dijkstra
--
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
Fabio Napoleoni - ZENIT wrote: > Thank you for your analysis, after that I think that the problem is not the > nfs overhead, because the despooling phase (over nfs filsystem) has a > transfer rate of 7.3 MBps, so it's fine. Instead the first phase (bacula-fd > -> bacula-sd) happens at 1.2 MBps which is very poor value I think. Is this a full backup? You can't get accurate results with anything else. compression=GZIP # compress backup This will have a MARKED effect on your speeds. Consider switching it off and see how fast things go. -- Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
(Dangit, that last reply was supposed to go to the list, not just the
sender of the email...let me try this again)
On 03/03/2011 12:40 PM, Fabio Napoleoni - ZENIT wrote:
> Thank you for your analysis, after that I think that the problem is not the
> nfs overhead, because the despooling phase (over nfs filsystem) has a
> transfer rate of 7.3 MBps, so it's fine. Instead the first phase (bacula-fd
> -> bacula-sd) happens at 1.2 MBps which is very poor value I think.
>
> So the bottleneck should be the client configuration or something similar.
> What I should check to improve performances?
>
> This is the fileset used for that backup
>
> FileSet {
> Name = "FileSystem Full"
> Include {
> Options {
> compression=GZIP # compress backup
[...]
In my experience, slow performance like this (i.e. <5MB/s on at least
100Mb ethernet) usually turns out to be the client's fault. Compression
seems to be a very common culprit. Try switching compression off
completely and see how much of a difference that makes. The latency
introduced by waiting for compression - even pretty fast compression -
seems to substantially choke throughput down.
If bacula ends up with additional compression type options in addition
to gzip at some point, this might help (LZO compression doesn't compress
as well as gzip, but seems to have a lot less overhead, for example),
but in the meantime if you're not backing up over a very slow link OR
you are not desperate for space on your backup media, you are better off
without the compression.
The other thing (that I'm guessing doesn't apply here) is on Windows
systems and some Macs that run antivirus software that does "on-access"
scanning. I've seen that bog down backups as well.
--
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
Il giorno 03/mar/2011, alle ore 20.53, Alan Brown ha scritto: > Fabio Napoleoni - ZENIT wrote: > >> Thank you for your analysis, after that I think that the problem is not the >> nfs overhead, because the despooling phase (over nfs filsystem) has a >> transfer rate of 7.3 MBps, so it's fine. Instead the first phase (bacula-fd >> -> bacula-sd) happens at 1.2 MBps which is very poor value I think. > > Is this a full backup? Yes it is. > You can't get accurate results with anything else. > > compression=GZIP # compress backup > > This will have a MARKED effect on your speeds. Consider switching it off and > see how fast things go. I tried to disable both compression and signature with this results: Job write elapsed time = 00:03:31, Transfer rate = 7.918 M Bytes/second Despooling elapsed time = 00:03:50, Transfer rate = 7.272 M Bytes/second This is the performances with signature only (no compression) Job write elapsed time = 00:03:42, Transfer rate = 7.528 M Bytes/second And this is the performances with compression only (no signature) Job write elapsed time = 00:12:51, Transfer rate = 1.322 M Bytes/second So the poor throughput is given by software compression. I don't know what to choose in the speed vs space tradeoff. Ideally the best option could be compression on the storage director (it has more powerful hardware than this client) but I think that bacula currently doesn't support this feature. Thanks for help me to solve this. -- Fabio Napoleoni - ZENIT fa...@zenit.org "Computer Science is no more about computers than astronomy is about telescopes" Edsger W. Dijkstra -- What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
On Thu, Mar 3, 2011 at 4:11 PM, Fabio Napoleoni - ZENIT wrote: > > Il giorno 03/mar/2011, alle ore 20.53, Alan Brown ha scritto: > >> Fabio Napoleoni - ZENIT wrote: >> >>> Thank you for your analysis, after that I think that the problem is not the >>> nfs overhead, because the despooling phase (over nfs filsystem) has a >>> transfer rate of 7.3 MBps, so it's fine. Instead the first phase (bacula-fd >>> -> bacula-sd) happens at 1.2 MBps which is very poor value I think. >> >> Is this a full backup? > > Yes it is. > >> You can't get accurate results with anything else. >> >> compression=GZIP # compress backup >> >> This will have a MARKED effect on your speeds. Consider switching it off and >> see how fast things go. > > I tried to disable both compression and signature with this results: > > Job write elapsed time = 00:03:31, Transfer rate = 7.918 M Bytes/second > Despooling elapsed time = 00:03:50, Transfer rate = 7.272 M Bytes/second > > This is the performances with signature only (no compression) > > Job write elapsed time = 00:03:42, Transfer rate = 7.528 M Bytes/second > > And this is the performances with compression only (no signature) > > Job write elapsed time = 00:12:51, Transfer rate = 1.322 M Bytes/second > > So the poor throughput is given by software compression. I don't know what to > choose in the speed vs space tradeoff. Ideally the best option could be > compression on the storage director (it has more powerful hardware than this > client) but I think that bacula currently doesn't support this feature. > You could do that with filesystem compression. There are a few options for that. Although I admit none of them are in the mainline kernel. John -- What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
On 03/03/2011 03:21 PM, John Drescher wrote: > On Thu, Mar 3, 2011 at 4:11 PM, Fabio Napoleoni - ZENIT > wrote: [...] >> So the poor throughput is given by software compression. I don't know what >> to choose in the speed vs space tradeoff. Ideally the best option could be >> compression on the storage director (it has more powerful hardware than this >> client) but I think that bacula currently doesn't support this feature. >> > You could do that with filesystem compression. There are a few options > for that. Although I admit none of them are in the mainline kernel. > > John Well, depending on what you consider "mainline". I'm currently experimenting with btrfs with filesystem compression. As of 2.6.38, this is available as both gzip/zlib and LZO. I am, daringly enough, experimenting with the 2.6.38 release candidates and LZO compression for a storage daemon on USB external disk volumes. It seems to work so far, but I've only been using it for a week or so and haven't had a chance to do any extensive checks. The one problem I see with filesystem-supplied compression is that trying not to run out of disk space for backup volumes becomes more difficult, since it's hard to predict exactly how much "real" space the compressed data will take up (whereas is bacula is doing the compression, you can just define e.g. 8 20GB volumes and stick them on a 160GB drive). This may or may not turn out to be a 'real' problem. It does at least take the compression burden off of the client computer, which removes that bottleneck. -- What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
On Thu, Mar 3, 2011 at 5:57 PM, Sean Clark wrote: > On 03/03/2011 03:21 PM, John Drescher wrote: >> On Thu, Mar 3, 2011 at 4:11 PM, Fabio Napoleoni - ZENIT >> wrote: > [...] >>> So the poor throughput is given by software compression. I don't know what >>> to choose in the speed vs space tradeoff. Ideally the best option could be >>> compression on the storage director (it has more powerful hardware than >>> this client) but I think that bacula currently doesn't support this feature. >>> >> You could do that with filesystem compression. There are a few options >> for that. Although I admit none of them are in the mainline kernel. >> >> John > Well, depending on what you consider "mainline". I'm currently > experimenting with btrfs with filesystem compression. As of 2.6.38, > this is available as both gzip/zlib and LZO. I am, daringly enough, > experimenting with the 2.6.38 release candidates and LZO compression for > a storage daemon on USB external disk volumes. It seems to work so far, > but I've only been using it for a week or so and haven't had a chance to > do any extensive checks. > That was one of the choices. btrfs is still considered highly experimental at this point. John -- What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
>> Well, depending on what you consider "mainline". What I meant by "mainline" was a filesystem that was considered stable in the mainline kernel. John -- What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
On 03/03/2011 05:12 PM, John Drescher wrote: > That was one of the choices. btrfs is still considered highly > experimental at this point. > > John Well, that's definitely true, though so far for me it's been more reliable than one might think. (I did, just yesterday, run into what appeared to be an odd bug in it, though, so not QUITE ready for everybody to use yet, just eager nuts like myself). If I can find an available gigabit port to plug the experimental storage daemon into I can do some real speed testing on the filesystem with bacula. -- What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
Sean Clark writes: > In my experience, slow performance like this (i.e. <5MB/s on at least > 100Mb ethernet) usually turns out to be the client's fault. > Compression seems to be a very common culprit. Try switching > compression off completely and see how much of a difference that > makes. The latency introduced by waiting for compression - even > pretty fast compression - seems to substantially choke throughput > down. the compression in bacula-fd is also running in a single thread, so you only exploit one cpu core. if your disks can handle it, consider splitting up the client job and run the jobs concurrently. or just turn off compression -- usually the data is kept in compressed form on the client anyway. -- Kjetil T. Homme Redpill Linpro AS - Changing the game -- What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Should I use data spooling when writing to nfs mounted storage?
Il 03/03/2011 18:05, Phil Stracchino ha scritto: > On 03/03/11 09:52, Fabio Napoleoni - ZENIT wrote: >> >> Il giorno 03/mar/2011, alle ore 04.52, Fabio Napoleoni - ZENIT ha scritto: >> >>> I have my director using an external raid device which exports its >>> filesystem with nfs protocol as storage. I just ended my bacula >>> configuration and just launched my first remote job. I have this in the >>> report >>> >>> JobId 3: Job write elapsed time = 00:14:23, Transfer rate = 1.182 M >>> Bytes/second >>> >>> which seems a little poor to me (our network is currently on a 100M switch, >>> we're planning to upgrade with something better in the near future). >>> Currently data flows from bacula-fd on client to bacula-sd on bacula server >>> and then the storage daemon writes data on a volume mounted in an nfs >>> partition, so data goes on the wire at the same time. >>> >>> Is advisable to enable data spooling in the storage daemon in this >>> situation? >> >> I made a try with this results >> >> JobId 7: Spooling data ... >> JobId 7: Job write elapsed time = 00:13:07, Transfer rate = 1.295 M >> Bytes/second >> JobId 7: Committing spooled data to Volume "FullVolume-0004". Despooling >> 1,021,072,888 bytes ... >> JobId 7: Despooling elapsed time = 00:02:19, Transfer rate = 7.345 M >> Bytes/second >> JobId 7: Sending spooled attrs to the Director. Despooling 6,816,214 bytes >> ... >> >> There is only a little improvement of performances. >> >> So I'm asking you what is the transfer rate in your jobs? These rates are >> normal or are they slow compared to yours? > > Is there any way you can run your SD directly on the storage device, > instead of the SD writing to storage mounted over NFS? > > Also, what about using a dedicated NIC to link SD with NFS storage ? That would avoid sharing bandwidth between nfs-traffic and client to storage daemon traffic. Just my 2 cents. -- Marcello Romani -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
