Re: [Bacula-users] compression with lto?
Josh Fisher via Bacula-users schrieb am 18.05.23 um 14:48: On 5/17/23 14:14, Phil Stracchino wrote: On 5/17/23 12:52, Marco Gaiarin wrote: I've LTO tape that work in very dusty environment by at least 6 years. Istead of dust, i've found that setting properly spooling and buffers prevent the spinup/spindown effect, that effectively can be very stressful... Yes, I went to great lengths to try to keep mine streaming and avoid shoe-shining, but with only moderate success. I have had many fewer problems, as well as much better performance, since I abandoned tape and went to disk-to-disk-to-removable-disk (with both of the destination disk stages being RAID). Full backup cycles that used to take 18 hours and two or three media changes, with about a 10% failure rate due to media errors, now take 3 or 4 hours with no media changes and nearly 100% success. However, we are getting further and further off the subject of compression. That approach actually affects both tape and disk. Software compression happens on the client, so performance greatly depends on the type of clients being backed up. For example, there may be NAS boxes with low power processors, Software compression will definitely slow the backup of such clients. Well, I do not see a big effect here. But I am running of a 64 CPU server with a bit of Ram ;-) At last I do not need to care about performance lost if it s running at night or weekends. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
On 5/17/23 14:14, Phil Stracchino wrote: On 5/17/23 12:52, Marco Gaiarin wrote: I've LTO tape that work in very dusty environment by at least 6 years. Istead of dust, i've found that setting properly spooling and buffers prevent the spinup/spindown effect, that effectively can be very stressful... Yes, I went to great lengths to try to keep mine streaming and avoid shoe-shining, but with only moderate success. I have had many fewer problems, as well as much better performance, since I abandoned tape and went to disk-to-disk-to-removable-disk (with both of the destination disk stages being RAID). Full backup cycles that used to take 18 hours and two or three media changes, with about a 10% failure rate due to media errors, now take 3 or 4 hours with no media changes and nearly 100% success. However, we are getting further and further off the subject of compression. That approach actually affects both tape and disk. Software compression happens on the client, so performance greatly depends on the type of clients being backed up. For example, there may be NAS boxes with low power processors, Software compression will definitely slow the backup of such clients. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
On 5/17/23 12:52, Marco Gaiarin wrote: I've LTO tape that work in very dusty environment by at least 6 years. Istead of dust, i've found that setting properly spooling and buffers prevent the spinup/spindown effect, that effectively can be very stressful... Yes, I went to great lengths to try to keep mine streaming and avoid shoe-shining, but with only moderate success. I have had many fewer problems, as well as much better performance, since I abandoned tape and went to disk-to-disk-to-removable-disk (with both of the destination disk stages being RAID). Full backup cycles that used to take 18 hours and two or three media changes, with about a 10% failure rate due to media errors, now take 3 or 4 hours with no media changes and nearly 100% success. However, we are getting further and further off the subject of compression. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
Mandi! Phil Stracchino In chel di` si favelave... > I have to admit I found the cost of the media for the last LTO > generation I used (LTO4) to be secondary to the cost of replacing drive > after drive after drive as they failed, because in my experience LTO > drives fail very quickly outside of a clean-room environment or a sealed > tape library. I've LTO tape that work in very dusty environment by at least 6 years. Istead of dust, i've found that setting properly spooling and buffers prevent the spinup/spindown effect, that effectively can be very stressful... -- La macchina del capo la guida Emilio Fede La macchina del capo la lava Emilio Fede La macchina del capo la parcheggia Emilio Fede ma la benzina gliela paghiamo noi [Dado] ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
On 5/17/23 11:48, Dr. Thorsten Brandau wrote: I decided to use the softwarecompression instead, as this actually compresses the files. At last it means 1.4:1 for me, which is a lot better than uncompressed backup (LTO9 tapes are very expensive...). I have to admit I found the cost of the media for the last LTO generation I used (LTO4) to be secondary to the cost of replacing drive after drive after drive as they failed, because in my experience LTO drives fail very quickly outside of a clean-room environment or a sealed tape library. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
Bill Arlofski via Bacula-users schrieb am 11.05.23 um 20:45: On 5/10/23 21:49, Dr. Thorsten Brandau wrote: Hi I use an lto-9 drive for backup. I activated hardware compression on the tape (i think) but all the logs of bacula show a tape change at 18TB which should be the native (uncompressed) capacity. From experience I would expect a ratio of at last 1:1.2. Is there any way to make sure to use compression (or as last resort activate software compression)? Hello Thorsten, Of course, with LTO tape drives, you always want to enable hardware compression and disable software compression in Bacula. How that is set on your specific drive(s) I do not know. :) Bacula will just write until it gets an EOT (end of tape) from the tape drive, so the compression you will see is dependent on your dataset. Some tips: In your Director's configuration for the Storage resource which points to the SD's Tape drive (or library with multiple drives), you should set `AllowCompression = no` This will tell the FDs to not try to compress data before sending it to the SD when using this Storage resource in a job, even if you have a `compression = (lzo|gzip|zstd)` in your fileset. For jobs using this Storage, if compression in enabled for the fileset in use, you will see a harmless info log entry in the job logs telling you that "compression is not allowed for this storage device". (or similar) I prefer to enable/disable this in the Director's Storage resource and always enable compression in my filesets so that the same job and fileset, when using a different Director Storage that allows compression, it will automatically "Just Work"™ P.S. You can enable/disable compression in the SD's Devices, but I prefer to set it at what I would call the beginning of the chain in the Director's Storage resource. Home some of this is helpful, Bill Hi Bill I decided to use the softwarecompression instead, as this actually compresses the files. At last it means 1.4:1 for me, which is a lot better than uncompressed backup (LTO9 tapes are very expensive...). Any other setting did not work and from the change rates I have seen the drive did not compress the data (in the autochanger). I have no idea if it is a problem of the autochanger or not. On the other hand, that also reduced my total backuptime quite a bit as less data had to be written and cached... Cheers T ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
Gary R. Schmidt schrieb am 12.05.23 um 16:26: On 11/05/2023 13:49, Dr. Thorsten Brandau wrote: Hi I use an lto-9 drive for backup. I activated hardware compression on the tape (i think) but all the logs of bacula show a tape change at 18TB which should be the native (uncompressed) capacity. From experience I would expect a ratio of at last 1:1.2. Is there any way to make sure to use compression (or as last resort activate software compression)? Are you using the actual compressed device in your bacula-sd.conf? On Solaris I use /dev/rmt/0cbn to turn on compression. (But I started out handling tape back when they were huge upright things, throwing pseudo-random characters into device names is reflex. :-) ) Cheers, Gary B-) Hi Gary, well I remember times were you could change the whole disc with a cake basket like holder and tapes could be used as paperweights and through from the 5th floor, but will work fine afterwards... I am running linux and the tape is /dev/nst0 which should referr to the compressed device. The loader is sg7 but that should not play any role. I also remember times where tapes had so many names in /dev but also our older LTOs do not fuss around and leave little options on the dev side, but you have to configure them with MT or MTX. However, I turned compression on with MT and MTX, so I still do not understand why it seems that the data is imcompressible... I will try software compression to see IF the data is compressible (s many text files, should shrink nice an easy...). Cheers TB ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
On 11/05/2023 13:49, Dr. Thorsten Brandau wrote: Hi I use an lto-9 drive for backup. I activated hardware compression on the tape (i think) but all the logs of bacula show a tape change at 18TB which should be the native (uncompressed) capacity. From experience I would expect a ratio of at last 1:1.2. Is there any way to make sure to use compression (or as last resort activate software compression)? Are you using the actual compressed device in your bacula-sd.conf? On Solaris I use /dev/rmt/0cbn to turn on compression. (But I started out handling tape back when they were huge upright things, throwing pseudo-random characters into device names is reflex. :-) ) Cheers, GaryB-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
Bill Arlofski via Bacula-users schrieb am 11.05.23 um 20:45: On 5/10/23 21:49, Dr. Thorsten Brandau wrote: Hi I use an lto-9 drive for backup. I activated hardware compression on the tape (i think) but all the logs of bacula show a tape change at 18TB which should be the native (uncompressed) capacity. From experience I would expect a ratio of at last 1:1.2. Is there any way to make sure to use compression (or as last resort activate software compression)? Hello Thorsten, Of course, with LTO tape drives, you always want to enable hardware compression and disable software compression in Bacula. How that is set on your specific drive(s) I do not know. :) Bacula will just write until it gets an EOT (end of tape) from the tape drive, so the compression you will see is dependent on your dataset. Some tips: In your Director's configuration for the Storage resource which points to the SD's Tape drive (or library with multiple drives), you should set `AllowCompression = no` This will tell the FDs to not try to compress data before sending it to the SD when using this Storage resource in a job, even if you have a `compression = (lzo|gzip|zstd)` in your fileset. For jobs using this Storage, if compression in enabled for the fileset in use, you will see a harmless info log entry in the job logs telling you that "compression is not allowed for this storage device". (or similar) I prefer to enable/disable this in the Director's Storage resource and always enable compression in my filesets so that the same job and fileset, when using a different Director Storage that allows compression, it will automatically "Just Work"™ P.S. You can enable/disable compression in the SD's Devices, but I prefer to set it at what I would call the beginning of the chain in the Director's Storage resource. Home some of this is helpful, Bill Hi Bill, thank you. It still does not solve my problem why it does not compress the data on the tape... I think this is strange. I try to use GZIP6 now and see if the written data/total data changes then to see if I can safe some meters of tapes. And Backuptime. It seems that the Raid here is not the fastest to read and that makes the backup slow. Cheers T ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] compression with lto?
On 5/10/23 21:49, Dr. Thorsten Brandau wrote: Hi I use an lto-9 drive for backup. I activated hardware compression on the tape (i think) but all the logs of bacula show a tape change at 18TB which should be the native (uncompressed) capacity. From experience I would expect a ratio of at last 1:1.2. Is there any way to make sure to use compression (or as last resort activate software compression)? Hello Thorsten, Of course, with LTO tape drives, you always want to enable hardware compression and disable software compression in Bacula. How that is set on your specific drive(s) I do not know. :) Bacula will just write until it gets an EOT (end of tape) from the tape drive, so the compression you will see is dependent on your dataset. Some tips: In your Director's configuration for the Storage resource which points to the SD's Tape drive (or library with multiple drives), you should set `AllowCompression = no` This will tell the FDs to not try to compress data before sending it to the SD when using this Storage resource in a job, even if you have a `compression = (lzo|gzip|zstd)` in your fileset. For jobs using this Storage, if compression in enabled for the fileset in use, you will see a harmless info log entry in the job logs telling you that "compression is not allowed for this storage device". (or similar) I prefer to enable/disable this in the Director's Storage resource and always enable compression in my filesets so that the same job and fileset, when using a different Director Storage that allows compression, it will automatically "Just Work"™ P.S. You can enable/disable compression in the SD's Devices, but I prefer to set it at what I would call the beginning of the chain in the Director's Storage resource. Home some of this is helpful, Bill -- Bill Arlofski w...@protonmail.com signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] compression with lto?
Hi I use an lto-9 drive for backup. I activated hardware compression on the tape (i think) but all the logs of bacula show a tape change at 18TB which should be the native (uncompressed) capacity. From experience I would expect a ratio of at last 1:1.2. Is there any way to make sure to use compression (or as last resort activate software compression)? - Originale Nachricht - Von: Chris Wilkinson Gesendet: 10.05.23 - 20:49 An: Phil Stracchino Betreff: Re: [Bacula-users] Storage protocols advice sought > It is a DLink DNS-325 which locked down so no chance of putting the SD there. > I guess I could reassign the SD uid:gid however that would upset some > existing backups on local disks. > > Best > -Chris- > > > > >> On 10 May 2023, at 19:11, Phil Stracchino wrote: >> >> On 5/10/23 11:19, Chris Wilkinson wrote: >>> This is not really a Bacula question but I'm hoping someone has come across >>> this when using NFS to mount a NAS storage device. >>> As I understand it, the UID:GID on the remote storage and the local SD >>> daemon must match or permission denied results. My NAS has NFS v3 and this >>> can't be changed. >>> I can create a new user on the NAS but I have no control over what UID:GID >>> it chooses and will not be the same as the SD daemon user:group. >>> I believe NFS v3 doesn't have the capability to map UIDs; that facility is >>> v4 only. >>> Has anyone found a workaround for this? >> >> >> What is the NAS? I'm guessing if you don't have the ability to control >> UIDs/GIDs on the NAS you probably can't install a SD on it directly either. >> NFS-mounting your Bacula backup storage is usually a bad idea unless you >> have no alternative. >> >> Honestly, in my experience off-the-shelf consumer/prosumer NAS appliances >> are utterly horrible from a management perspective — you WILL do things >> THEIR way, or not at all. I tried a consumer NAS a few uears ago — I'm >> blanking on the brand right now — and had to return it because not only was >> it useless for what I wanted, as well as having some pretty ghastly security >> issues. (Like making every file on every share world-read/write via NFS.) >> >> >> If you can't control the UID/GID assigned by the NAS, can you change the >> UID/GID of your storage daemon to match what the NAS assigns? That would at >> least solve the permission problems. >> >> >> -- >> Phil Stracchino >> Babylon Communications >> ph...@caerllewys.net >> p...@co.ordinate.org >> Landline: +1.603.293.8485 >> Mobile: +1.603.998.6958 >> >> >> >> ___ >> Bacula-users mailing list >> Bacula-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bacula-users > > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users -- Mit freundlichen Grüßen/Sincerely Yours/Avec mes meilleurs salutations/Yoroshiku onegai-shimasu BRACE GmbH Dr. Thorsten Brandau * Diese Email und die Anhänge sind vetraulich ++ This email and its attachments are confidential * - Sent from my mobile BRACE GmbH - Dr. Thorsten Brandau - Am Mittelberg 5 - 63791 Karlstein - Germany - www.brace.de smime.p7s Description: S/MIME Cryptographic Signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users