[Bacula-users] bacula compression -- LTO4
Hey there! New question for you, about job compression (on tapes) I'm running Bacula 7.4.4 server, with a 5.2.6 client (and PG db) My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go native, 1.6To compressed, so I've set my pool to have a Maximum Volume Bytes to 1600G I've read on Bacula docs that for tapes, it's better not to configure compression in FileSet, and leave the hardware part do the compression So what I did (as docs said) is to - cat /sys/class/scsi_tape/nst0/default_compression which has "1" value + root@server:~# tapeinfo -f /dev/nst0 Product Type: Tape Drive Vendor ID: 'TANDBERG' Product ID: 'LTO-4 HH' Revision: 'U519' Attached Changer API: No SerialNumber: 'HU1023AMW9' MinBlock: 1 MaxBlock: 16777215 SCSI ID: 1 SCSI LUN: 0 Ready: yes BufferedMode: yes Medium Type: Not Loaded Density Code: 0x46 BlockSize: 0 DataCompEnabled: yes DataCompCapable: yes DataDeCompEnabled: yes CompType: 0x1 DeCompType: 0x1 BOP: yes Block Position: 0 Partition 0 Remaining Kbytes: 800226 Partition 0 Size in Kbytes: 800226 ActivePartition: 0 EarlyWarningSize: 0 NumPartitions: 0 MaxPartitions: 0 So the compression seems to be enabled here The fact is, when I run my backup, only ~1.1 To is used on the tape before being Full +-++---+-+---+--+--+-+--+---+---+-+---+ | mediaid | volumename | volstatus | enabled | volbytes | volfiles | volretention | recycle | slot | inchanger | mediatype | lastwritten | expiresin | +-++---+-+---+--+--+-+--+---+---+-+---+ | 2 | | Full | 1 | 1,090,410,854,400 |1,090 | 2,592,000 | 1 |0 | 0 | LTO-4 | 2019-01-21 03:25:59 | 2,569,770 | +-++---+-+---+--+--+-+--+---+---+-+---+ On the error mail, I can see that 21-Jan 03:26 server-sd JobId 3: End of medium on Volume "x" Bytes=1,090,410,854,400 Blocks=16,902,449 at 21-Jan-2019 03:26. The number of blocks is almost the same than in the tapeinfo command. Is the "problem" coming from here ? Anything I can configure to have more that 1.1To used ? Thank you! --- Pool { Name = QUOT Pool Type = Backup Volume Retention = 30 days Storage = Lecteur_LTO4 Recycle = yes AutoPrune = yes RecyclePool = QUOT Maximum Volume Bytes = 1600G } Storage { Name = Lecteur_LTO4 Password = -- Address = server SDPort = 9103 Device = Lecteur_LTO4 Media Type = LTO-4 } Device { Name = Lecteur_LTO4 Archive Device = /dev/nst0 Media Type = LTO-4 LabelMedia = no Random Access = no AutomaticMount = yes RemovableMedia = no AlwaysOpen = yes } ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backing up to NetApp
Hello Dan, Put the following in your mtab or on the mount command line: rw,hard,nointr,rsize=1048576,wsize=1048576,bg,nfsvers=3,tcp Note: the rsize and wsize will never really be used but will be negotiated down by any Linux system to be the maximum possible (you might need to reduce it by powers of 2 until it works on FreeBSD if FreeBSD does not use negotiation such as Linux). With those parameters, the NetApp I had (some time ago) ran nearly as fast as local disks for writing Bacula volumes. You might also find the following useful: https://www.netapp.com/us/media/tr-4067.pdf Best regards, Kern On 1/20/19 12:06 PM, Dan Langille wrote: Have you backed up to NetApp? At $WORK we are likely to soon have heaps of NetApp storage at our disposal. Of course my thoughts turned to backups. Do you use a NetApp appliance with Bacula as a destination for backups? I know of the NetApp plugin for Bacula, but that is the wrong direction: that is for backing up the NetApp device. I've never mounted remote storage for bacula-sd over any of NFS, CIF, Samba, etc. I can't imagine NFS would be useful given the throughput. Mind you, I don't yet know how much we'll be backing up, but it'll be more than 1 Have you? -- Dan Langille - BSDCan / PGCon d...@langille.org ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Problem with volume are not present when the tape is empty on incremental
I don't think this is different in 7.4. Do you have an example volume with no files that is listed by query 5? As a workaround, you could have a cron job on the fd that touches a zero length file every day to prevent a "no files" backup. You could probably also do this in a ClientRunBeforeJob script. __Martin > On Mon, 21 Jan 2019 08:37:59 +0100, Olivier Delestre said: > > Yes, i understand. It s normal with v9.2.2 and > > > But before with the 7.4 , this volume are present. > > Tanks. > > Le 19/01/2019 à 12:00, Kern Sibbald a écrit : > > On a quick look this seems normal. Bacula will not list volumes that > > have no files stored on them. > > > > Kern > > > > On 1/10/19 2:39 PM, Olivier Delestre wrote: > >> hi, > >> > >> i use bacula v9.2.2 and storage on disk. > >> i notice that when an incremantal is empty, bacula make a tape with a > >> file ( 612 bytes ). > >> when you list with the query "5: List all backups for a Client" , > >> this volume is not present. but present in the pool and the file system. > >> see below on the vol conty-2546 > >> note : the query is issue from sample-query.sql > >> > >> How to have this volume in my list ? > >> A Bug or not ?? > >> > >> Thanks > >> > >> Choose a query (1-21): 5 > >> Enter Client Name: conty-fd > >> ++--+---+---+-+--+++ > >> | jobid | client | fileset | level | starttime | > >> jobfiles | jobbytes | volumename | > >> ++--+---+---+-+--+++ > >> | 15,276 | conty-fd | fileset_conty | F | 2018-12-01 22:00:02 | > >> 8,121 | 11,532,984,139 | conty-2471 | > >> | 15,323 | conty-fd | fileset_conty | I | 2018-12-03 22:00:02 | > >> 1 |221,917 | conty-2481 | > >> | 15,540 | conty-fd | fileset_conty | I | 2018-12-10 22:00:02 | > >> 1 |222,648 | conty-2487 | > >> | 15,683 | conty-fd | fileset_conty | I | 2018-12-14 22:00:03 | > >> 1 | 0 | conty-2503 | > >> | 15,755 | conty-fd | fileset_conty | I | 2018-12-17 22:00:02 | > >> 1 |223,377 | conty-2510 | > >> | 15,977 | conty-fd | fileset_conty | I | 2018-12-24 22:00:02 | > >> 1 |224,106 | conty-2514 | > >> | 16,198 | conty-fd | fileset_conty | I | 2018-12-31 22:00:01 | > >> 1 |224,835 | conty-2529 | > >> | 16,374 | conty-fd | fileset_conty | F | 2019-01-05 22:02:31 | > >> 8,120 | 11,532,987,786 | conty-2401 | > >> | 16,421 | conty-fd | fileset_conty | I | 2019-01-07 22:00:02 | > >> 1 |225,566 | conty-2407 | > >> | 16,495 | conty-fd | fileset_conty | I | 2019-01-09 22:00:02 | > >> 226 | 10,177,801,880 | conty-2412 | > >> ++--+---+---+-+--+++ > >> *list media pool=conty > >> +-++---+-++--+--+-+--+---+---+-+--+-+---+ > >> | mediaid | volumename | volstatus | enabled | volbytes | volfiles | > >> volretention | recycle | slot | inchanger | mediatype | voltype | volparts > >> | lastwritten | expiresin | > >> +-++---+-++--+--+-+--+---+---+-+--+-+---+ > >> | 2,401 | conty-2401 | Used | 1 | 11,542,823,537 |2 | > >>5,529,600 | 1 |0 | 0 | File | 1 |0 > >> | 2019-01-05 22:05:03 | 5,124,733 | > >> | 2,404 | conty-2404 | Used | 1 |612 |0 | > >>5,529,600 | 1 |0 | 0 | File | 1 |0 > >> | 2019-01-06 22:00:02 | 5,210,832 | > >> | 2,407 | conty-2407 | Used | 1 |226,476 |0 | > >>5,529,600 | 1 |0 | 0 | File | 1 |0 > >> | 2019-01-07 22:00:02 | 5,297,232 | > >> | 2,410 | conty-2410 | Used | 1 |612 |0 | > >>5,529,600 | 1 |0 | 0 | File | 1 |0 > >> | 2019-01-08 22:00:02 | 5,383,632 | > >> | 2,412 | conty-2412 | Used | 1 | 10,185,386,134 |2 | > >>5,529,600 | 1 |0 | 0 | File | 1 |0 > >> | 2019-01-09 22:00:34 | 5,470,064 | > >> | 2,415 | conty-2415 | Recycle | 1 | 1 |0 | > >>5,529,600 | 1 |0 | 0 | File | 1 |0 > >> | 2018-11-12 22:00:40 | 458,870 | > >> | 2,418 | conty-2418 | Purged| 1 | 56,360,634 |0 | > >>5,529,600 | 1 |0 | 0 | File | 1 |0 > >> | 2018-11-13 22:00:07 | 545,237 | > >> | 2,422
Re: [Bacula-users] bacula compression -- LTO4
On Mon, Jan 21, 2019, at 09:44, krashoverr...@free.fr wrote: > My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go > native, 1.6To compressed, so I've set my pool to have a Maximum Volume > Bytes to 1600G 800 GB is the real capacity of an LTO-4 cartridge. The 1,6 TB "compressed capacity" is purely a marketing number based on the unfounded assumption that your data can be compressed to 50%. Compression depends heavily on the type of data. Much redundancy (eg. logfiles) -> much compression Little redundancy (eg. videos) -> little or no compression > The fact is, when I run my backup, only ~1.1 To is used on the tape > before being Full This confirms that compression is active (otherwise it would be full after 800 GB) and it is probably all you can get out of compression with your data. -- Tilman Schmidt til...@imap.cc ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula compression -- LTO4
Hello, You might be interested in checking this thread: https://sourceforge.net/p/bacula/mailman/message/36386670/ A perl script is mentioned there: https://github.com/hreinecke/sg3_utils/issues/18 which can provide you with e.g. daily reports of raw space remaining on tapes. Thanks, Adam On 21/01/19 08:44, krashoverr...@free.fr wrote: Hey there! New question for you, about job compression (on tapes) I'm running Bacula 7.4.4 server, with a 5.2.6 client (and PG db) My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go native, 1.6To compressed, so I've set my pool to have a Maximum Volume Bytes to 1600G I've read on Bacula docs that for tapes, it's better not to configure compression in FileSet, and leave the hardware part do the compression So what I did (as docs said) is to - cat /sys/class/scsi_tape/nst0/default_compression which has "1" value + root@server:~# tapeinfo -f /dev/nst0 Product Type: Tape Drive Vendor ID: 'TANDBERG' Product ID: 'LTO-4 HH' Revision: 'U519' Attached Changer API: No SerialNumber: 'HU1023AMW9' MinBlock: 1 MaxBlock: 16777215 SCSI ID: 1 SCSI LUN: 0 Ready: yes BufferedMode: yes Medium Type: Not Loaded Density Code: 0x46 BlockSize: 0 DataCompEnabled: yes DataCompCapable: yes DataDeCompEnabled: yes CompType: 0x1 DeCompType: 0x1 BOP: yes Block Position: 0 Partition 0 Remaining Kbytes: 800226 Partition 0 Size in Kbytes: 800226 ActivePartition: 0 EarlyWarningSize: 0 NumPartitions: 0 MaxPartitions: 0 So the compression seems to be enabled here The fact is, when I run my backup, only ~1.1 To is used on the tape before being Full +-++---+-+---+--+--+-+--+---+---+-+---+ | mediaid | volumename | volstatus | enabled | volbytes | volfiles | volretention | recycle | slot | inchanger | mediatype | lastwritten | expiresin | +-++---+-+---+--+--+-+--+---+---+-+---+ | 2 | | Full | 1 | 1,090,410,854,400 |1,090 | 2,592,000 | 1 |0 | 0 | LTO-4 | 2019-01-21 03:25:59 | 2,569,770 | +-++---+-+---+--+--+-+--+---+---+-+---+ On the error mail, I can see that 21-Jan 03:26 server-sd JobId 3: End of medium on Volume "x" Bytes=1,090,410,854,400 Blocks=16,902,449 at 21-Jan-2019 03:26. The number of blocks is almost the same than in the tapeinfo command. Is the "problem" coming from here ? Anything I can configure to have more that 1.1To used ? Thank you! --- Pool { Name = QUOT Pool Type = Backup Volume Retention = 30 days Storage = Lecteur_LTO4 Recycle = yes AutoPrune = yes RecyclePool = QUOT Maximum Volume Bytes = 1600G } Storage { Name = Lecteur_LTO4 Password = -- Address = server SDPort = 9103 Device = Lecteur_LTO4 Media Type = LTO-4 } Device { Name = Lecteur_LTO4 Archive Device = /dev/nst0 Media Type = LTO-4 LabelMedia = no Random Access = no AutomaticMount = yes RemovableMedia = no AlwaysOpen = yes } ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users