date:20190121

[Bacula-users] bacula compression -- LTO4

2019-01-21 Thread krashoverride

Hey there!

New question for you, about job compression (on tapes)
I'm running Bacula 7.4.4 server, with a 5.2.6 client (and PG db)

My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go native, 
1.6To compressed, so I've set my pool to have a Maximum Volume Bytes to 1600G
I've read on Bacula docs that for tapes, it's better not to configure 
compression in FileSet, and leave the hardware part do the compression

So what I did (as docs said) is to 
- cat /sys/class/scsi_tape/nst0/default_compression which has "1" value
+
root@server:~# tapeinfo -f /dev/nst0
Product Type: Tape Drive
Vendor ID: 'TANDBERG'
Product ID: 'LTO-4 HH'
Revision: 'U519'
Attached Changer API: No
SerialNumber: 'HU1023AMW9'
MinBlock: 1
MaxBlock: 16777215
SCSI ID: 1
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x46
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: 800226
Partition 0 Size in Kbytes: 800226
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0

So the compression seems to be enabled here

The fact is, when I run my backup, only ~1.1 To is used on the tape before 
being Full
+-++---+-+---+--+--+-+--+---+---+-+---+
| mediaid | volumename | volstatus | enabled | volbytes  | volfiles | 
volretention | recycle | slot | inchanger | mediatype | lastwritten | 
expiresin |
+-++---+-+---+--+--+-+--+---+---+-+---+
|   2 |    | Full  |   1 | 1,090,410,854,400 |1,090 |   
 2,592,000 |   1 |0 | 0 | LTO-4 | 2019-01-21 03:25:59 | 
2,569,770 |
+-++---+-+---+--+--+-+--+---+---+-+---+

On the error mail, I can see that 
21-Jan 03:26 server-sd JobId 3: End of medium on Volume "x" 
Bytes=1,090,410,854,400 Blocks=16,902,449 at 21-Jan-2019 03:26.
The number of blocks is almost the same than in the tapeinfo command. 

Is the "problem" coming from here ? 
Anything I can configure to have more that 1.1To used ?

Thank you!

---

Pool {
  Name = QUOT
  Pool Type = Backup
  Volume Retention = 30 days
  Storage = Lecteur_LTO4
  Recycle = yes
  AutoPrune = yes
  RecyclePool = QUOT
  Maximum Volume Bytes = 1600G
}

Storage {
  Name = Lecteur_LTO4
  Password = --
  Address = server
  SDPort = 9103
  Device = Lecteur_LTO4
  Media Type = LTO-4
}

Device {
  Name = Lecteur_LTO4
  Archive Device = /dev/nst0
  Media Type = LTO-4
  LabelMedia = no
  Random Access = no
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = yes
}


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Backing up to NetApp

2019-01-21 Thread Kern Sibbald


  
  
Hello Dan,


Put the following in your mtab or on
  the mount command line:


   
  rw,hard,nointr,rsize=1048576,wsize=1048576,bg,nfsvers=3,tcp


Note: the rsize and wsize will never
  really be used but will be negotiated
down by any Linux system to be the
  maximum possible (you might need to
reduce it by powers of 2 until it works
  on FreeBSD if FreeBSD does not use
negotiation such as Linux).


With those parameters, the NetApp I had
  (some time ago) ran nearly as fast as
local disks for writing Bacula volumes.


You might also find the following
  useful:


  
  https://www.netapp.com/us/media/tr-4067.pdf



Best regards,
Kern





On 1/20/19 12:06 PM, Dan Langille
  wrote:


  Have you backed up to NetApp?

At $WORK we are likely to soon have heaps of NetApp storage at our disposal.

Of course my thoughts turned to backups.

Do you use a NetApp appliance with Bacula as a destination for backups?

I know of the NetApp plugin for Bacula, but that is the wrong direction: that is for backing up the NetApp device.

I've never mounted remote storage for bacula-sd over any of NFS, CIF, Samba, etc.

I can't imagine NFS would be useful given the throughput.  Mind you, I don't yet know how much we'll be backing up, but it'll be more than 1

Have you?

--
Dan Langille - BSDCan / PGCon
d...@langille.org



  
  
  
  
  ___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users




  


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Problem with volume are not present when the tape is empty on incremental

2019-01-21 Thread Martin Simmons

I don't think this is different in 7.4.  Do you have an example volume with no
files that is listed by query 5?

As a workaround, you could have a cron job on the fd that touches a zero
length file every day to prevent a "no files" backup.  You could probably also
do this in a ClientRunBeforeJob script.

__Martin


> On Mon, 21 Jan 2019 08:37:59 +0100, Olivier Delestre said:
> 
> Yes, i understand. It s normal with v9.2.2 and >
> 
> But before with the 7.4 , this volume are present.
> 
> Tanks.
> 
> Le 19/01/2019 à 12:00, Kern Sibbald a écrit :
> > On a quick look this seems normal.  Bacula will not list volumes that 
> > have no files stored on them.
> >
> > Kern
> >
> > On 1/10/19 2:39 PM, Olivier Delestre wrote:
> >> hi,
> >>
> >> i use bacula v9.2.2 and storage on disk.
> >> i notice that when an incremantal is empty, bacula make a tape with a 
> >> file ( 612 bytes ).
> >> when you list with the query "5: List all backups for a Client" , 
> >> this volume is not present. but present in the pool and the file system.
> >> see below on the vol conty-2546
> >> note : the query is issue from sample-query.sql
> >>
> >> How to have this volume in my list ?
> >> A Bug or not ??
> >>
> >> Thanks
> >> 
> >> Choose a query (1-21): 5
> >> Enter Client Name: conty-fd
> >> ++--+---+---+-+--+++
> >> | jobid  | client   | fileset   | level | starttime   | 
> >> jobfiles | jobbytes   | volumename |
> >> ++--+---+---+-+--+++
> >> | 15,276 | conty-fd | fileset_conty | F | 2018-12-01 22:00:02 |
> >> 8,121 | 11,532,984,139 | conty-2471 |
> >> | 15,323 | conty-fd | fileset_conty | I | 2018-12-03 22:00:02 |
> >> 1 |221,917 | conty-2481 |
> >> | 15,540 | conty-fd | fileset_conty | I | 2018-12-10 22:00:02 |
> >> 1 |222,648 | conty-2487 |
> >> | 15,683 | conty-fd | fileset_conty | I | 2018-12-14 22:00:03 |
> >> 1 |  0 | conty-2503 |
> >> | 15,755 | conty-fd | fileset_conty | I | 2018-12-17 22:00:02 |
> >> 1 |223,377 | conty-2510 |
> >> | 15,977 | conty-fd | fileset_conty | I | 2018-12-24 22:00:02 |
> >> 1 |224,106 | conty-2514 |
> >> | 16,198 | conty-fd | fileset_conty | I | 2018-12-31 22:00:01 |
> >> 1 |224,835 | conty-2529 |
> >> | 16,374 | conty-fd | fileset_conty | F | 2019-01-05 22:02:31 |
> >> 8,120 | 11,532,987,786 | conty-2401 |
> >> | 16,421 | conty-fd | fileset_conty | I | 2019-01-07 22:00:02 |
> >> 1 |225,566 | conty-2407 |
> >> | 16,495 | conty-fd | fileset_conty | I | 2019-01-09 22:00:02 |  
> >> 226 | 10,177,801,880 | conty-2412 |
> >> ++--+---+---+-+--+++
> >> *list media pool=conty
> >> +-++---+-++--+--+-+--+---+---+-+--+-+---+
> >> | mediaid | volumename | volstatus | enabled | volbytes   | volfiles | 
> >> volretention | recycle | slot | inchanger | mediatype | voltype | volparts 
> >> | lastwritten | expiresin |
> >> +-++---+-++--+--+-+--+---+---+-+--+-+---+
> >> |   2,401 | conty-2401 | Used  |   1 | 11,542,823,537 |2 | 
> >>5,529,600 |   1 |0 | 0 | File  |   1 |0 
> >> | 2019-01-05 22:05:03 | 5,124,733 |
> >> |   2,404 | conty-2404 | Used  |   1 |612 |0 | 
> >>5,529,600 |   1 |0 | 0 | File  |   1 |0 
> >> | 2019-01-06 22:00:02 | 5,210,832 |
> >> |   2,407 | conty-2407 | Used  |   1 |226,476 |0 | 
> >>5,529,600 |   1 |0 | 0 | File  |   1 |0 
> >> | 2019-01-07 22:00:02 | 5,297,232 |
> >> |   2,410 | conty-2410 | Used  |   1 |612 |0 | 
> >>5,529,600 |   1 |0 | 0 | File  |   1 |0 
> >> | 2019-01-08 22:00:02 | 5,383,632 |
> >> |   2,412 | conty-2412 | Used  |   1 | 10,185,386,134 |2 | 
> >>5,529,600 |   1 |0 | 0 | File  |   1 |0 
> >> | 2019-01-09 22:00:34 | 5,470,064 |
> >> |   2,415 | conty-2415 | Recycle   |   1 |  1 |0 | 
> >>5,529,600 |   1 |0 | 0 | File  |   1 |0 
> >> | 2018-11-12 22:00:40 |   458,870 |
> >> |   2,418 | conty-2418 | Purged|   1 | 56,360,634 |0 | 
> >>5,529,600 |   1 |0 | 0 | File  |   1 |0 
> >> | 2018-11-13 22:00:07 |   545,237 |
> >> |   2,422

Re: [Bacula-users] bacula compression -- LTO4

2019-01-21 Thread Tilman Schmidt

On Mon, Jan 21, 2019, at 09:44, krashoverr...@free.fr wrote:
> My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go 
> native, 1.6To compressed, so I've set my pool to have a Maximum Volume 
> Bytes to 1600G

800 GB is the real capacity of an LTO-4 cartridge.
The 1,6 TB "compressed capacity" is purely a marketing number based on the 
unfounded assumption that your data can be compressed to 50%.
Compression depends heavily on the type of data.
Much redundancy (eg. logfiles) -> much compression
Little redundancy (eg. videos) -> little or no compression

> The fact is, when I run my backup, only ~1.1 To is used on the tape 
> before being Full

This confirms that compression is active (otherwise it would be full after 800 
GB) and it is probably all you can get out of compression with your data.

-- 
Tilman Schmidt
til...@imap.cc

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] bacula compression -- LTO4

2019-01-21 Thread Adam Weremczuk


Hello,

You might be interested in checking this thread: 
https://sourceforge.net/p/bacula/mailman/message/36386670/


A perl script is mentioned there: 
https://github.com/hreinecke/sg3_utils/issues/18 which can provide you 
with e.g. daily reports of raw space remaining on tapes.


Thanks,
Adam


On 21/01/19 08:44, krashoverr...@free.fr wrote:

Hey there!

New question for you, about job compression (on tapes)
I'm running Bacula 7.4.4 server, with a 5.2.6 client (and PG db)

My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go native, 
1.6To compressed, so I've set my pool to have a Maximum Volume Bytes to 1600G
I've read on Bacula docs that for tapes, it's better not to configure 
compression in FileSet, and leave the hardware part do the compression

So what I did (as docs said) is to
- cat /sys/class/scsi_tape/nst0/default_compression which has "1" value
+
root@server:~# tapeinfo -f /dev/nst0
Product Type: Tape Drive
Vendor ID: 'TANDBERG'
Product ID: 'LTO-4 HH'
Revision: 'U519'
Attached Changer API: No
SerialNumber: 'HU1023AMW9'
MinBlock: 1
MaxBlock: 16777215
SCSI ID: 1
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x46
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: 800226
Partition 0 Size in Kbytes: 800226
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0

So the compression seems to be enabled here

The fact is, when I run my backup, only ~1.1 To is used on the tape before 
being Full
+-++---+-+---+--+--+-+--+---+---+-+---+
| mediaid | volumename | volstatus | enabled | volbytes  | volfiles | 
volretention | recycle | slot | inchanger | mediatype | lastwritten | 
expiresin |
+-++---+-+---+--+--+-+--+---+---+-+---+
|   2 |    | Full  |   1 | 1,090,410,854,400 |1,090 |   
 2,592,000 |   1 |0 | 0 | LTO-4 | 2019-01-21 03:25:59 | 
2,569,770 |
+-++---+-+---+--+--+-+--+---+---+-+---+

On the error mail, I can see that
21-Jan 03:26 server-sd JobId 3: End of medium on Volume "x" 
Bytes=1,090,410,854,400 Blocks=16,902,449 at 21-Jan-2019 03:26.
The number of blocks is almost the same than in the tapeinfo command.

Is the "problem" coming from here ?
Anything I can configure to have more that 1.1To used ?

Thank you!

---

Pool {
   Name = QUOT
   Pool Type = Backup
   Volume Retention = 30 days
   Storage = Lecteur_LTO4
   Recycle = yes
   AutoPrune = yes
   RecyclePool = QUOT
   Maximum Volume Bytes = 1600G
}

Storage {
   Name = Lecteur_LTO4
   Password = --
   Address = server
   SDPort = 9103
   Device = Lecteur_LTO4
   Media Type = LTO-4
}

Device {
   Name = Lecteur_LTO4
   Archive Device = /dev/nst0
   Media Type = LTO-4
   LabelMedia = no
   Random Access = no
   AutomaticMount = yes
   RemovableMedia = no
   AlwaysOpen = yes
}


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] bacula compression -- LTO4

Re: [Bacula-users] Backing up to NetApp

Re: [Bacula-users] Problem with volume are not present when the tape is empty on incremental

Re: [Bacula-users] bacula compression -- LTO4

Re: [Bacula-users] bacula compression -- LTO4

5 matches

Site Navigation

Mail list logo

Footer information