Re: [Bacula-users] Possible to compile only Bacaula-FD ?
> Today i download source from Bacula-9.4.4 and i compiled succesffuly, > so please exist the possibilities to compile only the Bacula-FD or > only the Bacula-SD section? > > so my Client don't need to have all installed in there are enough > if the FD are available. Yes it's possible but you have to manage the install yourself, only copying the files you need. I don't believe the Bacula install scripts can do this for you. You can perhaps get some ideas and a list of files from the Arch Linux AUR build scripts. These compile the code once, then create separate packages from that. One each for SD, FD, Dir, Console, BAT and a "common" one that contains files needed by the others. The script is here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=bacula Look for the "cp" commands in the package_bacula-XXX functions. I'm not sure if this is the best example as I think a few other Linux distributions also split out the build into multiple packages, but I'm not so familiar with those. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape SL150 (Oracle). Drive Read-Only. Licence?
> The 2 drive is LTO5. The tape is LTO5. Who manufactured the drives? HP or IBM? Can you find out the model numbers? Have you tried running the OEM diagnostics on them to see what they say? The HP ones are particularly detailed and informative. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fwd: Is there a test for LTO-4 tapes or just pull them from service?
> They passed the rewind - write EOF test. Is there something I should try > to make them usable again (a test/format of sorts) or is the Bacula error > indication sufficient that I should just pull them from service? Normally LTO tapes don't get write errors as the drive rewrites any failed blocks automatically, so all you see is your write speeds dropping and in bad cases the total capacity of the tape decreasing. Best thing would be to run the diagnostic utility from your tape drive manufacturer, as these are extremely detailed and will tell you whether the tape is at fault or if it's an issue with the drive. Not sure about the IBM utility but the HP one will let you test individual tapes to confirm they are good and will even let you pull the log files from the drive that the firmware writes, which can be quite useful in diagnosing a fault. (It was here I found messages about the head not moving the expected distance which led me to disassemble a failing LTO5 drive[1] to clean the head mechanism and get the drive back in service). I'm assuming you've done the obvious and used a cleaning tape recently of course. Cheers, Adam. [1]: Just in case anyone is curious, I took photos of the disassembly and repair process: http://sbfmdrv.blogspot.com/2018/11/disassembling-and-cleaning-lto-tape.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Does bacula schedule cleaning by default?
> I am looking for a way the system can work with > bacula so that clean up scheduling happen automatically. IS it > possible also to know if their has been cleaning in the past from > bacula's perspective? Would really appreciate any lead I had to modify the autoloader script to enable LTO encryption after loading each tape, so this script is already called every time Bacula wants a tape loaded. I imagine you could add some commands to this same script to check whether the drive is requesting cleaning, and if so, load a cleaning tape, wait until it's ejected, then load the tape originally requested and continue. This way Bacula wouldn't notice anything except once in a while it takes much longer than normal to load a tape, but the cleaning tape would be automatically used as needed. Having said that, my library (an old HP unit that came with an LTO3 drive) can be set to automatically load a cleaning tape when the drive requests one, so I imagine most tape libraries have this functionality in them somewhere. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Resource SD (Media Type) mix lto4 and lto3
> I have tape library 8 slot and 1drive. the drive is lto4. > > I have lto4 and lto3 (50/50) > > Can i use value "Media Type" in resource bacula-sd.conf with LTO4 > example: *Media Type = LTO4"?? But the pool is mix: have lto4 and > LTO3 > > Is this a problem? It could be because as far as I recall you can only specify one media type for your drive. If you want to treat LTO3 and LTO4 differently, you will need two drives. Normally you just set everything to LTO, and any tape will go into the drive. If you specifically only want certain backups on certain tapes then it's probably better to handle that at the pool level. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Windows backup much slower than Linux backup
> I'm backing up 2 servers with Bacula, one with Windows 2016, the other > one with CentOS. The hardware is described below. The Windows server is > much more powerful than the Linux server in all respects, and should > theoretically deliver data to the Bacula server at a much higher rate. > But in reality, the Linux server delivers data about 7 times faster over > the network, than the Windows server. It's very hard to say, because many small files will take longer to transfer than a couple of large ones, and if there is other machine activity reading or writing from the disk at the time then this can sometimes slow the transfer speed dramatically, at least with magnetic disks. You could try to create a large file on both machines, say 10-20GB, and then try to back up just that one file. This should eliminate one variable (many vs few files) and give you a better idea how different the machines are. Perhaps you could also run some disk speed tests locally on each machine. Linux has hdparm included in most distros, but I'm not sure of a Windows equivalent although I'm sure there are many out there. Because you only have a single average transfer speed for the Windows machine, it's conceivable that a failing disk will work fine for a while but then become stuck on a few sectors and sit there for even a few minutes retrying the read operation before continuing. This could result in a slow overall transfer rate but a fast speed test, if the speed test doesn't read any of the failing data areas. If the speed test comes back good, then it might be worth trying to find some SMART tools for your disk/controller which can usually query the drives for the number of errors they've been encountering recently. If those are high for one drive then that could explain what's going on. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Spooling vs buffering
> I have a different understanding of the function. there is no need > to read "data off clients as fast as possible" - if your clients are > fast, they have no problems to provide the data at lower rates, > too. It's the other way round; if your clients are slow so they > cannot feed the data fast enough to keep the tape streaming. This > often happens when you run for example incremental backups over a > big data set (say, millions of files) with only little changes. It may also help in this case, but so would buffering instead of spooling. I was under the impression that in the case of slow clients, Bacula is designed to read from many clients at the same time, so that it can get the throughput required for the tape without spooling. > In this case the client gets the time it needs to traverse the fole > directory tree, and when done, you have all date to be backed up > collected in the spool area which is then fast enough to kepp the > tape happily streaming. This is true, but the drawback of the spool file is that you need enough disk space to hold a full tape's worth of data for it to perform optimally. If the spool file is not an exact multiple of the tape size, performance will drop. > > Consequently spooling works best when the spool file is large > > enough to contain one whole tape's worth of data, and you have > > enough clients backing up that there is always a complete spool > > file ready to write out to tape. > > This is not necessary. Or only one possible special case. It's not necessary, but if you do not do this, performance will suffer and your tape will shoe shine. > > Anything less than this and spooling will slow things down. > > This is not correct, if you consider incremental and differential > backups. I am only referring to getting data from the spool file onto tape. Let's say you have a 100GB spool file and you are writing to an 800GB tape. The process will go like this: * Read 100GB from client, tape is idle * Write 100GB to tape, pause tape * Read next 100GB from client while tape is paused * Start up tape again and write next 100GB Thus even if your clients can keep up with the tape 100% of the time, you will still introduce extra shoe shining if your spool file is not exactly one tape in size. (If your spool file is larger than one tape, then you will fill up one tape in one continuous operation which is perfect, but then the second tape will pause once the end of the spool file has been reached which is not ideal either.) So you can see that using a spool file is typically worse for a tape drive, as it will almost always introduce additional stop/start cycles (shoe shining) which would not be there otherwise, unless you have a very slow client. This is why in my opinion buffering is better, because a small FIFO buffer can read data from clients *while* writing to tape, so there is no extra shoe shining. A buffer will also not harm performance if it is not required, however using a spool file when one is not needed will make performance worse. With my own experience writing data to tape using mbuffer and tar, a buffer of 4GB was enough to prevent all shoe shining, and it did not slow down the process at all. However with Bacula, my spool file must be 800GB to achieve the same result, and even this makes the process take much longer because the tape is idle while the spool file is filling up the first time. I don't have 800GB available for the spool file either, which means my choices are: 1. Use a smaller spool file and live with tape shoe shine. 2. Don't use a spool file at all and live with tape shoe shine caused by slow clients. 3. Buy more disk I can't use for real storage because it must be reserved for Bacula scratch space, and live with shoe shine as well because tapes are never exactly 800GB. 4. Implement buffering support in Bacula so that I can eliminate shoe shining and speed up my backups, without buying new hardware. I definitely favour #4 because having support for large tape buffers in Bacula would provide some big performance benefits. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Spooling vs buffering (was: Autochanger Configuration Help)
> Spooling can reduce overall throughput because the data is > sequentially written to disk and then read back. This is what got me. I thought it was a buffer to ride out variations in disk read speed (like the mbuffer program) but it's not. The purpose is to get data off clients as fast as possible for later writing to slow tape. Consequently spooling works best when the spool file is large enough to contain one whole tape's worth of data, and you have enough clients backing up that there is always a complete spool file ready to write out to tape. Anything less than this and spooling will slow things down. I think we need a FIFO buffering option in Bacula that will let a few GB of data be buffered in memory, so that the tape doesn't shoe shine when the disks briefly go slow for some reason. I looked at the code for this but it seems like it could be a bit tricky, because you need to return success/fail for each written data block, but you can't really do this if you're caching them for writing later. This is because if there's a write error, it's difficult to handle the error when the block was already written long ago. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Lack Persistent Device Naming to use with Alert Command
> I ended up figuring out on accident my udev rules file did work, I had just
> been trying to test it by restarting the service but it took a full reboot
> for the new rules to get loaded.
Normally you do something like this[1]:
# udevadm control --reload-rules && udevadm trigger
> *SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}!="8",
This will match all devices that are not type 8 (so everything except
autochangers).
> This ends up creating symlinks for every sg* device, not just the tape
> drives, so ideally it would be nice to figure out how to filter it.
If you want to restrict it to a particular type, you can view udev
attributes for specific devices like this:
# udevadm info -a -n /dev/sch0 | grep type
ATTRS{type}=="8"
# udevadm info -a -n /dev/nst0 | grep type
ATTRS{type}=="1"
Here it suggests that changing the rule from type 8 to type 1 will only
match tape drives. On my system a DVD drive is 5 and a hard drive is
0, so I guess using type 1 should work for tapes only.
Cheers,
Adam.
[1]:
https://unix.stackexchange.com/questions/39370/how-to-reload-udev-rules-without-reboot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Lack Persistent Device Naming to use with Alert Command
> There are symlinks for the archive devices but not the tape drive control > devices which is what the tape alert commands need. Ah I see, you need the /dev/sg* equivalent for the /dev/st* in use. Can you copy the udev rules that produce symlinks for the autochangers (see link in my last post) and tweak them so that they also work for the drives themselves? It looks like they pick up anything that's a "type 8" device so if you duplicate the rule but change it to whatever type number that tape drives are, presumably you'd end up with the same result. > My only other idea is that maybe you could create a script that rewrites > the bacula-sd.conf every boot by parsing the output of this command as it > shows which sg* devices correspond to the st* ones: If you wanted to do that it might be easier to have the script create/update symlinks in /dev so you don't need to rewrite the Bacula config file. But this is pretty much what udev does already so better to get that working if you can. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Lack Persistent Device Naming to use with Alert Command
> I'm using Ubuntu 18.04.1 with Bacula 9.0.6 from the main repo. I'd like to > enable the above alerts from the sample bacula-sd.conf but can't because my > system is assigning random device names after each reboot so that > /dev/nst0, /dev/nst1, etc. can be any /dev/sg* number. I looked into > creating a udev rule for the drives but there's no output from the control > device that can be uniquely identified (DEVPATH can't be used in a udev > rule I've found out): On my Arch Linux system, udev already has persistent rules for tapes in /dev/tape: $ ls /dev/tape/by-id/ total 0 drwxr-xr-x 2 root root 100 Dec 18 20:45 . drwxr-xr-x 4 root root 80 Dec 18 18:22 .. lrwxrwxrwx 1 root root 9 Dec 19 23:21 scsi-350014380125815e0 -> ../../sg6 lrwxrwxrwx 1 root root 9 Dec 19 23:21 scsi-350014380125815e1 -> ../../st0 lrwxrwxrwx 1 root root 10 Dec 19 23:21 scsi-350014380125815e1-nst -> ../../nst0 sg6 is the autochanger on my system. If your system doesn't have these, and no update to udev is available, maybe you can copy the rules across from a newer version of udev[1]? > Is there another way do make this work? There's a more recent firmware for > my tape library, a Dell ML6000, that I've held off installing. Do you > think a new firmware might allow the sg driver to provide more output? Nope, this is all part of the udev rule system. Cheers, Adam. [1]: https://github.com/systemd/systemd/blob/master/rules/60-persistent-storage-tape.rules ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Running a script each time a client is addressed
> Thanks. I'm aware of this possibility however there's only 1 remote machine > in each remote network. > This makes setting up OpenVPN for each of those a hassle. It feels like > using a shotgun to kill a fly. > It also requires to do extra firewalling while with SSH only the needed > ports are exposed to the client. Well it's only one port you need to open on the server, and each client can then connect back to that so there's no firewall changes on the clients. With SSH each client machine needs an open port so there's a larger attack surface compared with OpenVPN where the clients don't need to open any ports at all. Having one machine in each remote network is no problem. Each machine runs the OpenVPN client, which connects to your server on its public IP and each gets assigned a unique IP address in a private VPN subnet of your choice. Then they are configured to contact Bacula on the server's private VPN IP and everything is securely routed. It's really a pretty lightweight solution - OpenVPN is small and fast - and will give you a much more robust solution than using SSH with port forwarding. I can't imagine it would take even one hour to set up something like this, much less if you've used OpenVPN before. I know you want an answer that will work for SSH, but this is the first clue that you're trying to bang a nail in with a shoe, instead of using a hammer[1]. Cheers, Adam. [1]: https://weblogs.asp.net/alex_papadimoulis/408925 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Running a script each time a client is addressed
> I have a client that's remote. The connectivity to the remote client is > established through an SSH script. > In general the SSH connection is up or re-established when a job runs > through Run Before Job in a JobDefs piece. > This works properly. Just curious - any reason why you're using SSH rather than something like OpenVPN? OpenVPN will automatically connect as needed so it's much easier to manage in scripted environments, so just wondering what makes SSH the preferred option in this case. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to LTO-8
> I found another person with the very same symptoms and they resolved the
> problem by changing out the SAS host adapter
Wow that's weird that a SAS controller would be to blame! My library
and drive are HP and they supply "HP Tape Tools" which you can use to
run tests on the library (and drive) to see if they are working
properly. Are there any vendor diagnostics for your library that you
could run to confirm it is working correctly?
It seems a bit odd it would be the SAS controller if it worked before
upgrading to the latest Bacula version. Perhaps it's an issue with a
kernel driver, assuming you upgraded the whole OS at the same time?
> I Note that both my bacula-dir and my bacula-sd conf files have a
> Storage section, with a Device section only in bacula-sd.
>
> Is this correct or should the Storage and Device be in one or the other
> confs? If they or one or the other sections should be in both confs,
> should they be identical???
This is correct. The entries in bacula-dir tell it to connect to
the bacula-sd daemon, and the entries in bacula-sd tell it which
hardware device to use. This is because you can have multiple
bacula-sd instances on different machines, so the Director needs to
know which device is controlled by which machine.
You should have both the autochanger and the drive listed in both
config files, however the one listed in bacula-dir.conf will be much
shorter as it won't have any of the configuration, basically just the
password for the bacula-sd daemon.
I am not sure whether it is necessary to list both the drive and the
autochanger (from your config files it looks like you only include the
autochanger and not the drive) but since Bacula is calling the
mtx-changer script it would seem to be fine the way you have it now.
In case it's helpful, here are the full Storage sections from my own
bacula-dir.conf:
Autochanger {
Name = peapod-x
Address = localhost
SDPort = 9103
Password = "xxx"
Device = peapod-x
Media Type = LTO-4
Maximum Concurrent Jobs = 10
}
Storage {
Name = peapod-d1
Address = localhost
SDPort = 9103
Password = "xxx"
Device = peapod-d1
Media Type = LTO-4
Autochanger = peapod-x
Maximum Concurrent Jobs = 10
}
Your Storage section with "Autochanger = yes" is equivalent to the
Autochanger section, this is just a newer way of writing it that
works with recent Bacula versions. (Definitely 9.x, not sure
about 7.x though.)
Cheers,
Adam.
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to LTO-8
> As mentioned earlier, the command > > ./mtx-changer /dev/sg3 load 1 /dev/st0 0 > > actually put the desired tape into the drive, however > > ./mtx-changer /dev/sg3 unload 1 /dev/st0 0 > > returns the following error message This to me looks like the mtx-changer script is not able to talk to your autochanger properly. I had to modify the script to work with my autochanger, and this is appears to be by design (the mtx-changer script supplied is a default that you are meant to customise for your hardware) so I guess modifying this script is the next step. If the script was present and working with your old Bacula set up, that might be a good place to start looking to see what has changed. > ̣root@LTOMag scripts]# ./mtx-changer /dev/sg3 unload 1 /dev/st0 0 > Unloading drive 0 into Storage Element 1... > mtx: Request Sense: Sense Key=Illegal Request > MOVE MEDIUM from Element Address 256 to 4096 Failed I don't know much about low level autochanger commands, but it seems odd that telling the autochanger to put the tape in slot 1 results in a command asking it to put the tape in "element 4096", which the autochanger says is an illegal request. Maybe that's normal but it seems a bit weird. I would start by modifying mtx-changer to output the commands it's trying to run, and then running those commands manually, tweaking them until you figure out how they are supposed to work, and then applying those changes back to mtx-changer. This is assuming nobody else has posted a working mtx-changer script for your autochanger already of course! > I split my bacula-dir into components so here are the relevant ones Everything else here looks like it should be fine. Once you get mtx-changer working properly I think everything else should be fine. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to LTO-8
> Automatically selected Storage: LTO-6
This seems to suggest that the Director is configured to use a device
called "LTO-6". Care to post the bacula-dir.conf? I know you said
there's no mention of "LTO-6" here, but perhaps you need to mention the
storage device specifically? With your config names, my bacula-dir.conf
would say this:
JobDefs {
Name = "DefaultJob"
Type = Backup
Storage = Autochanger
...
This would tell Bacula to try to write to "Autochanger" rather than
"LTO-6".
> Connecting to Storage daemon LTO-6 at ltomag.asaambiental.mx:9103 ...
> 3999 Device "LTO-6" not found or could not be opened.
This would make sense because your Storage device is called "LTO-8" so
it can't find one called "LTO-6".
In my opinion I'd rename your device to use something other than
"LTO-8" otherwise you can't tell whether it's referring to the device
or the media format:
Autochanger {
Device = ltomag-drive
...
}
Device {
Name = ltomag-drive
Drive Index = 0
Media Type = LTO-7
...
Here, you will know "ltomag-drive" means the Device section, while
"LTO-*" means a media type.
> How can I get bacula to acknowledge the LTO-8 drive with the LTO-7
> tapes???
This should all be automatic. In my own setup I'm using an LTO-5
autochanger with LTO-4 tapes, and it works fine. I don't think Bacula
actually checks the LTO media, I suspect it just uses the "Media Type"
you've specified in the "Device" config. In my above example I wrote
"LTO-7", but think this just marks any media that passes through the
drive as LTO-7, regardless of what it actually is.
I don't think this will cause any problems unless you actually have a
single Bacula setup that uses multiple tape types at the same time,
and it needs to look at the Media Type in order to decide which drive
to load the tape into.
Cheers,
Adam.
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Volume data error at 0:0! Wanted ID: "BB02", got "". Buffer discarded.
Very interesting! Thanks for letting us know about the solution. > Next I tried a package update (Arch Linux pacman -Syu) and reboot. This > resolved all issues! This server is kept up-to-date, so the only change > of significance was a new kernel version (4.19.12). Any idea what else was upgraded? Just wondering whether any system libraries that Bacula uses could also be involved. I had a quick look at the kernel history but couldn't really see anything that might cause incorrect data to be read back from a tape. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Volume data error at 0:0! Wanted ID: "BB02", got "". Buffer discarded.
> Last month I upgraded to Bacula 9.2.2. Backups worked without error but > I noticed "list volumes" reporting a 25 TB volume (not 6 TB as usual). A > subsequent test restore failed due to a volume data error. > > Downgrading back to Bacula 7.4.5 did not resolve the issue. It sounds then like this error existed before the Bacula upgrade. Are you encrypting your backups with the LTO native drive encryption? Could the tape have been originally labelled and used with an encryption key loaded, which was then changed (or removed) causing the tape to become unreadable? Are you trying to recover data from this tape, or are you just trying to work out why it is not working? If you don't need to recover data from it, I'd suggest rewinding the tape and writing some EOF marks to make the tape look empty, then relabelling it through Bacula. If the tape is in good condition, this should allow it to be reused, but of course doesn't explain what happened in the first place for it to get into this state. > btape: btape.c:612-0 Wrote 1 EOF to "LTO-7" (/dev/nst0) > btape: btape.c:1219-0 Rewind OK. > 29-Dec 11:19 btape JobId 0: Fatal error: block_util.c:425 Volume data > error at 0:0! Wanted ID: "BB02", got "". Buffer discarded. > btape: btape.c:1236-0 Read block 1 failed! ERR=block_util.c:425 Volume > data error at 0:0! Wanted ID: "BB02", got "". Buffer discarded. I'm not familiar with exactly how btape works, but this suggests that writing the labelling block failed - the data written could not be read back again. It suggests something wrong with the tape. Presumably if you ran this test with another tape then it would complete successfully. If you have IBM's diagnostic tools installed (itdt) you could see whether that has anything you can run to check the health of the tape. I'm not familiar with IBM's tools, but the HP Tape Tools diagnostic does have a test for tape quality. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
> I once tried MariaDB and found that it cannot be installed on the same > machine with MySQL unless you do a lot of tweaking at a very low level. > Currently I have both Postgres and MySQL installed on the same machines, > so supporting an additional DB is painful. MariaDB is a fork of MySQL, maintained by the original MySQL developers. Oracle continues to maintain MySQL, but MariaDB is the "spiritual" successor of MySQL. "MySQL" was named after the creator's first daughter, "My", and so since Oracle now owns this name the fork has been named after his second daughter instead, "Maria". The reason it is so hard to install MariaDB alongside MySQL is the same as why it would be difficult to install Bacula 9.4 and 9.2 at the same time - they are both different versions of the same project. > I expect that if MariaDB replaces all the MySQLs, eventually if no > community patches come for MariaDB I will be forced to add support for > it myself ... MariaDB has already replaced MySQL in most Linux distributions, at least those aimed at the desktop. There have been very few compatibility problems. I am currently running Bacula with MariaDB on the Arch Linux distribution and haven't encountered any issues so far, but then my backups are only home-user sized. As long as Bacula sticks to the common feature set and doesn't use any of the new Oracle-MySQL-specific features then it will probably continue to work just fine with both MySQL and MariaDB for some time yet, with no special maintenance effort. If you did want to test against MySQL and MariaDB separately, this is something you could easily do with Docker containers, as you can install whichever versions of whichever packages you like inside the containers, without affecting anything on your host machine. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Catalog from old tape ?
> to open device "Default" (/dev/nst0): ERR=Device or resource busy Something else is running that is accessing the tape. Have you unmounted the tape or temporarily stopped the bacula-sd daemon? Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape shoe-shining with spooling enabled - why?
> So it looks like when Bacula/btape writes an EOF mark, this causes the > buffers to completely flush out and the drive then runs out of data > and is forced to stop and shoe-shine. > > I've had a look through the storage daemon options but I can't see > anything relevant. I take it disabling the writing of EOF marks is > not an option? Is there any way to somehow disable flushing on EOF? So some investigation reveals that the MTWEOF ioctl does indeed flush the buffer. From the Linux st.txt: Writing filemarks without the immediate bit set in the SCSI command block acts as a synchronization point, i.e., all remaining data form the drive buffers is written to tape before the command returns. This makes sure that write errors are caught at that point, but this takes time. In some applications, several consecutive files must be written fast. The MTWEOFI operation can be used to write the filemarks without flushing the drive buffer. Writing filemark at close() is always flushing the drive buffers. However, if the previous operation is MTWEOFI, close() does not write a filemark. This can be used if the program wants to close/open the tape device between files and wants to skip waiting. It looks like others have encountered the same issue with other software[1], and switching to MTWEOFI fixed the problem there as well. So it looks like the solution is to change Bacula's tape_dev.c:951 from using MTWEOF to MTWEOFI. Support for MTWEOFI was added to the kernel[2] back in October 2010, and looks like it was first released in 2.6.37, so it's not something new. I am not familiar with Bacula's cross-platform support, so is this something that can simply be changed? Or could it just be wrapped by an `#ifdef MTWEOFI`? Cheers, Adam. [1] http://www.syncer.de/?p=862 [2] https://github.com/torvalds/linux/commit/3e51d3c924aea8a1f1372e6c615b0a37b528121d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape shoe-shining with spooling enabled - why?
> > I think I have everything set up, all the tapes are labelled and Bacula > > switches tapes when each one fills up, however after the first tape > > started writing at 80MB/sec, the rest of them only write at 60-65MB/sec. > > [...] > > Are you sure that both the drive and the tapes are still in a decent > shape, not being worn-out? Have you tested the drive with btape? If yes, > what was the result? The drive passes all the vendor tests without any issues and the tapes are brand new, so there should be no issues there. I had not thought to try the btape program, thank you for pointing that out. Combined with the "tapestat" program which prints throughput in MB/sec, it has revealed some interesting behaviour and the probable cause of the issue. btape: btape.c:1074-0 Test with random data, should give the minimum throughput. btape: btape.c:911-0 Begin writing 8 files of 1.073 GB with raw blocks of 1048576 bytes. +++ btape: btape.c:612-0 Wrote 1 EOF to "peapod-d1" (/dev/tape/by-id/scsi-350014380125815e1-nst) btape: btape.c:411-0 Volume bytes=1.073 GB. Write rate = 82.59 MB/s ++ btape: btape.c:612-0 Wrote 1 EOF to "peapod-d1" (/dev/tape/by-id/scsi-350014380125815e1-nst) btape: btape.c:411-0 Volume bytes=1.073 GB. Write rate = 67.10 MB/s ++ btape: btape.c:612-0 Wrote 1 EOF to "peapod-d1" (/dev/tape/by-id/scsi-350014380125815e1-nst) btape: btape.c:411-0 Volume bytes=1.073 GB. Write rate = 71.58 MB/s ++ btape: btape.c:612-0 Wrote 1 EOF to "peapod-d1" (/dev/tape/by-id/scsi-350014380125815e1-nst) btape: btape.c:411-0 Volume bytes=1.073 GB. Write rate = 67.10 MB/s [...] btape: btape.c:385-0 Total Volume bytes=8.589 GB. Total Write rate = 70.40 MB/s At each point where btape says it wrote an EOF mark, the throughput drops to zero, the drive stops, backtracks, then starts again with the next file. The throughput btape reports for the second and subsequent files is slower, because it is including the tape backtrack time in the overall calculation. So it looks like when Bacula/btape writes an EOF mark, this causes the buffers to completely flush out and the drive then runs out of data and is forced to stop and shoe-shine. To confirm this, instead of writing 8x 1GB files as above, I tried writing 1x 8GB file, and sure enough there is no shoe-shining: btape: btape.c:1074-0 Test with random data, should give the minimum throughput. btape: btape.c:911-0 Begin writing 1 files of 8.589 GB with raw blocks of 1048576 bytes. + btape: btape.c:612-0 Wrote 1 EOF to "peapod-d1" (/dev/tape/by-id/scsi-350014380125815e1-nst) btape: btape.c:411-0 Volume bytes=8.589 GB. Write rate = 110.1 MB/s btape: btape.c:385-0 Total Volume bytes=8.589 GB. Total Write rate = 110.1 MB/s I've had a look through the storage daemon options but I can't see anything relevant. I take it disabling the writing of EOF marks is not an option? Is there any way to somehow disable flushing on EOF? I'm running Linux kernel 4.18.16 if that's relevant. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape shoe-shining with spooling enabled - why?
> > I did not expect the drive to shoe-shine many times during the same > > despool operation. Do you have any idea why this might be > > happening? > > Are you using the -nst device (non-rewindable) configuration on SD > Archive Device directive value? Yes I am. Sorry, when I say "rewind" it is only backtracking for maybe one second or so to search for the end of the data, ready to append again. It is proper "shoe-shining", the drive behaves exactly the same if I write data to it with another application but write too slowly. Eventually the drive stops, rewinds a little, then gets ready to go again when more data comes through. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape shoe-shining with spooling enabled - why?
Hi Heitor, Thanks for your reply! > > Listening to the unit the drive stops, rewinds, then starts again > > quite a lot, so the drop in write speed appears to be due to > > shoe-shining. > > For a continuous tape writing with spooling spool size should be at > least the same size of the tape. IMHO this is not shoe-shining, but > Bacula just stop writing to tape when data is being spooled. > Shoe-shining is when you provide a very lower writing stream for the > tape and it is unable to fill the tape strip in the first pass. This > is not the case. This also happens - at the end of the 8GB spool file Bacula stops writing to the tape and starts a new spool file, however this is not what I am having a problem with (at least not yet!) My issue is that while writing the spool file to tape, the tape stops and starts frequently. The spool file is 8GB, and the tape stops, rewinds, and restarts approximately 8-10 times during each despool operation. This is definitely shoe-shining, as the despool operation is supposed to happen in a single pass, with no stopping or rewinding until each despool has finished. After despooling has finished the drive is idle until the next despooling operation but this is expected. Because my spool file is 8GB, the drive should only stop writing after each 8GB of data, but it seems to stop more like every 1GB of data. I did not expect the drive to shoe-shine many times during the same despool operation. Do you have any idea why this might be happening? Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Tape shoe-shining with spooling enabled - why?
Hi all, I've been using Bacula for a number of years but recently got hold of an old tape autochanger with an LTO-4 drive, which I would like to use to have Bacula write to tape instead of my previous disk/file config. I think I have everything set up, all the tapes are labelled and Bacula switches tapes when each one fills up, however after the first tape started writing at 80MB/sec, the rest of them only write at 60-65MB/sec. Listening to the unit the drive stops, rewinds, then starts again quite a lot, so the drop in write speed appears to be due to shoe-shining. I have set up a spool file on my /tmp partition, which is a Linux tmpfs filesystem (data is stored in RAM), so there should be no speed issue reading or writing the spool file. Certainly Bacula writes to it at over 120MB/sec when the file daemon can deliver data fast enough, yet when despooling the shoe-shining occurs. The machine has no swap file (and 64GB RAM) so the data is definitely staying in memory the whole time. I am not sure why the system cannot basically write from memory to the tape at its full speed. The bacula-sd process only uses 30% of one CPU when writing, so it can't be an issue with CRC calculations (although I would've thought those would happen during spooling rather than despooling). In the manual there is some mention of setting the maximum block size to be larger to avoid shoe-shining, so I will try that when the current full backup has finished (probably another day or two at this rate) but I'm not sure why that would affect things either. Does anyone know why despooling 8GB of data from memory onto the tape drive would result in shoe-shining? Many thanks, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
