Re: [Bacula-users] Could not connect to server Director daemon ....
Hello Olaf,
Have you checked if this address/port (tux64.home.lan:9101) is not being
used by another process? Your original bacula-dir.conf (the last post) was
configured to use locahost and not tux64.home.lan as your are trying now.
Best regards,
Ana
On Sat, Jun 27, 2015 at 4:38 AM, olx69 wrote:
> Hello Ana,
>
> > I´m not sure about this, but I noticed something about the names for
> > your storage:
> >
> > ---bacula-traymon.conf--
> > Storage {
> > Name = bacula-tux64-sd
> > ...
> > }
> >
> > bacula-dir-full.conf
> > Storage {
> > Name = File
> >...
> > }
> >
> > bacula-sd-full.conf-
> >
> > Storage { # definition of myself
> >Name = bacula-tux64-sd
> >...
> > }
> >
> > Could you try "Name = bacula-tux64-sd" in your bacula-dir-full.conf or
> > "Name = File" in your bacula-traymon.conf for the storage definition?
>
> If I change this but 'bat' doesn't work any more (probably bconsole too
> - not tested). Also it doesn't change the behaviour of traymonitor.
>
> I had a look at the original bacula-dir.conf, e.g.
>
> # Definition of a second file Virtual Autochanger device
> # Possibly pointing to a different disk drive
> Storage {
>Name = File2
> # Do not use "localhost" here
>Address = localhost# N.B. Use a fully qualified name
> here
>SDPort = 9103
>Password = "@@SD_PASSWORD@@"
>Device = FileChgr2
>Media Type = File2
>Maximum Concurrent Jobs = 10# run up to 10 jobs a the same time
> }
>
> so this seems to be correct.
>
> Also I notice a warning:
> 26-Jun 19:36 bacula-tux64-dir: Warning: Cannot bind port 9101: ERR=Die
> angeforderte Adresse kann nicht zugewiesen werden: Retrying ...
>
> english: Can't assign requested address. Retrying ...
>
> Maybe it's relevant.
>
> This configuration did work with bacula 5.x, I was not aware of thise
> changes in 7.x. Maybe v5.x was more sloppy with config files ...
>
>
> Thank you,
> Olaf
>
> >
> > Best regards,
> > Ana
> >
> > On Fri, Jun 26, 2015 at 2:36 AM, olx69 > > wrote:
> >
> > Hello Ane,
> >
> > > Sorry I was out a few days Have you tested your Bacula with
> these
> > > "fake passwords"? Or you have just changed them for posting here? I
> > > found no errors in your passwords configurations, but I'm not sure
> if
> > > you are using these ones posted here. In the case you're not,
> could you
> > > verify your director to storage daemon password configuration (not
> the
> > > one for communicating with monitor)? The error message points to a
> > > mismatch here.
> >
> >
> > No, these are my real passwords! I trust in my secure LAN ;-)
> >
> > Thank you,
> > Olaf
> >
> >
> > > On Tue, Jun 23, 2015 at 11:04 AM, olx69 ope-li...@gmx.de>
> > > >> wrote:
> > >
> > > Am 18.06.2015 um 16:13 schrieb olx69:
> > > > Hello Ana,
> > > >
> > > >> Sorry I'm late in this post.
> > > >> It seems there is still some password mismatching
> problems.
> > > Could you
> > > >> post your resources definitions for monitor (in
> > bacula-dir.conf,
> > > >> bacula-fd.conf and bacula-sd.conf) and your
> > tray-monitor.conf?
> > > >
> > > > thank you for investigating! attached the files,
> > >
> > > no idea? Here the reduced password related part. Imo all is
> > correct,
> > > isn't it?
> > >
> > >
> > > ---bacula-traymon.conf--
> > > Monitor {
> > > Name = bacula-tux64-mon
> > > Password = "MON_DIR_PASSWORD" # password for the
> > Directors
> > > RefreshInterval = 3 seconds
> > > }
> > > Client {
> > > Name = bacula-tux64-fd
> > > Address = tux64.home.lan
> > > Password = "MON_FD_PASSWORD" # password for
> > FileDaemon
> > > }
> > > Storage {
> > > Name = bacula-tux64-sd
> > > Address = tux64.home.lan
> > > Password = "MON_SD_PASSWORD" # password for
> > StorageDaemon
> > > }
> > > Director {
> > > Name = bacula-tux64-dir
> > > Address = tux64.home.lan
> > > }
> > > bacula-dir-full.conf
> > > Director {
> > > Name = bacula-tux64-dir
> > > Password = "DIR_PASSWORD"# BConsole password
> > >
> > > }
> > > Console {
> > > Name = bacula-tux64-mon
> > > Password = "MON_DIR_PASSWORD"
> > > ...
> > > }
> > > Client {
> > > Name = bacula-tux64-fd
> > > Password = "FD_PASSWORD"# password for
> FileDaemon
> > > ...
> > > }
> > >
Re: [Bacula-users] Bacula causing high disk-io on clients
I wonder if the thread starter (SPQR) solved his problem. The discussion became hot and he flew. lol === Heitor Medrado de Faria - LPIC-III | ITIL-F | Bacula Systems Certified Administrator II Do you need Bacula training? https://www.udemy.com/bacula-backup-software/?couponCode=bacula-list +55 61 8268-4220 Site: http://bacula.us FB: heitor.faria === - Original Message - > From: "Dmitri Maziuk" > To: "bacula-users" > Sent: Saturday, June 27, 2015 12:14:47 PM > Subject: Re: [Bacula-users] Bacula causing high disk-io on clients > On 6/27/2015 8:55 AM, Alex Domoradov wrote: >> FYI >> >> I have 1Gb uplinks between bacula sd and client and get the following >> results >> >> Compression: NONE >> Time: 07:16:58 >> Size: 831.14 GB >> Files: 11,288,747 >> Speed: 32.46 MB/s >> Compression: 0.00 >> >> Compression: LZO >> Time: 07:56:38 >> Size: 653.04 GB >> Files: 11,288,747 >> Speed: 23.38 MB/s >> Compression: 0.21 >> >> Compression: GZIP >> Time: 10:09:07 >> Size: 636.41 GB >> Files: 11,288,747 >> Speed: 17.83 MB/s >> Compression: 0.23 > > What that means is over a gigabit link you can read from 3 clients in > parallel w/o compression, 4 clients w/ lzo compression, and about 6 w/ > gzip. I agree, if you have a single roid-warrior fablet client who shows > up on office vlan once a month, client-side compression is wrong for > them. I would say as is bacula: get them btsync or something. I'm > backing up linux servers, plural, so I'd rather gzip on the clients. > > The only times I see high i/o load on ext4 is a) recursive chown/chmod > on a large directory tree on nfs file server (because you can't train > users to do it themselves) and b) when a cheap non-tler disk is dead but > doesn't know it yet. > > Dima > > > -- > Monitor 25 network devices or servers for free with OpManager! > OpManager is web-based network management software that monitors > network devices and physical & virtual servers, alerts via email & sms > for fault. Monitor 25 devices for free with no restriction. Download now > http://ad.doubleclick.net/ddm/clk/292181274;119417398;o > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula causing high disk-io on clients
On 6/27/2015 8:55 AM, Alex Domoradov wrote: > FYI > > I have 1Gb uplinks between bacula sd and client and get the following > results > > Compression: NONE > Time: 07:16:58 > Size: 831.14 GB > Files: 11,288,747 > Speed: 32.46 MB/s > Compression: 0.00 > > Compression: LZO > Time: 07:56:38 > Size: 653.04 GB > Files: 11,288,747 > Speed: 23.38 MB/s > Compression: 0.21 > > Compression: GZIP > Time: 10:09:07 > Size: 636.41 GB > Files: 11,288,747 > Speed: 17.83 MB/s > Compression: 0.23 What that means is over a gigabit link you can read from 3 clients in parallel w/o compression, 4 clients w/ lzo compression, and about 6 w/ gzip. I agree, if you have a single roid-warrior fablet client who shows up on office vlan once a month, client-side compression is wrong for them. I would say as is bacula: get them btsync or something. I'm backing up linux servers, plural, so I'd rather gzip on the clients. The only times I see high i/o load on ext4 is a) recursive chown/chmod on a large directory tree on nfs file server (because you can't train users to do it themselves) and b) when a cheap non-tler disk is dead but doesn't know it yet. Dima -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula causing high disk-io on clients
FYI I have 1Gb uplinks between bacula sd and client and get the following results Compression: NONE Time: 07:16:58 Size: 831.14 GB Files: 11,288,747 Speed: 32.46 MB/s Compression: 0.00 Compression: LZO Time: 07:56:38 Size: 653.04 GB Files: 11,288,747 Speed: 23.38 MB/s Compression: 0.21 Compression: GZIP Time: 10:09:07 Size: 636.41 GB Files: 11,288,747 Speed: 17.83 MB/s Compression: 0.23 On Sat, Jun 27, 2015 at 4:19 PM, Alan Brown wrote: > On 27/06/15 10:45, Dmitri Maziuk wrote: > > Compressing data on the client means fewer bytes to send over the wire. > Block-level compression like bzip2 tends to be completely cpu-bound and > anything bigger than a cellphone tends to have plenty of cycles to spare. > > > Not entirely true and certainly not accurate for Gb/s and faster links. > > Individual CPU cores haven't changed much in terms of speed for nearly a > decade and most compression algorithms are single threaded when fed from > stdin (even pigz and pbzip). On top of that you're facing memory <-> CPU > bottlenecks. > > The result when running gzip compression over a 1Gb/s link is that you'll > usually achieve a maximum end-to-end rate of 30MB/s, vs the wire speed of > 110MB/s ( b = bit, B=BYTE, aka o = octet if you're french ) > > bzip is significantly worse. Even LZO gives little gain at 1Gb/s in > exchange for making your CPU run hot and drawing more power than you need > to. > > At 10Gb/s, LZO is a significant bottleneck. > > > The takeaway: Unless you're backing up over a WAN (and a slow or > overloaded one at that), wireside compression is not a net win. > > We have 300 people on our campus linking to the main site at 1Gb/s and > whilst backing up machines on the main site to our satellite one we still > achieve faster backups without compression than with it. > > > The other side of compression is that it's a huge CPU drain. I already > have enough problems with people rebooting windows systems mid-backup > without adding even more incentive for them to hit the Big Red Button. > > > > -- > Monitor 25 network devices or servers for free with OpManager! > OpManager is web-based network management software that monitors > network devices and physical & virtual servers, alerts via email & sms > for fault. Monitor 25 devices for free with no restriction. Download now > http://ad.doubleclick.net/ddm/clk/292181274;119417398;o > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > > -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula causing high disk-io on clients
On 6/27/2015 5:45 AM, Dmitri Maziuk wrote: > On 6/26/2015 7:26 AM, Josh Fisher wrote: > >> However, for backup devices lacking hardware compression (such as disk), >> compression may be warranted regardless of client connection speed. This >> is why a SD level compression feature would be useful. > Compressing data on the client means fewer bytes to send over the wire. > Block-level compression like bzip2 tends to be completely cpu-bound and > anything bigger than a cellphone tends to have plenty of cycles to > spare. SD level compression is saving bandwidth where it's scarce at the > cost of cycles that are abundant -- why would anyone *not* want it? Weak client machines on a fast (1Gbps or better) network. By weak clients I mean ARM and Atom based pads and small laptops. The road warrior users must be backed up when they are in the office and actively using the devices. On these clients it is definitely noticeable. I have one user who figured out how to kill the bacula-fd client when it slowed his device down. -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula causing high disk-io on clients
On 27/06/15 10:45, Dmitri Maziuk wrote: > Compressing data on the client means fewer bytes to send over the > wire. Block-level compression like bzip2 tends to be completely > cpu-bound and anything bigger than a cellphone tends to have plenty of > cycles to spare. Not entirely true and certainly not accurate for Gb/s and faster links. Individual CPU cores haven't changed much in terms of speed for nearly a decade and most compression algorithms are single threaded when fed from stdin (even pigz and pbzip). On top of that you're facing memory <-> CPU bottlenecks. The result when running gzip compression over a 1Gb/s link is that you'll usually achieve a maximum end-to-end rate of 30MB/s, vs the wire speed of 110MB/s ( b = bit, B=BYTE, aka o = octet if you're french ) bzip is significantly worse. Even LZO gives little gain at 1Gb/s in exchange for making your CPU run hot and drawing more power than you need to. At 10Gb/s, LZO is a significant bottleneck. The takeaway: Unless you're backing up over a WAN (and a slow or overloaded one at that), wireside compression is not a net win. We have 300 people on our campus linking to the main site at 1Gb/s and whilst backing up machines on the main site to our satellite one we still achieve faster backups without compression than with it. The other side of compression is that it's a huge CPU drain. I already have enough problems with people rebooting windows systems mid-backup without adding even more incentive for them to hit the Big Red Button. -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Performance settings for large file LTO-6 backup
On 6/27/2015 1:37 AM, Andrew Noonan wrote: > On Fri, Jun 26, 2015 at 2:17 PM, Ana Emília M. Arruda > wrote: >> Are you going to generate a .tar of about 250TB every day? Which will >> be the nature of your restores? You´re going to need always the >> restore of the whole data set or occasionally you will need to >> restore a small set of files? > After the backfill, I expect the daily backups to be in the hundreds > of gigs a day range. During the backfill, I'll want to maximize > writes to catch up as soon as I can. In general, we already have a > copy of the data, but given how it syncs, this doesn't protect against > an accidental "rm" getting synced downstream, so I'd imagine other > then for testing purposes, restores will be for accidental deletes or > for some sort of massive disaster situation. This is why I figure the > backfill should be close to the size of the tape, though I'm not 100%. Keep in mind that one file spanning N tapes is N times more likely to fail. The loss of a single tape results in the loss of the file. For this reason, I prefer to break large data sets like that into N different backup jobs so that a job's full backup fits on a single tape, if at all possible. Tapes do sometimes fail after they have been successfully written. -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula causing high disk-io on clients
On 6/26/2015 7:26 AM, Josh Fisher wrote: > However, for backup devices lacking hardware compression (such as disk), > compression may be warranted regardless of client connection speed. This > is why a SD level compression feature would be useful. Compressing data on the client means fewer bytes to send over the wire. Block-level compression like bzip2 tends to be completely cpu-bound and anything bigger than a cellphone tends to have plenty of cycles to spare. SD level compression is saving bandwidth where it's scarce at the cost of cycles that are abundant -- why would anyone *not* want it? Dima -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Could not connect to server Director daemon ....
Hello Ana,
> I´m not sure about this, but I noticed something about the names for
> your storage:
>
> ---bacula-traymon.conf--
> Storage {
> Name = bacula-tux64-sd
> ...
> }
>
> bacula-dir-full.conf
> Storage {
> Name = File
>...
> }
>
> bacula-sd-full.conf-
>
> Storage { # definition of myself
>Name = bacula-tux64-sd
>...
> }
>
> Could you try "Name = bacula-tux64-sd" in your bacula-dir-full.conf or
> "Name = File" in your bacula-traymon.conf for the storage definition?
If I change this but 'bat' doesn't work any more (probably bconsole too
- not tested). Also it doesn't change the behaviour of traymonitor.
I had a look at the original bacula-dir.conf, e.g.
# Definition of a second file Virtual Autochanger device
# Possibly pointing to a different disk drive
Storage {
Name = File2
# Do not use "localhost" here
Address = localhost# N.B. Use a fully qualified name here
SDPort = 9103
Password = "@@SD_PASSWORD@@"
Device = FileChgr2
Media Type = File2
Maximum Concurrent Jobs = 10# run up to 10 jobs a the same time
}
so this seems to be correct.
Also I notice a warning:
26-Jun 19:36 bacula-tux64-dir: Warning: Cannot bind port 9101: ERR=Die
angeforderte Adresse kann nicht zugewiesen werden: Retrying ...
english: Can't assign requested address. Retrying ...
Maybe it's relevant.
This configuration did work with bacula 5.x, I was not aware of thise
changes in 7.x. Maybe v5.x was more sloppy with config files ...
Thank you,
Olaf
>
> Best regards,
> Ana
>
> On Fri, Jun 26, 2015 at 2:36 AM, olx69 > wrote:
>
> Hello Ane,
>
> > Sorry I was out a few days Have you tested your Bacula with these
> > "fake passwords"? Or you have just changed them for posting here? I
> > found no errors in your passwords configurations, but I'm not sure if
> > you are using these ones posted here. In the case you're not, could you
> > verify your director to storage daemon password configuration (not the
> > one for communicating with monitor)? The error message points to a
> > mismatch here.
>
>
> No, these are my real passwords! I trust in my secure LAN ;-)
>
> Thank you,
> Olaf
>
>
> > On Tue, Jun 23, 2015 at 11:04 AM, olx69
> > >> wrote:
> >
> > Am 18.06.2015 um 16:13 schrieb olx69:
> > > Hello Ana,
> > >
> > >> Sorry I'm late in this post.
> > >> It seems there is still some password mismatching problems.
> > Could you
> > >> post your resources definitions for monitor (in
> bacula-dir.conf,
> > >> bacula-fd.conf and bacula-sd.conf) and your
> tray-monitor.conf?
> > >
> > > thank you for investigating! attached the files,
> >
> > no idea? Here the reduced password related part. Imo all is
> correct,
> > isn't it?
> >
> >
> > ---bacula-traymon.conf--
> > Monitor {
> > Name = bacula-tux64-mon
> > Password = "MON_DIR_PASSWORD" # password for the
> Directors
> > RefreshInterval = 3 seconds
> > }
> > Client {
> > Name = bacula-tux64-fd
> > Address = tux64.home.lan
> > Password = "MON_FD_PASSWORD" # password for
> FileDaemon
> > }
> > Storage {
> > Name = bacula-tux64-sd
> > Address = tux64.home.lan
> > Password = "MON_SD_PASSWORD" # password for
> StorageDaemon
> > }
> > Director {
> > Name = bacula-tux64-dir
> > Address = tux64.home.lan
> > }
> > bacula-dir-full.conf
> > Director {
> > Name = bacula-tux64-dir
> > Password = "DIR_PASSWORD"# BConsole password
> >
> > }
> > Console {
> > Name = bacula-tux64-mon
> > Password = "MON_DIR_PASSWORD"
> > ...
> > }
> > Client {
> > Name = bacula-tux64-fd
> > Password = "FD_PASSWORD"# password for FileDaemon
> > ...
> > }
> > -bacula-fd-full.conf-
> > Director {
> > Name = bacula-tux64-dir
> > Password = "FD_PASSWORD"
> > }
> > Director {
> > Name = bacula-tux64-mon
> > Password = "MON_FD_PASSWORD"
> > }
> > bacula-sd-full.conf-
> > Director {
> > Name = bacula-tux64-dir
> > Password = "SD_PASSWORD"
> > }
> > Director {
> > Name = bacula-tux64-mon
> > Password = "MON_SD_PASSWORD"
> > ...
> >
