Re: [BackupPC-users] FS and backuppc performance
With the amount of data I reported and number of files I just have 6% of inodes occupied so I don't think that is really a problem, do you use XFS for any special purpose besides dynamic inode creation? What do you think about recovery and maintenance tools for XFS. And least but not lest don't you have a bigger processor overhead with XFS? Usually people tend to say processor is not important while backing up but from what I've seen if you have like 8 or more hosts backing up data the processor and memory are stressed up. if you have to manage a FS with a large processor demand can't this be a bottleneck? Cheers, Pedro M. S. Oliveira On Wednesday 18 March 2009 19:30:33 Carl Wilhelm Soderstrom wrote: On 03/18 05:48 , Pedro M. S. Oliveira wrote: What FS do you guys use recommend/used and why? I typically use XFS for backuppc data pools, and ext3 for the root filesystem. I don't want to run out of inodes like ext3 can do. :) -- -- Pedro M. S. Oliveira IT Consultant Email: pmsolive...@gmail.com URL: http://pedro.linux-geex.com Cellular: +351 96 5867227 -- -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] cloning the pool
Koen Linders wrote: If you want an idea what isn't possible; A year ago I tried copying a much pool much smaller too an USB disk than my current (see lower), using a Xeon 2.8 GHz/1 MB with 2 GB DDR and it ran out of memory copying via rsync -H Somewhere in the mailing is other information. Someone said he does an rsync on a 2million file pool worked perfectly for him with 2 GB of memory. Not for me. Rsync 3.x may need less memory - and the requirements may be different when the target is on a remote machine. -- Les Mikesell lesmikes...@gmail.com -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] cloning the pool
If you want an idea what isn't possible; A year ago I tried copying a much pool much smaller too an USB disk than my current (see lower), using a Xeon 2.8 GHz/1 MB with 2 GB DDR and it ran out of memory copying via rsync -H Somewhere in the mailing is other information. Someone said he does an rsync on a 2million file pool worked perfectly for him with 2 GB of memory. Not for me. Now I stop backuppc at night and do: dd if=/dev/sda5 of=/dev/sdb1 bs=4K It works perfectly. I managed to copy this pool back to another server with much bigger raid1 array formatted ext3 with same blocksize. And it works with a problem afterwards afaik. Pool is 235.52GB comprising 718235 files and 4369 directories (as of 19/3 04:12), Pool hashing gives 121 repeated files with longest chain 11, Nightly cleanup removed 6736 files of size 5.27GB (around 19/3 04:12), Pool file system was recently at 61% (19/3 10:02), today's max is 61% (19/3 04:00) and yesterday's max was 61%. Greetings, Koen Linders -Oorspronkelijk bericht- Van: stoffell [mailto:stoff...@gmail.com] Verzonden: woensdag 18 maart 2009 21:57 Aan: General list for user discussion, questions and support Onderwerp: Re: [BackupPC-users] cloning the pool I want to clone the pool to a local disk attached via USB. I can't made it with a dd because the pool is on a raid volume that don't contain the pool only. We're about to do exactly the same thing. This to get ourselves a weekly off-site copy. We will use 500 GB external disks to rsync -aH the complete backuppc directory to this disk. We will use lvm and some encrypted filesystem for enhanced security. We'll have to test it out because the wiki is not very clear about it: rsync has different limitations than cp - don't ask me whether it's better or worse. It's simply something different to try. It might be nice to have some case studies / usage scenarios on the backuppc wiki ? I'll report our experiences after we tested it all out.. cheers stoffell -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] FS and backuppc performance
On 03/19 11:56 , Pedro M. S. Oliveira wrote: With the amount of data I reported and number of files I just have 6% of inodes occupied so I don't think that is really a problem, do you use XFS for any special purpose besides dynamic inode creation? The ability to be resized while mounted is good as well; tho I don't use it much. There may be a performance improvement over ext3; tho it's very hard to say. (Backuppc is a fairly unusual load situation; and hard to benchmark well). I've not noticed a performance problem from it. I used to use reiserfs on backuppc installations; but after a couple of years, some corruption bugs turned up which made me abandon it. I didn't want to go back to the inode limitations of ext3 tho; so I went with XFS. Usually people tend to say processor is not important while backing up but Backuppc will use all the processor, ram, and disk speed you give it. I've not had a box where they weren't all pegged. I tend to limit concurrent backups to 2; maybe 3 or 4 on a really high-end box (multiple processors and a proven fast disk array); to control disk-head thrashing. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] FS and backuppc performance
Carl Wilhelm Soderstrom wrote: Backuppc will use all the processor, ram, and disk speed you give it. I've not had a box where they weren't all pegged. I tend to limit concurrent backups to 2; maybe 3 or 4 on a really high-end box (multiple processors and a proven fast disk array); to control disk-head thrashing. One thing I think is missing from backuppc that amanda has had for years is a concept of grouping (or excluding...) by network connectivity. I have a mix of local and remote targets and would like to be able to control concurrency to permit 1 or 2 local backups plus separate limits for each independent WAN path. -- Les Mikesell lesmikes...@gmail.com -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Debian Etch to Lenny problem with external disks
Les Mikesell a écrit : [...] The whole archive needs to be on the same filesystem. If you use a separate disk/partition it needs to be mounted or symlinked at the /var/lib/backuppc level. Thanks a lot, this worked out ! -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] FS and backuppc performance
2009/3/18 Pedro M. S. Oliveira pmsolive...@gmail.com: From what I've seen on the list there are some people using XFS, Ext3, and so on. What's your experience with the different file systems? What FS do you guys use recommend/used and why? We use XFS on a 3-disk raid 5 (3x500gb). Just because we're used to using XFS and it performs well with a lot of small files. cheers stoffell -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Grouping by network connectivity (aka replacement queue mechanism)
This was originally part of:
Subject: Re: [BackupPC-users] FS and backuppc performance
In-Reply-To: 49c29c96.4030...@gmail.com
I am starting a new thread on this rather then hijacking the original
thread.
On Thu, Mar 19, 2009 at 02:27:18PM -0500, Les Mikesell wrote:
Carl Wilhelm Soderstrom wrote:
Backuppc will use all the processor, ram, and disk speed you give it. I've
not had a box where they weren't all pegged. I tend to limit concurrent
backups to 2; maybe 3 or 4 on a really high-end box (multiple processors
and a proven fast disk array); to control disk-head thrashing.
One thing I think is missing from backuppc that amanda has had for years
is a concept of grouping (or excluding...) by network connectivity. I
have a mix of local and remote targets and would like to be able to
control concurrency to permit 1 or 2 local backups plus separate limits
for each independent WAN path.
I would like this too. I currently use semaphore to create a set of
available slots and lock the slot during the backup using a pre/post
dump command.
Most of our hosts are named after the site they are at:
box1.site1.example.com
box1.site2.example.com
box2.site3.example.com
etc. With semaphore I create one resource pool for each remote site
based on how many parallel backups I am willing to allow from that
site:
Semaphore site1 has 20 resources.
Resource 0 is available.
Resource 1 is available.
Resource 2 is available.
Resource 3 is available.
Resource 4 is available.
Resource 5 is available.
Resource 6 is available.
Resource 7 is available.
Resource 8 is available.
Resource 9 is available.
Resource 10 is available.
Resource 11 is available.
Resource 12 is available.
Resource 13 is available.
Resource 14 is available.
Resource 15 is available.
Resource 16 is available.
Resource 17 is available.
Resource 18 is available.
Resource 19 is available.
Semaphore site2 has 2 resources.
Resource 0 is available.
Resource 1 is taken by PID 29224.
Semaphore site3 has 2 resources.
Resource 0 is available.
Resource 1 is available.
Using some home written scripts (runUserCmds, CheckQueue), I set:
$Conf{DumpPreUserCmd} = '/etc/BackupPC/bin/runUserCmds -t $type \
-c $client -H $host -P $cmdType CheckQueue';
$Conf{DumpPostUserCmd}= '/etc/BackupPC/bin/runUserCmds -t $type \
-c $client -H $host -P $cmdType CheckQueue';
which locks one of the available semaphores if it's a PreUserCmd
and unlocks if it's a PostUserCmd. If it can't lock a semaphore, it
exits with exit code 1, and because:
$Conf{UserCmdCheckStatus} = 1;
is enabled in the config, the host is skipped for that cycle.
So it is doable in BackupPC without any core changes and the upside of
this is that you can group by factors other than remote site. The
downside is that the log file shows:
2009-03-02 08:50:07 DumpPreUserCmd returned error status 256... exiting
every time the host is scheduled to be backed up but is unable to
reserve a slot. Also you can have backups fail when they are starved
for resources.
For example:
One thing I have to watch is bandwidth usage. My plan for handling
that is to allocate bandwidth in 64KB/s (512Kb/s) chunks, and use the
CheckQueue script to determine what the bw limit is for the given host
(by scanning /etc/BackupPC/pc/hostname.pl or config.pl). Then I just
reserve the proper number of chunks to reserve that bandwidth.
So I have a site that is bw limited to 2Mb/s (approx 4 chunks), I will
allocate 4 resources in the pool for the site.
If one of the hosts (one_mb) at that site has a bwlimit of 1Mb/s, then
it won't run unless there are at least 2 free resources. So no more
than 2 512Mb/sec hosts can be running.
Semaphore does support fair queing where nothing queued after one_mb
will run till one_mb has run. This guarantees that one_mb will get run
at some point. However this doesn't work with BackupPC's queing
mechanism. With
$Conf{MaxBackups} = 8;
to keep reasonable on the system, any backup that is run and queued
waiting for a resource uses one of these 8 slots. So I could have 7
jobs waiting on a resource for site2, but yet backups for site1 and
site3 have plenty of resources available. the only way I can see
around this is to set:
$Conf{MaxBackups} = 1;
or some such number, and have an additional queue:
Semaphore actual_running_backups has 8 resources.
Resource 0 is available.
Resource 1 is taken by PID 29224.
Resource 2 is available.
Resource 3 is available.
Resource 4 is available.
Resource 5 is available.
Resource 6 is available.
Resource 7 is available.
so basically BackupPC will queue a host that needs a backup, and the
control of how many are actually running is totally external to
BackupPC. I haven't tried this yet, but I think it will work.
(BTW, semaphore is a ksh impletation of semaphores written by John
Spurgeon
[BackupPC-users] Error: auth required, but service web_backup is open/insecure
Hi everyone.
As the subject says, I've run into this issue where BackupPC believes the
connection is unsecure when it is not. I did some general debugging and info
collection...
ON BACKUPPC CLIENT
# cat /etc/hosts.allow
empty
# cat /etc/hosts.deny
empty
# /usr/bin/rsync --version
rsync version 2.6.8 protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
# cat /etc/rsyncd.conf
motd file = /etc/rsyncd/rsyncd.motd
log file= /var/log/rsyncd.log
pid file= /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
port= 873
syslog facility = local5
max connections = 4
use chroot = no
uid = root
gid = root
[web_backup]
comment = Backups Restores
path= /websites
auth users = backuppc
secrets file= /etc/rsyncd/rsyncd.secrets
# cat /etc/rsyncd/rsyncd.secrets
backuppc:password
ON BACKUPPC SERVER
Software versions are:
- Red Hat Enterprise Linux Server release 5.3
- BackupPC: v3.1.0
- File-RsyncP: v0.68
$ id
uid=101(backuppc) gid=102(backuppc) groups=102(backuppc)
$ cat /etc/hosts
192.168.0.67myserver
$ pwd
/var/lib/BackupPC/temp
$ ls -la
total 8
drwxr-xr-x 2 backuppc backuppc 4096 Mar 19 20:24 .
drwxr-x--- 8 backuppc root 4096 Mar 19 20:01 ..
$ rsync -qazrt --delete backu...@myserver::web_backup /var/lib/BackupPC/temp
Password: entered password
(a few seconds passed and then back to prompt)
$ ls -la
total 16
drwxr-xr-x 4 backuppc backuppc 4096 Mar 19 19:50 .
drwxr-x--- 8 backuppc root 4096 Mar 19 19:45 ..
drwxr-x--- 4 backuppc backuppc 4096 Mar 19 19:48 test_folder_1
drwsrws--- 4 backuppc backuppc 4096 Mar 19 19:48 www.mysite1.com
(Confirmed rsync was working and authentication was required)
$ ./BackupPC_dump -f -v myserver
cmdSystemOrEval: about to system /bin/ping -c 1 -w 3 myserver
cmdSystemOrEval: finished: got output PING myserver (192.168.0.67) 56(84) bytes
of data.
64 bytes from myserver (192.168.0.67): icmp_seq=1 ttl=64 time=0.101 ms
--- myserver ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.101/0.101/0.101/0.000 ms
cmdSystemOrEval: about to system /bin/ping -c 1 -w 3 myserver
cmdSystemOrEval: finished: got output PING myserver (192.168.0.67) 56(84) bytes
of data.
64 bytes from myserver (192.168.0.67): icmp_seq=1 ttl=64 time=0.081 ms
--- myserver ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.081/0.081/0.081/0.000 ms
CheckHostAlive: returning 0.081
full backup started for directory web_backup
started full dump, share=web_backup
Connected to myserver:873, remote version 29
Negotiated protocol version 28
Error connecting to module web_backup at myserver:873: auth required, but
service web_backup is open/insecure
Got fatal error during xfer (auth required, but service web_backup is
open/insecure)
cmdSystemOrEval: about to system /bin/ping -c 1 -w 3 myserver
cmdSystemOrEval: finished: got output PING myserver (192.168.0.67) 56(84) bytes
of data.
64 bytes from myserver (192.168.0.67): icmp_seq=1 ttl=64 time=0.095 ms
--- myserver ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.095/0.095/0.095/0.000 ms
cmdSystemOrEval: about to system /bin/ping -c 1 -w 3 myserver
cmdSystemOrEval: finished: got output PING myserver (192.168.0.67) 56(84) bytes
of data.
64 bytes from myserver (192.168.0.67): icmp_seq=1 ttl=64 time=0.082 ms
--- myserver ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.082/0.082/0.082/0.000 ms
CheckHostAlive: returning 0.082
Backup aborted (auth required, but service web_backup is open/insecure)
Not saving this as a partial backup since it has fewer files than the prior one
(got 0 and 0 files versus 0)
dump failed: auth required, but service web_backup is open/insecure
-bash-3.2$
Now, I changed this to not require authentication just for testing...
$Conf{RsyncdAuthRequired} = 0
... and I got this error (snipped only relevant info...)
$ ./BackupPC_dump -f -v myserver
...
full backup started for directory web_backup
started full dump, share=web_backup
Connected to myserver:873, remote version 29
Negotiated protocol version 28
Error connecting to module web_backup at myserver:873: unexpected response: ''
Got fatal error during xfer (unexpected response: '')
cmdSystemOrEval: about to system /bin/ping -c 1 -w 3 myserver
cmdSystemOrEval: finished: got output PING myserver (192.168.0.67) 56(84) bytes
of data.
64 bytes from myserver (192.168.0.67): icmp_seq=1 ttl=64 time=0.094 ms
So, I've confirmed everything is working on the outside of BackupPC. I found
this thread which touches on what may be the issue:
http://www.adsm.org/lists/html/BackupPC-users/2008-06/msg00295.html
However, this patch did not work for me.
