Les Mikesell <[EMAIL PROTECTED]> wrote on 01/09/2007 01:45:10 AM:
> Timothy J. Massey wrote:
> >
> > Actually, the sluggishness comes from I/O competition, I think, not
RAM
> > or even CPU usage...
>
> What kind of disk do you have and are you sure it is using DMA?
> What does 'hdparm -T -t' show for the buffered disk speed?
I've gotten results the original server and from another server that I
just put in place. Here is a summary of the two machines, with the
information you (and others) have asked for:
Machine 1:
Via EPIA M-II Motherboard with 1GHz C3 CPU
2 x PATA IDE Hard drive
OS and BackupPC are on hda
Pool is on hdc
128MB RAM
Knoppix 3.7 (2.6.9 kernel)
BackupPC 2.1.0pl1
Transfer via rsyncd
hdparm /dev/hdc:
/dev/hdc:
multcount = 16 (on)
IO_support = 1 (32-bit)
unmaskirq = 1 (on)
using_dma = 1 (on)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 256 (on)
geometry = 24321/255/63, sectors = 200049647616, start = 0
hdparm -T -t /dev/hdc:
/dev/hdc:
Timing cached reads: 516 MB in 2.01 seconds = 257.14 MB/sec
Timing buffered disk reads: 126 MB in 3.04 seconds = 41.49 MB/sec
vmstat 3 10:
[EMAIL PROTECTED]:~# vmstat 3 10
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
1 0 33992 684 21520 23740 2 3 36 1451 2064 1855 39 9
49 3
1 0 33992 712 21528 23824 0 0 11 41 3468 4682 82
18 0 0
1 0 33992 1088 21528 23864 0 0 3 5499 3396 4497 81
19 0 0
1 0 33992 1168 21472 23660 0 0 0 1068 3412 4586 82
18 0 0
2 0 33992 964 21436 23708 0 0 1 4411 3409 4560 81
18 0 0
1 0 33992 1048 21400 23756 0 0 0 5485 3386 4544 82
18 0 0
1 0 33992 780 21344 24076 0 0 3 5 3423 4629 83
17 0 0
1 0 33992 1092 21328 23832 0 0 0 5543 3524 4796 81
19 0 0
1 0 33992 948 21336 24084 0 0 0 1011 3531 4802 80
19 1 0
1 0 33992 744 21304 24116 0 0 0 4753 3392 4569 81
19 0 0
top:
top - 21:09:02 up 3:55, 2 users, load average: 1.15, 1.12, 1.06
Tasks: 45 total, 2 running, 42 sleeping, 0 stopped, 1 zombie
Cpu(s): 82.1% us, 11.3% sy, 0.0% ni, 0.0% id, 0.3% wa, 2.7% hi, 3.7% si
Mem: 109068k total, 108408k used, 660k free, 21416k buffers
Swap: 385552k total, 33992k used, 351560k free, 24004k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2032 backuppc 25 0 63328 41m 3160 R 97.7 39.3 82:45.61 BackupPC_dump
44 root 15 0 0 0 0 S 0.7 0.0 0:32.12 pdflush
2022 backuppc 15 0 59552 39m 3160 S 0.7 37.1 3:41.52 BackupPC_dump
2063 root 16 0 2096 1032 1888 R 0.7 0.9 0:00.06 top
45 root 15 0 0 0 0 S 0.3 0.0 0:43.85 kswapd0
1 root 16 0 76 40 56 S 0.0 0.0 0:02.67 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 events/0
5 root 5 -10 0 0 0 S 0.0 0.0 0:00.02 khelper
6 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
33 root 5 -10 0 0 0 S 0.0 0.0 0:00.03 kblockd/0
43 root 15 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
46 root 10 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
631 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
659 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 ata/0
711 root 16 0 0 0 0 S 0.0 0.0 0:00.05 kjournald
Target server:
IBM xSeries 225
2.66GHz Xeon
512MB RAM
RHEL 3
3 x 10,000RPM SCSI drives in RAID-5
IBM ServeRAID family controller
I do not have hard performance information from the target server. I
will try to get it tonight. IIRC, there is significant CPU usage, but
not 100%.
Backup performance:
Totals Existing Files New Files
Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB
0 full 241021 160049.8 2.97 58068 15233.0 222383 144825.2
So: 2.97MB/s, with 100% CPU load on my backup server.
Machine 2:
Via EPIA M-II Motherboard with 1GHz C3 CPU
PATA IDE Hard drive x 1
512MB RAM
CentOS 3.8
BackupPC 3.0beta3
Transfer via rsyncd
hdparm /dev/hda:
/dev/hda:
multcount = 64 (on)
IO_support = 1 (32-bit)
unmaskirq = 1 (on)
using_dma = 1 (on)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 60801/255/63, sectors = 976773168, start = 0
hdparm -T -t /dev/hda:
/dev/hda:
Timing buffer-cache reads: 484 MB in 2.00 seconds = 242.00 MB/sec
Timing buffered disk reads: 160 MB in 3.02 seconds = 52.98 MB/sec
vmstat 3 10:
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us
sy id wa
1 0 0 9704 142252 294312 0 0 58 303 266 195 2
1 96 1
1 2 0 5468 142260 298644 0 0 0 4447 3518 4640 38
8 53 1
0 0 0 5616 142356 298796 0 0 3 15 2696 3484 53
10 34 3
2 0 0 5420 142540 298496 0 0 3 4109 2541 3166 44
8 42 6
1 0 0 5612 142664 298152 0 0 3 0 2483 3220 55
8 37 0
2 0 0 5376 142792 298540 0 0 0 3692 2462 2949 65
8 25 2
2 2 0 5244 142904 298360 0 0 3 3759 2882 3729 50
12 37 1
2 0 0 5824 143012 297628 0 0 4 79 2901 3668 51
9 39 1
1 0 0 5392 143136 297512 0 0 3 4020 2848 3594 49
10 38 3
0 0 0 5396 143184 297052 0 0 1 0 2677 3476 38
10 52 0
top:
14:38:38 up 14:01, 1 user, load average: 0.50, 0.41, 0.20
50 processes: 47 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 36.5% 0.0% 7.7% 4.9% 5.7% 3.1% 41.7%
Mem: 495756k av, 490496k used, 5260k free, 0k shrd, 144552k
buff
261396k actv, 158896k in_d, 6752k in_c
Swap: 2048248k av, 0k used, 2048248k free 294000k
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
10305 backuppc 25 0 15552 15M 2096 R 43.7 3.1 1:41 0
BackupPC_dump
9676 root 15 0 0 0 0 SW 0.1 0.0 0:07 0 kjournald
10301 backuppc 15 0 10568 10M 2088 S 0.1 2.1 0:07 0
BackupPC_dump
10311 root 15 0 1084 1084 888 R 0.1 0.2 0:00 0 top
1 root 15 0 496 496 436 S 0.0 0.1 0:05 0 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd
4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0
ksoftirqd/0
7 root 15 0 0 0 0 SW 0.0 0.0 0:22 0 bdflush
5 root 15 0 0 0 0 SW 0.0 0.0 0:35 0 kswapd
6 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand
8 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0
mdrecoveryd
13 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
68 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
Target server:
IBM NetVista desktop computer
1.6GHz Pentium 4
512MB RAM
Windows Server 2003
IDE drive
Intel controller in Ultra DMA mode 5
I don't know how to get vmstat-like text performance information from a
Windows system. However, watching the CPU graph, it is high. It's not
flatlined, but it bounces consistently between 80 and 100%.
Backup performance:
Totals Existing Files New Files
Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB
0 full 20585 5995.1 2.36 5773 1252.9 16408 4742.9
While I don't get 100% CPU usage on this backup server, I get an even
worse transfer rate on this configuration. My guess is that this is
because of the much lower specs of the target machine in this case.
In any case, all of my backups seem to be pretty slow compared to what
others claim to get. I have *no* idea how people could be getting
sufficient power from Pentium Pro 200MHz machines! Maybe they're not
using rsyncd? Let's see how that affects things:
Copying a 666MB file from my file server to the target server of Machine
2 (my workstation) gets 9MB/s at about 30% CPU load peak. Obviously,
the computer, network and hard drive are certainly capable of moving
data much faster than BackupPC is letting it. I then used smbclient on
my BackupPC server to copy the same file from the target computer to
Machine 2 (just like it would during a backup, but with SMB):
smb: \> get KnoppMythR5D1.iso
getting file \KnoppMythR5D1.iso of size 666017792 as KnoppMythR5D1.iso
(7039.9 kb/s) (average 7039.9 kb/s)
Here's a vmstat 3 10 during the copy:
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us
sy id wa
1 0 0 5544 153912 298764 0 0 64 326 291 214 3
1 95 1
1 0 0 5360 153928 299188 0 0 0 11603 7436 8573 4
22 49 25
0 2 0 5328 153948 298960 0 0 0 10825 7493 8675 3
22 52 23
1 0 0 5380 153976 299084 0 0 0 583 9185 10830 5
23 70 3
1 0 0 5372 154004 299180 0 0 0 11216 7458 8617 3
22 50 25
1 0 0 5320 154032 299028 0 0 1 553 9328 11012 2
26 71 1
1 0 0 5316 154060 299144 0 0 0 10783 8533 9874 3
24 58 15
0 2 0 5268 154076 298948 0 0 0 11755 7313 8473 4
23 49 25
1 0 0 5276 154104 299104 0 0 0 580 8978 10590 5
29 61 6
1 0 0 5276 154124 299100 0 0 0 11451 9208 10647 3
25 53 18
Those are much better-looking numbers: it's triple what I'm getting with
BackupPC, and plenty of idle CPU!
Now, here's the speed in doing a straight command-line rsync from
Machine #2 pulling the same file from the target server, using the same
settings as BackupPC:
This time, the Windows client was flat-out pegged at 100%. Here's a
vmstat 3 10 from the backup server:
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us
sy id wa
3 0 0 5352 158204 294240 0 0 82 331 299 223 3
1 95 1
1 0 0 5380 158220 294280 0 0 0 4993 3529 4109 5
7 76 12
1 0 0 5412 158236 294112 0 0 0 4549 3536 4120 9
8 73 9
1 0 0 5280 158244 294328 0 0 0 480 3476 4191 8
6 86 0
0 0 0 5468 158260 294080 0 0 0 5007 3520 4121 5
7 78 9
0 0 0 5232 158268 294328 0 0 0 0 3479 4184 6
10 85 0
1 0 0 5456 158280 294144 0 0 0 5028 3529 4109 6
8 78 8
1 0 0 5316 158296 294176 0 0 0 4552 3534 4103 12
8 72 8
1 0 0 5396 158308 294180 0 0 0 475 3471 4155 6
11 83 0
1 0 0 5480 158320 294040 0 0 0 4953 3447 4015 4
5 80 11
The backup server has plenty of CPU left. Obviously, the limit is on
the target computer. Throughput is horrible:
[EMAIL PROTECTED] data]# date; rsync
[EMAIL PROTECTED]::D/KnoppMythR5D1.iso .; date
Tue Jan 9 15:05:42 EST 2007
Tue Jan 9 15:09:32 EST 2007
[EMAIL PROTECTED] data]#
That's a measly 2.76MB/s, which is very much in line with my BackupPC
results.
So, it seems to me that the culprit is rsync. I think the reason my
production backup servers are usually at 100% CPU utilization is that
they're backing up reasonably high-performance file servers that have
enough CPU power to max out my backup server. It will be interesting to
see how much CPU load is put on the target of Machine 1: I will check
tonight.
I guess the best way to improve this would be to avoid rsync...
However, I like rsyncd. I never realized how heavy the overhead is with
rsync, though. Unless I'm missing something?
Thoughts?
Tim Massey
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
BackupPC-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
