On 5/9/19 1:49 AM, Lothar Schilling wrote:
Hi everybody,
for years I have used CentOS for our server landscape. Now I decided to
give Debian a try. I just set up a Stretch 9.8 system supposed to become
our main backup server. So I set up a backup job wih rsync. But the
going is really very very slooooow. Trying to figure out what's happening:
* iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
* dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
that's fine as well
But whenever I try to rsync, cp or scp - whether copy files on the local
hard disk only or over the network - speed goes down to 500 kB/s.
Filesystem is ext4.
I don't have any clue about what's going on. Any kind of help would be
appreciated, thank you!
Lothar
On 5/9/19 2:46 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:14
schrieb Jonas Smedegaard:
>> Is _only_ transfer speed affected?
>>
>> I am no expert in this, but imagine that if you rsync massive amounts
>> involving hardlinks then memory becomes a problem too.
>>
>> Perhaps run atop to monitor bottlenecks live
>>
>>
>> - Jonas
>>
> It is most definitely not a memory or cpu problem. It's a HP Proliant
> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
> problem stays the same if I just copy one large file.
On 5/9/19 3:26 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:51
schrieb Kevin DAGNEAUX:
>> Check if it's related to the disk speed (hdparm and/or iotop).
>>
>> Kevin
>>
>
> hdparm -tT /dev/sda
> /dev/sda:
> Timing cached reads: 13348 MB in 2.00 seconds = 6683.42 MB/sec
> Timing buffered disk reads: 1014 MB in 3.00 seconds = 337.72 MB/sec
>
> iotop -o (for rsync and cp)
> Total DISK READ : 0.00 B/s | Total DISK WRITE : 476.15 K/s
> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 487.86 K/s
> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
> 19531 be/4 root 0.00 B/s 476.15 K/s 0.00 % 99.24 % rsync
> --info=progress2 /daten/testfile /daten/testfile2
>
> iotop -o (for dd)
> Total DISK READ : 0.00 B/s | Total DISK WRITE : 297.68 M/s
> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 297.68 M/s
> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
> 19557 be/4 root 0.00 B/s 297.68 M/s 0.00 % 99.99 % dd
> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
On 5/9/19 5:43 AM, Lothar Schilling wrote:> Am 09.05.2019 um 13:27
schrieb Martin:
>> [..]
>>> hdparm -tT /dev/sda
>>> /dev/sda:
>>> Timing cached reads: 13348 MB in 2.00 seconds = 6683.42 MB/sec
>>> Timing buffered disk reads: 1014 MB in 3.00 seconds = 337.72 MB/sec
>>>
>>> iotop -o (for rsync and cp)
>>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 476.15 K/s
>>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 487.86 K/s
>>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
>>> 19531 be/4 root 0.00 B/s 476.15 K/s 0.00 % 99.24 % rsync
>>> --info=progress2 /daten/testfile /daten/testfile2
>>>
>>> iotop -o (for dd)
>>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 297.68 M/s
>>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 297.68 M/s
>>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
>>> 19557 be/4 root 0.00 B/s 297.68 M/s 0.00 % 99.99 % dd
>>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>> Show us the 'dd if=/daten/testfile bs=1G oflag=direct of=/dev/null',
please.
>> If this is as slow as this ~480k/s above, check your disk's health
status. Like with smartmontools or some disk-utility software.
>>
>> Martin
>>
> Fast enough...
>
> dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
> 10+0 Datensätze ein
> 10+0 Datensätze aus
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s
>
> dd if=/daten/testfile of=/dev/null
> 20971520+0 Datensätze ein
> 20971520+0 Datensätze aus
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s
rsync(1) can be fickle, especially when host operating systems and/or
rsync versions mismatch. But, you also seem to have file system issues.
The above information could be more useful if it were presented in
context -- e.g. complete console sessions -- meaningful prompts (date,
time, username, hostname, working directory), exact commands input, and
exact output printed.
Please use the Bash shell for the root account and the following line to
/root/.bashrc:
export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '
Then start a new shell, su(1) to root, change to a directory within the
file system that is having issues, and run the following commands.
Adjust the commands for your hosts and network, as required ('po' is my
workstation, 'dpchrist' is my user account, 'samba' is my file server,
and '/var/local/samba/dpchrist' is my storage directory on my file server):
2019-05-09 21:59:55 root@po /mnt/scratch
# cat /etc/debian_version
9.9
2019-05-09 22:00:00 root@po /mnt/scratch
# uname -a
Linux po 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux
2019-05-09 22:00:05 root@po /mnt/scratch
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 55.9G 0 disk
|-sda1 8:1 0 953M 0 part /boot
|-sda2 8:2 0 1.9G 0 part
| `-sda2_crypt 253:1 0 1.9G 0 crypt [SWAP]
|-sda3 8:3 0 9.3G 0 part
| `-sda3_crypt 253:0 0 9.3G 0 crypt /
`-sda4 8:4 0 38.2G 0 part
`-sda4_crypt 253:2 0 38.2G 0 crypt /mnt/scratch
sr0 11:0 1 1024M 0 rom
2019-05-09 22:00:10 root@po /mnt/scratch
# mount | egrep 'sd[a-z]'
/dev/mapper/sda3_crypt on / type btrfs
(rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/sda1 on /boot type btrfs
(rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/mapper/sda4_crypt on /mnt/scratch type btrfs
(rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
2019-05-09 22:00:19 root@po /mnt/scratch
# df | egrep 'sd[a-z]'
/dev/mapper/sda3_crypt 9763840 5540800 4160128 58% /
/dev/sda1 975872 80892 800244 10% /boot
/dev/mapper/sda4_crypt 40056832 1066752 38732224 3% /mnt/scratch
2019-05-09 22:00:27 root@po /mnt/scratch
# time dd if=/dev/urandom of=foo bs=1M count=1K conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.987 s, 67.2 MB/s
real 0m15.988s
user 0m0.000s
sys 0m3.976s
2019-05-09 22:02:13 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && time dd if=foo of=/dev/null bs=1M
count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.49821 s, 430 MB/s
real 0m2.518s
user 0m0.000s
sys 0m0.252s
2019-05-09 22:27:32 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar &&
uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && ssh $uh rm -f
$p/$g && rsync -a --stats $f $uh:$p/$g
Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 1,073,741,824 bytes
Total transferred file size: 1,073,741,824 bytes
Literal data: 1,073,741,824 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1,074,004,051
Total bytes received: 35
sent 1,074,004,051 bytes received 35 bytes 113,053,061.68 bytes/sec
total size is 1,073,741,824 speedup is 1.00
2019-05-09 22:27:52 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar &&
uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && rm -f $g &&
rsync -a --stats $uh:$p/$g $g
Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 1,073,741,824 bytes
Total transferred file size: 1,073,741,824 bytes
Literal data: 1,073,741,824 bytes
Matched data: 0 bytes
File list size: 43
File list generation time: 0.022 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 43
Total bytes received: 1,074,004,061
sent 43 bytes received 1,074,004,061 bytes 102,286,105.14 bytes/sec
total size is 1,073,741,824 speedup is 1.00
Add in other commands as you see fit, such as whatever generated the
output in your previous posts.
Please reply with your *complete* console session (you can remove all of
the above in your reply).
David
