-------- Original Message --------
Subject: btrfs stuck with lot's of files
From: Peter Volkov <p...@gentoo.org>
To: linux-btrfs@vger.kernel.org <linux-btrfs@vger.kernel.org>
Date: 2014年12月01日 19:46
Hi, guys.
We have a problem with btrfs file system: sometimes it became stuck
without leaving me any way to interrupt it (shutdown -r now is unable to
restart server). By stuck I mean some processes that previously were
able to write on disk are unable to cope with load and load average goes
up:
top - 13:10:58 up 1 day, 9:26, 5 users, load average: 157.76, 156.61,
149.29
Tasks: 235 total, 2 running, 233 sleeping, 0 stopped, 0 zombie
%Cpu(s): 19.8 us, 15.0 sy, 0.0 ni, 60.7 id, 3.9 wa, 0.0 hi, 0.6 si,
0.0 st
KiB Mem: 65922104 total, 65414856 used, 507248 free, 1844 buffers
KiB Swap: 0 total, 0 used, 0 free. 62570804 cached
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
8644 root 20 0 0 0 0 R 96.5 0.0 127:21.95
kworker/u16:16
5047 dvr 20 0 6884292 122668 4132 S 6.4 0.2 258:59.49
dvrserver
30223 root 20 0 20140 2600 2132 R 6.4 0.0 0:00.01
top
1 root 20 0 4276 1628 1524 S 0.0 0.0 0:40.19
init
There are about 300 treads on server, some of which are writing on disk.
A bit information about this btrfs filesystem: this is 22 disk file
system with raid1 for metadata and raid0 for data:
# btrfs filesystem df /store/
Data, single: total=11.92TiB, used=10.86TiB
System, RAID1: total=8.00MiB, used=1.27MiB
System, single: total=4.00MiB, used=0.00B
Metadata, RAID1: total=46.00GiB, used=33.49GiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=128.00KiB
# btrfs property get /store/
ro=false
label=store
# btrfs device stats /store/
(shows all zeros)
# btrfs balance status /store/
No balance found on '/store/'
# btrfs filesystem show /store/
Btrfs v3.17.1
(btw, is it supposed to have only version here?)
This is a small bug that if there is appending '/' in the path for
'btrfs fi show', it can't recognize it....
Patch is already sent and maybe included next version.
As for load we write quite small files of size (some of 313K, some of
800K), that's why metadata takes that much. So back to the problem.
iostat 1 exposes following problem:
avg-cpu: %user %nice %system %iowait %steal %idle
16.96 0.00 17.09 65.95 0.00 0.00
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.00 0.00 0.00 0 0
sdc 0.00 0.00 0.00 0 0
sdb 0.00 0.00 0.00 0 0
sde 0.00 0.00 0.00 0 0
sdd 0.00 0.00 0.00 0 0
sdf 0.00 0.00 0.00 0 0
sdg 0.00 0.00 0.00 0 0
sdj 0.00 0.00 0.00 0 0
sdh 0.00 0.00 0.00 0 0
sdk 0.00 0.00 0.00 0 0
sdi 1.00 0.00 200.00 0 200
sdl 0.00 0.00 0.00 0 0
sdn 48.00 0.00 17260.00 0 17260
sdm 0.00 0.00 0.00 0 0
sdp 0.00 0.00 0.00 0 0
sdo 0.00 0.00 0.00 0 0
sdq 0.00 0.00 0.00 0 0
sdr 0.00 0.00 0.00 0 0
sds 0.00 0.00 0.00 0 0
sdt 0.00 0.00 0.00 0 0
sdv 0.00 0.00 0.00 0 0
sdw 0.00 0.00 0.00 0 0
sdu 0.00 0.00 0.00 0 0
write goes to one disk. I've tried to debug what's going in kworker and
did
$ echo workqueue:workqueue_queue_work
/sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > trace_pipe.out2
trace_pipe2.out.xz in attachment. Could you comment, what goes wrong
here?
It seems that attachment is blocked by mail-list so I didn't see the
attachment.
Server has 64Gb of RAM. Is it possible that it is unable to keep all
metadata in memory, can we encrease this memory limit, if exists?
Not possible, it will never happen (if nothing goes wrong....).
Kernel has the outstanding page cache mechanism, when memory comes short,
some cached metadata/data can be flushed back(if dirty) to disk to free
space.
And re-read from disk if needed later.
So kernel don't need to load all the metadata/data into memory, and
that's mostly impossible for large fs.
And one missing important informantion: kernel version.
What I can see is only the btrfs-progs version, which doesn't really
help for such kernel stuck problem.
Thanks,
Qu
Thanks in advance for any pointers,
--
Peter.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
