lots of data written, but no space occupied

André Weidemann Thu, 22 Feb 2018 10:53:44 -0800

Hi,

I've been using btrfs for some time now on my server and am pretty satisfied with it's performance and features. I'm running Ubuntu 16.04 64bit with kernel 4.4.0-112.


The other day I installed collectd, InfulxDB and Grafana on my server.

I was surprised to see on the graphs, that there was data constantly written to my btrfs RAID1 drives (about 2MB/s).

Since the RAID1 only contains /, /home and a docker container. There is no application actually writing to the disks, only logs go to /var/log. All other data goes to ZFS-storage.


So I dug out iotop in order to figure out what is writing to the disks.
The only process I could find is called "[btrfs-transacti]".
I let "iotop -o -a" run for about 3 hours and got this as a result:

Total DISK READ :      60.13 M/s | Total DISK WRITE :      17.96 M/s
Actual DISK READ:      37.67 M/s | Actual DISK WRITE:      10.32 K/s
    TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND

419 be/4 root 4.16 M 9.59 G 0.00 % 2.26 % [btrfs-transacti]


9.59GB were apparently written to my disk.
The interesting thing is, that the output of  "btrfs filesystem usage /"
has not changed within those 3 hours. It still looks exactly like this:

Overall:
      Device size:                   1.76TiB
      Device allocated:            408.12GiB
      Device unallocated:            1.36TiB
      Device missing:                  0.00B
      Used:                        394.78GiB
      Free (estimated):            698.69GiB      (min: 698.69GiB)
      Data ratio:                       2.00
      Metadata ratio:                   2.00
      Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID1: Size:197.00GiB, Used:194.25GiB
     /dev/sda2     197.00GiB
     /dev/sdh2     197.00GiB

Metadata,RAID1: Size:7.00GiB, Used:3.14GiB
     /dev/sda2       7.00GiB
     /dev/sdh2       7.00GiB

System,RAID1: Size:64.00MiB, Used:48.00KiB
     /dev/sda2      64.00MiB
     /dev/sdh2      64.00MiB

Unallocated:
     /dev/sda2     695.94GiB
     /dev/sdh2     695.94GiB

Nothing has changed for the output of "btrfs filesystem df /" either:

Data, RAID1: total=197.00GiB, used=194.25GiB
System, RAID1: total=64.00MiB, used=48.00KiB
Metadata, RAID1: total=7.00GiB, used=3.14GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


These are my subvolumes: "btrfs subvolume list /":

ID 257 gen 626146 top level 5 path @
ID 258 gen 626146 top level 5 path @home

ID 660 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/05c868ba36d465647ee407f1b99a7115c249af680759e552fc1e9f0835e1c1d3 ID 661 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/7ff20b948f811be05dfae0ab331d1e7f28d3d9472536060f2bb4b1d2895c9ce7 ID 662 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/41bda929a8e4a8a0022e3caad3bd0108ac1d0db2d0a7484867872b86b4a89ad0 ID 663 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/7e315f9dbace5c5585e84ab39b22ad1aabd90f8da3637b09ca0526a5f4f0c904 ID 664 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/ab01bee8ce430c90bae89bfbf7252911478888fc5853eee1071dc841808a4092 ID 665 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/b853e32ba79c3fdd6fcf15769f4e2c8f546543aeeedbe7b9a39011581a840dc5 ID 666 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/42cb357d696677255047ded26d604a53fac414e738d18e97b55444abc877f155 ID 667 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/177558b388af2c6088286fb65cfab31422228f4f83ec2f77214b208afc4dfc3e ID 668 gen 611310 top level 257 path var/lib/docker/btrfs/subvolumes/d89b36b29c6477c91ce73b9288e98684da8cdec80455d498fe7627281461b190-init ID 669 gen 612469 top level 257 path var/lib/docker/btrfs/subvolumes/d89b36b29c6477c91ce73b9288e98684da8cdec80455d498fe7627281461b190

ID 691 gen 611319 top level 257 path mnt/nextcloud-test

This is the output for all mounted BTRFS' on my machine: "grep btrf /proc/mounts"

/dev/sdh2 / btrfs rw,relatime,space_cache,subvolid=257,subvol=/@ 0 0
/dev/sdh2 /home btrfs rw,relatime,space_cache,subvolid=258,subvol=/@home 0 0

/dev/sdh2 /var/lib/docker/btrfs btrfs rw,relatime,space_cache,subvolid=257,subvol=/@/var/lib/docker/btrfs 0 0


To verify the Grafana output, I started this:

while (true); do awk '/sdh\ |sda\ / {print "disk: "$3"\tread:"$6 / 2 / 1024"\twritten:"$10 / 2 / 1024}' /proc/diskstats; date; sleep 3600; done and let it run for a few hours. The results pretty much matched the number in the Grafana graphs.


Those are my calculations from above run:
bytes r diff    MB/s            bytes w diff    MB/s
490758                          746976          
490793  35      0,009722222     753345  6369    1,769166667
490827  34      0,009444444     759868  6523    1,811944444
490867  40      0,011111111     766357  6489    1,8025
491551  684     0,19            775604  9247    2,568611111
491581  30      0,008333333     782609  7005    1,945833333
491638  57      0,015833333     789722  7113    1,975833333
491723  85      0,023611111     796819  7097    1,971388889
491804  81      0,0225          804167  7348    2,041111111

I also stopped docker after 4 hours of running above line because I thought, that the disk writes might be related to it. But the behavior did not change.

So my question is, what is writing to the disks all the time, but also not writing, since there is no additional space occupied? I could understand if btrfs had to relocate data and had to write it to disk, but this seems unlikely since there is almost nothing read from the disks...



Thank you a lot in advance.
  André

smime.p7s
Description: S/MIME Cryptographic Signature

lots of data written, but no space occupied

Reply via email to