I have a RAID6 array that had a failed HDD. The drive failed completely and has been removed from the system. I'm running a 'device replace' operation with a new disk. The array is ~20TB so this will take a few days.
Yesterday the system crashed hard with OOM errors about 24 hours into the replace. Rebooting after the crash and remounting the array automatically resumed the replace where it left off. Today I kept a close eye on it and have watched the memory usage creep up slowly. htop says this is user process memory (green bar) but shows no user processes using this much memory free says this is almost entirely cached/buffered memory that is taking up the space. slabtop reveals that there is a highly unusual amount of SLAB going to 'bio' which has to do with block allocation apparently. slabtop output is attached. 'sync && echo 3 > /proc/sys/vm/drop_caches' clears the high usage (~4GB) from dentry but 'bio' does not release any (11GB) memory and continues to grow slowly. This is running the Rockstor distro based on CentOS. The system has 16GB of RAM. Kernel: 4.4.5-1.el7.elrepo.x86_64 btrfs-progs: 4.4.1 Kernel messages aren't showing anything of note during the replace until it starts throwing out OOM errors. I would like to collect enough information for a useful bug report here, but I also can't babysit this rebuild during the work week and reboot it once a day for OOM crashes. Should I cancel the replace operation and use 'dev delete missing' instead? Will using 'delete missing' cause any problem if it's done after a partially completed and canceled replace?
# slabtop -o -s=a Active / Total Objects (% used) : 33431432 / 33664160 (99.3%) Active / Total Slabs (% used) : 1346736 / 1346736 (100.0%) Active / Total Caches (% used) : 78 / 114 (68.4%) Active / Total Size (% used) : 10512136.19K / 10737701.80K (97.9%) Minimum / Average / Maximum Object : 0.01K / 0.32K / 15.62K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 32493650 32492775 99% 0.31K 1299746 25 10397968K bio-1 323505 323447 99% 0.19K 15405 21 61620K dentry 176680 176680 100% 0.07K 3155 56 12620K btrfs_free_space 118208 41288 34% 0.12K 3694 32 14776K kmalloc-128 94528 43378 45% 0.25K 2954 32 23632K kmalloc-256 91872 41682 45% 0.50K 2871 32 45936K kmalloc-512 83048 39031 46% 4.00K 10381 8 332192K kmalloc-4096 69049 69049 100% 0.27K 2381 29 19048K btrfs_extent_buffer 46872 46385 98% 0.57K 1674 28 26784K radix_tree_node 23460 23460 100% 0.12K 690 34 2760K kernfs_node_cache 17536 17536 100% 0.98K 548 32 17536K btrfs_inode 16380 16007 97% 0.14K 585 28 2340K btrfs_path 12444 11635 93% 0.08K 244 51 976K Acpi-State 12404 12404 100% 0.55K 443 28 7088K inode_cache 11648 10851 93% 0.06K 182 64 728K kmalloc-64 10404 5716 54% 0.08K 204 51 816K btrfs_extent_state 8954 8703 97% 0.18K 407 22 1628K vm_area_struct 5888 4946 84% 0.03K 46 128 184K kmalloc-32 5632 5632 100% 0.01K 11 512 44K kmalloc-8 5049 4905 97% 0.08K 99 51 396K anon_vma 4352 4352 100% 0.02K 17 256 68K kmalloc-16 3723 3723 100% 0.05K 51 73 204K Acpi-Parse 3230 3230 100% 0.05K 38 85 152K ftrace_event_field 3213 2949 91% 0.19K 153 21 612K kmalloc-192 3120 3090 99% 0.61K 120 26 1920K proc_inode_cache 2814 2814 100% 0.09K 67 42 268K kmalloc-96 1984 1510 76% 1.00K 62 32 1984K kmalloc-1024 1904 1904 100% 0.07K 34 56 136K Acpi-Operand 1472 1472 100% 0.09K 32 46 128K trace_event_file 1224 1224 100% 0.04K 12 102 48K Acpi-Namespace 1152 1152 100% 0.64K 48 24 768K shmem_inode_cache 592 581 98% 2.00K 37 16 1184K kmalloc-2048 528 457 86% 0.36K 24 22 192K blkdev_requests 462 355 76% 0.38K 22 21 176K mnt_cache 450 433 96% 1.06K 15 30 480K signal_cache 429 429 100% 0.20K 11 39 88K btrfs_delayed_ref_head 420 420 100% 2.05K 28 15 896K idr_layer_cache 408 408 100% 0.04K 4 102 16K btrfs_delayed_extent_op 400 400 100% 0.62K 16 25 256K sock_inode_cache 364 364 100% 0.30K 14 26 112K btrfs_delayed_node 351 351 100% 0.10K 9 39 36K buffer_head 345 312 90% 2.06K 23 15 736K sighand_cache 318 298 93% 5.25K 53 6 1696K task_struct 256 256 100% 0.06K 4 64 16K kmem_cache_node 256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
