Delivery Failure

[admin]

-
The message you sent to vhakonigroup.co.za/mashudu was rejected because it 
would exceed the quota for the mailbox.

The subject of the message follows: 
Re:Start to earn 15k daily.So easy money

-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Best strategie to remove devices from pool

[Cloud Admin]

Hi,
I want to remove two devices from a BTRFS RAID 1 pool. It should be
enough free space to do it, but what is the best strategie. Remove both
device in one call 'btrfs dev rem /dev/sda1 /dev/sdb1' (for example) or
should it be better in two separate calls? What is faster? Are there
other constraints to think about?
Bye
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

WARNING: ... at fs/btrfs/ctree.h:1559 btrfs_update_device+0x1be/0x1d0 [btrfs]

[Cloud Admin]

Hi,
I update kernel from 4.11.10 to 4.13.4 and since that I get the following 
message in the kernel journal calling 'scrub' or 'balance'. I use Fedora 26 
with btrfs-progs v4.9.1.
What does this mean and (more important) what can I do? 
Bye
Frank

BTRFS info (device dm-7): relocating block group 44050690342912 flags 
system|raid1
BTRFS info (device dm-7): found 117 extents
[ cut here ]
WARNING: CPU: 3 PID: 22095 at fs/btrfs/ctree.h:1559 
btrfs_update_device+0x1be/0x1d0 [btrfs]
Modules linked in: rpcsec_gss_krb5 veth xt_nat xt_addrtype br_netfilter 
xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_sane xt_CT 
ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink 
ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 
nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security 
iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack 
libcrc32c iptable_mangle iptable_raw iptable_security ebtable_filter ebtables 
ip6table_filter ip6_tables ftsteutates btrfs xor raid6_pq tda18212 cxd2841er 
tpm_crb intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm 
iTCO_wdt irqbypass iTCO_vendor_support intel_cstate intel_uncore mei_wdt 
intel_rapl_perf ppdev hci_uart ddbridge dvb_core
Okt 09 19:08:32 hypercloud.fritz.box kernel:  btbcm btqca btintel mei_me 
i2c_i801 shpchp bluetooth joydev mei intel_pch_thermal wmi intel_lpss_acpi 
intel_lpss pinctrl_sunrisepoint fujitsu_laptop parport_pc sparse_keymap 
ecdh_generic parport tpm_tis pinctrl_intel tpm_tis_core rfkill tpm acpi_pad 
nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_crypt i915 crct10dif_pclmul 
i2c_algo_bit crc32_pclmul drm_kms_helper crc32c_intel e1000e drm 
ghash_clmulni_intel serio_raw ptp pps_core video i2c_hid
CPU: 3 PID: 22095 Comm: btrfs Tainted: GW   4.13.4-200.fc26.x86_64 
#1
Hardware name: FUJITSU D3417-B1/D3417-B1, BIOS V5.0.0.11 R1.12.0 for D3417-B1x  
  02/09/2016
task: 8ecb59b026c0 task.stack: b805cae9
RIP: 0010:btrfs_update_device+0x1be/0x1d0 [btrfs]
RSP: 0018:b805cae93ac8 EFLAGS: 00010206
RAX: 0fff RBX: 8ed094bb11c0 RCX: 074702251e00
RDX: 0004 RSI: 3efa RDI: 8ec9eb032c08
RBP: b805cae93b10 R08: 3efe R09: b805cae93a80
R10: 1000 R11: 0003 R12: 8ed0c7a3f000
R13:  R14: 3eda R15: 8ec9eb032c08
FS:  7f10b256a8c0() GS:8ed0ee4c() knlGS:
CS:  0010 DS:  ES:  CR0: 80050033
CR2: 7f6261c0d5f0 CR3: 0005cad4c000 CR4: 003406e0
DR0:  DR1:  DR2: 
DR3:  DR6: fffe0ff0 DR7: 0400
Call Trace:
 btrfs_remove_chunk+0x365/0x870 [btrfs]
 btrfs_relocate_chunk+0x7e/0xd0 [btrfs]
 btrfs_balance+0xc07/0x1390 [btrfs]
 btrfs_ioctl_balance+0x319/0x380 [btrfs]
 btrfs_ioctl+0x9d5/0x24a0 [btrfs]
 ? lru_cache_add+0x3a/0x80
 ? lru_cache_add_active_or_unevictable+0x4c/0xf0
 ? __handle_mm_fault+0x939/0x10b0
 do_vfs_ioctl+0xa5/0x600
 ? do_brk_flags+0x230/0x360
 ? do_vfs_ioctl+0xa5/0x600
 SyS_ioctl+0x79/0x90
 entry_SYSCALL_64_fastpath+0x1a/0xa5
RIP: 0033:0x7f10b15e65e7
RSP: 002b:7ffc9402ebe8 EFLAGS: 0246 ORIG_RAX: 0010
RAX: ffda RBX: 8041 RCX: 7f10b15e65e7
RDX: 7ffc9402ec80 RSI: c4009420 RDI: 0003
RBP: 7f10b18abae0 R08: 55b07a3b3010 R09: 0078
R10: 7f10b18abb38 R11: 0246 R12: 
R13: 7f10b18abb38 R14: 8060 R15: 
Code: 4c 89 ff 45 31 c0 ba 10 00 00 00 4c 89 f6 e8 fa 20 ff ff 4c 89 ff e8 72 
ef fc ff e9 d3 fe ff ff 41 bd f4 ff ff ff e9 d0 fe ff ff <0f> ff eb b6 e8 39 fd 
78 c6 66 0f 1f 84 00 00 00 00 00 0f 1f 44 
---[ end trace d1e1c8aff99bfeb8 ]---
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

How to disable/revoke 'compression'?

[Cloud Admin]

Hi,
I used the mount option 'compression' on some mounted sub volumes. How
can I revoke the compression? Means to delete the option and get all
data uncompressed on this volume.
Is it enough to remount the sub volume without this option? Or is it
necessary to do some addional step (balancing?) to get all stored data
uncompressed. Beside of it, is it possible to find out what the real
and compressed size of a file, for example or the ratio?
Bye
   Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Replace missing disc => strange result!?

[Cloud Admin]

Hi,
I had a disc failure and must replace it. I followed the description on
 https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devi
ces and started the replacement.
Setup is a two disc RAID1!
After it was done, I called 'btrfs fi us /mn/btrfsroot' and I got the
output below. What is wrong?
Is it a rebalancing issue? I thought the replace command started it
automatically...

Overall:
Device size:   3.64TiB
Device allocated:      1.04TiB
Device unallocated:    2.60TiB
Device missing:  0.00B
Used:    519.76GiB
Free (estimated):      1.56TiB  (min: 1.56TiB)
Data ratio:   2.00
Metadata ratio:   1.60
Global reserve:  279.11MiB  (used: 0.00B)

Data,single: Size:1.00GiB, Used:0.00B
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8   1.00
GiB

Data,RAID1: Size:265.00GiB, Used:259.60GiB
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8 265.00
GiB
   /dev/mapper/luks-ff4bf5da-48af-4563-abb2-db083bd01512 265.00
GiB

Data,DUP: Size:264.00GiB, Used:0.00B
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8 528.00
GiB

Metadata,single: Size:1.00GiB, Used:0.00B
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8   1.00
GiB

Metadata,RAID1: Size:1.00GiB, Used:286.03MiB
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8   1.00
GiB
   /dev/mapper/luks-ff4bf5da-48af-4563-abb2-db083bd01512   1.00
GiB

Metadata,DUP: Size:512.00MiB, Used:112.00KiB
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8   1.00
GiB

System,single: Size:32.00MiB, Used:0.00B
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8  32.00
MiB

System,RAID1: Size:8.00MiB, Used:48.00KiB
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8   8.00
MiB
   /dev/mapper/luks-ff4bf5da-48af-4563-abb2-db083bd01512   8.00
MiB

System,DUP: Size:32.00MiB, Used:48.00KiB
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8  64.00
MiB

Unallocated:
   /dev/mapper/luks-3ff6c412-4d3a-4d33-85a3-cc70e95c26f8   1.04
TiB
   /dev/mapper/luks-ff4bf5da-48af-4563-abb2-db083bd01512   1.56
TiB

Bye
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Save to use 'clear_cache' in mount -o remount?

[Cloud Admin]

Hi,
is it safe (has it an effect?) to use the 'clear_cache' option in a
'mount -o remount'? I recognize messages in my kernel log regarding
'BTRFS info (device dm-7): The free space cache file (31215079915520)
is invalid. skip it'. I would like to fix it and would do it (in best
case) without rebooting.
Bye
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best Practice: Add new device to RAID1 pool (Summary)

[Cloud Admin]

Am Montag, den 24.07.2017, 18:40 +0200 schrieb Cloud Admin:
> Am Montag, den 24.07.2017, 10:25 -0400 schrieb Austin S. Hemmelgarn:
> > On 2017-07-24 10:12, Cloud Admin wrote:
> > > Am Montag, den 24.07.2017, 09:46 -0400 schrieb Austin S.
> > > Hemmelgarn:
> > > > On 2017-07-24 07:27, Cloud Admin wrote:
> > > > > Hi,
> > > > > I have a multi-device pool (three discs) as RAID1. Now I want
> > > > > to
> > > > > add a
> > > > > new disc to increase the pool. I followed the description on
> > > > > https:
> > > > > //bt
> > > > > rfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devic
> > > > > es
> > > > > and
> > > > > used 'btrfs add  '. After that I called a
> > > > > balance
> > > > > for rebalancing the RAID1 using 'btrfs balance start  > > > > path>'.
> > > > > Is that anything or should I need to call a resize (for
> > > > > example) or
> > > > > anything else? Or do I need to specify filter/profile
> > > > > parameters
> > > > > for
> > > > > balancing?
> > > > > I am a little bit confused because the balance command is
> > > > > running
> > > > > since
> > > > > 12 hours and only 3GB of data are touched. This would mean
> > > > > the
> > > > > whole
> > > > > balance process (new disc has 8TB) would run a long, long
> > > > > time...
> > > > > and
> > > > > is using one cpu by 100%.
> > > > 
> > > > Based on what you're saying, it sounds like you've either run
> > > > into a
> > > > bug, or have a huge number of snapshots on this filesystem.
> > > 
> > > It depends what you define as huge. The call of 'btrfs sub list
> > >  > > path>' returns a list of 255 subvolume.
> > 
> > OK, this isn't horrible, especially if most of them aren't
> > snapshots 
> > (it's cross-subvolume reflinks that are most of the issue when it
> > comes 
> > to snapshots, not the fact that they're subvolumes).
> > > I think this is not too huge. The most of this subvolumes was
> > > created
> > > using docker itself. I cancel the balance (this will take awhile)
> > > and will try to delete such of these subvolumes/snapshots.
> > > What can I do more?
> > 
> > As Roman mentioned in his reply, it may also be qgroup related.  If
> > you run:
> > btrfs quota disable
> 
> It seems quota was one part of it. Thanks for the tip. I disabled and
> started balance new.
> Now approx. each 5 min. one chunk will be relocated. But if I take
> the
> reported 10860 chunks and calc. the time it will take ~37 days to
> finish... So, it seems I have to investigate more time into figure
> out
> the subvolume / snapshots structure created by docker.
> A first deeper look shows, there is a subvolume with a snapshot,
> which
> has itself a snapshot, and so forth.
> > 
> > 
Now, the balance process finished after 127h the new disc is in the
pool... Not so long as expected but in my opinion long enough. Quota
seems one big driver in my case. What I could see over the time at the
beginning many extends was relocated ignoring the new disc. Properly it
could be a good idea to rebalance using filter (like -dusage=30 for
example) before add the new disc to decrease the time. 
But only theory. It will try to keep it in my mind for the next time.

Thanks all for your tips, ideas and time!
Frank

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best Practice: Add new device to RAID1 pool

[Cloud Admin]

Am Montag, den 24.07.2017, 20:42 + schrieb Hugo Mills:
> On Mon, Jul 24, 2017 at 02:35:05PM -0600, Chris Murphy wrote:
> > On Mon, Jul 24, 2017 at 5:27 AM, Cloud Admin <admin@cloud.haefemeie
> > r.eu> wrote:
> > 
> > > I am a little bit confused because the balance command is running
> > > since
> > > 12 hours and only 3GB of data are touched.
> > 
> > That's incredibly slow. Something isn't right.
> > 
> > Using btrfs-debug -b from btrfs-progs, I've selected a few 100%
> > full chunks.
> > 
> > [156777.077378] f26s.localdomain sudo[13757]:chris : TTY=pts/2
> > ;
> > PWD=/home/chris ; USER=root ; COMMAND=/sbin/btrfs balance start
> > -dvrange=157970071552..159043813376 /
> > [156773.328606] f26s.localdomain kernel: BTRFS info (device sda1):
> > relocating block group 157970071552 flags data
> > [156800.408918] f26s.localdomain kernel: BTRFS info (device sda1):
> > found 38952 extents
> > [156861.343067] f26s.localdomain kernel: BTRFS info (device sda1):
> > found 38951 extents
> > 
> > That 1GiB chunk with quite a few fragments took 88s. That's 11MB/s.
> > Even for a hard drive, that's slow. I've got maybe a dozen
> > snapshots
> > on this particular volume and quotas are not enabled. By definition
> > all of those extents are sequential. So I'm not sure why it's
> > taking
> > so long. Seems almost like a regression somewhere. A nearby chunk
> > with
> > ~23k extents only takes 45s to balance. And another chunk with
> > ~32000
> > extents took 55s to balance.
> 
>    In my experience, it's pretty consistent at about a minute per 1
> GiB for data on rotational drives on RAID-1. For metadata, it can go
> up to several hours (or more) per 256 MiB chunk, depending on what
> kind of metadata it is. With extents shared between lots of files, it
> slows down. In my case, with a few hundred snapshots of the same
> thing, my system was taking 4h per chunk for the chunks full of the
> extent tree.
After disabling quota the balancing is no working faster. After 27h
approx. 1.3TB are done. It has taken around 4h of rearrange the data on
the old three discs the process started to use the new one. Since there
it is processing much faster.

Bye
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best Practice: Add new device to RAID1 pool

[Cloud Admin]

Am Montag, den 24.07.2017, 23:12 +0200 schrieb waxhead:
> 
> Chris Murphy wrote:
> 
> This may be a stupid question , but are your pool of butter (or
> BTRFS 
> pool) by any chance hooked up via USB? If this is USB2.0 at 
No, it is a SATA array with (currently) four 8TB discs.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best Practice: Add new device to RAID1 pool

[Cloud Admin]

Am Montag, den 24.07.2017, 10:25 -0400 schrieb Austin S. Hemmelgarn:
> On 2017-07-24 10:12, Cloud Admin wrote:
> > Am Montag, den 24.07.2017, 09:46 -0400 schrieb Austin S.
> > Hemmelgarn:
> > > On 2017-07-24 07:27, Cloud Admin wrote:
> > > > Hi,
> > > > I have a multi-device pool (three discs) as RAID1. Now I want
> > > > to
> > > > add a
> > > > new disc to increase the pool. I followed the description on
> > > > https:
> > > > //bt
> > > > rfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices
> > > > and
> > > > used 'btrfs add  '. After that I called a
> > > > balance
> > > > for rebalancing the RAID1 using 'btrfs balance start  > > > path>'.
> > > > Is that anything or should I need to call a resize (for
> > > > example) or
> > > > anything else? Or do I need to specify filter/profile
> > > > parameters
> > > > for
> > > > balancing?
> > > > I am a little bit confused because the balance command is
> > > > running
> > > > since
> > > > 12 hours and only 3GB of data are touched. This would mean the
> > > > whole
> > > > balance process (new disc has 8TB) would run a long, long
> > > > time...
> > > > and
> > > > is using one cpu by 100%.
> > > 
> > > Based on what you're saying, it sounds like you've either run
> > > into a
> > > bug, or have a huge number of snapshots on this filesystem.
> > 
> > It depends what you define as huge. The call of 'btrfs sub list
> >  > path>' returns a list of 255 subvolume.
> 
> OK, this isn't horrible, especially if most of them aren't snapshots 
> (it's cross-subvolume reflinks that are most of the issue when it
> comes 
> to snapshots, not the fact that they're subvolumes).
> > I think this is not too huge. The most of this subvolumes was
> > created
> > using docker itself. I cancel the balance (this will take awhile)
> > and will try to delete such of these subvolumes/snapshots.
> > What can I do more?
> 
> As Roman mentioned in his reply, it may also be qgroup related.  If
> you run:
> btrfs quota disable
It seems quota was one part of it. Thanks for the tip. I disabled and
started balance new.
Now approx. each 5 min. one chunk will be relocated. But if I take the
reported 10860 chunks and calc. the time it will take ~37 days to
finish... So, it seems I have to investigate more time into figure out
the subvolume / snapshots structure created by docker.
A first deeper look shows, there is a subvolume with a snapshot, which
has itself a snapshot, and so forth.
> 
> On the filesystem in question, that may help too, and if you are
> using 
> quotas, turning them off with that command will get you a much
> bigger 
> performance improvement than removing all the snapshots.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-
> btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best Practice: Add new device to RAID1 pool

[Cloud Admin]

Am Montag, den 24.07.2017, 19:08 +0500 schrieb Roman Mamedov:
> On Mon, 24 Jul 2017 09:46:34 -0400
> "Austin S. Hemmelgarn"  wrote:
> 
> > > I am a little bit confused because the balance command is running
> > > since
> > > 12 hours and only 3GB of data are touched. This would mean the
> > > whole
> > > balance process (new disc has 8TB) would run a long, long time...
> > > and
> > > is using one cpu by 100%.
> > 
> > Based on what you're saying, it sounds like you've either run into
> > a 
> > bug, or have a huge number of snapshots
> 
> ...and possibly quotas (qgroups) enabled. (perhaps automatically by
> some tool,
> and not by you). Try:
> 
>   btrfs quota disable 
> 
It seems this was one part of my problem. See my answer to Austin.
> 
With respect,
> Roman
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best Practice: Add new device to RAID1 pool

[Cloud Admin]

Am Montag, den 24.07.2017, 09:46 -0400 schrieb Austin S. Hemmelgarn:
> On 2017-07-24 07:27, Cloud Admin wrote:
> > Hi,
> > I have a multi-device pool (three discs) as RAID1. Now I want to
> > add a
> > new disc to increase the pool. I followed the description on https:
> > //bt
> > rfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices and
> > used 'btrfs add  '. After that I called a
> > balance
> > for rebalancing the RAID1 using 'btrfs balance start '.
> > Is that anything or should I need to call a resize (for example) or
> > anything else? Or do I need to specify filter/profile parameters
> > for
> > balancing?
> > I am a little bit confused because the balance command is running
> > since
> > 12 hours and only 3GB of data are touched. This would mean the
> > whole
> > balance process (new disc has 8TB) would run a long, long time...
> > and
> > is using one cpu by 100%.
> 
> Based on what you're saying, it sounds like you've either run into a 
> bug, or have a huge number of snapshots on this filesystem.  

It depends what you define as huge. The call of 'btrfs sub list ' returns a list of 255 subvolume.
I think this is not too huge. The most of this subvolumes was created
using docker itself. I cancel the balance (this will take awhile) and will try 
to delete such of these subvolumes/snapshots.
What can I do more?

> What you 
> described is exactly what you should be doing when expanding an
> array 
> (add the device, then run a full balance).  The fact that it's
> taking 
> this long isn't normal, unless you have very slow storage devices.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Best Practice: Add new device to RAID1 pool

[Cloud Admin]

Hi,
I have a multi-device pool (three discs) as RAID1. Now I want to add a
new disc to increase the pool. I followed the description on https://bt
rfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices and
used 'btrfs add  '. After that I called a balance
for rebalancing the RAID1 using 'btrfs balance start '. 
Is that anything or should I need to call a resize (for example) or
anything else? Or do I need to specify filter/profile parameters for
balancing?
I am a little bit confused because the balance command is running since
12 hours and only 3GB of data are touched. This would mean the whole
balance process (new disc has 8TB) would run a long, long time... and
is using one cpu by 100%.
Thanks for your help and time.
Bye
Frank

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

[admin]

Hi,

The source is a software raid 5 (md) of 4x 4TB Western Digital RE4 disks. The 
destinations is a hardware raid 5 enclosure containing 4x 8TB Seagate Archival 
disks connected using e-sata. 

I am currently trying Duncans suggestions. With them, the page allocation stall 
doesn't seem to appear and overall system responsiveness is also much better 
during copying.

Thanks,
David Arendt


Xin Zhou – Thu., 15. December 2016 0:24
> Hi,
> 
> The dirty data is in large amount, probably unable to commit to disk.
> And this seems to happen when copying from 7200rpm to 5600rpm disks, 
> according to previous post.
> 
> Probably the I/Os are buffered and pending, unable to get finished in-time.
> It might be helpful to know if this only happens for specific types of 5600 
> rpm disks?
> 
> And are these disks on RAID groups? Thanks.
> Xin
>  
>  
> 
> Sent: Wednesday, December 14, 2016 at 3:38 AM
> From: admin <ad...@prnet.org>
> To: "Michal Hocko" <mho...@kernel.org>
> Cc: linux-btrfs@vger.kernel.org, linux-ker...@vger.kernel.org, "David Sterba" 
> <dste...@suse.cz>, "Chris Mason" <c...@fb.com>
> Subject: Re: page allocation stall in kernel 4.9 when copying files from one 
> btrfs hdd to another
> Hi,
> 
> I verified the log files and see no prior oom killer invocation. 
> Unfortunately the machine has been rebooted since. Next time it happens, I 
> will also look in dmesg.
> 
> Thanks,
> David Arendt
> 
> 
> Michal Hocko – Wed., 14. December 2016 11:31
> > Btw. the stall should be preceded by the OOM killer invocation. Could
> > you share the OOM report please. I am asking because such an OOM killer
> > would be clearly pre-mature as per your meminfo. I am trying to change
> > that code and seeing your numbers might help me.
> >
> > Thanks!
> >
> > On Wed 14-12-16 11:17:43, Michal Hocko wrote:
> > > On Tue 13-12-16 18:11:01, David Arendt wrote:
> > > > Hi,
> > > >
> > > > I receive the following page allocation stall while copying lots of
> > > > large files from one btrfs hdd to another.
> > > >
> > > > Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls 
> > > > for 12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
> > > > Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8 
> > > > Tainted: P O 4.9.0 #1
> > > [...]
> > > > Dec 13 13:04:29 server kernel: Call Trace:
> > > > Dec 13 13:04:29 server kernel: [] ? 
> > > > dump_stack+0x46/0x5d
> > > > Dec 13 13:04:29 server kernel: [] ? 
> > > > warn_alloc+0x111/0x130
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > __alloc_pages_nodemask+0xbe8/0xd30
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > pagecache_get_page+0xe4/0x230
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > alloc_extent_buffer+0x10b/0x400
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > btrfs_alloc_tree_block+0x125/0x560
> > >
> > > OK, so this is
> > > find_or_create_page(mapping, index, GFP_NOFS|__GFP_NOFAIL)
> > >
> > > The main question is whether this really needs to be NOFS request...
> > >
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > read_extent_buffer_pages+0x21f/0x280
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > __btrfs_cow_block+0x141/0x580
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > btrfs_cow_block+0x100/0x150
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > btrfs_search_slot+0x1e9/0x9c0
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > __set_extent_bit+0x512/0x550
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > lookup_inline_extent_backref+0xf5/0x5e0
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > set_extent_bit+0x24/0x30
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > update_block_group.isra.34+0x114/0x380
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > __btrfs_free_extent.isra.35+0xf4/0xd20
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > btrfs_merge_delayed_refs+0x61/0x5d0
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > __btrfs_run_delayed_refs+0x902/0x10a0
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > btrfs_run_delayed_refs+0x90/0x2a0
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > delayed_ref_async_start+0x84/0xa0

Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

[admin]

Hi,

I verified the log files and see no prior oom killer invocation. Unfortunately 
the machine has been rebooted since. Next time it happens, I will also look in 
dmesg.

Thanks,
David Arendt 


Michal Hocko – Wed., 14. December 2016 11:31
> Btw. the stall should be preceded by the OOM killer invocation. Could
> you share the OOM report please. I am asking because such an OOM killer
> would be clearly pre-mature as per your meminfo. I am trying to change
> that code and seeing your numbers might help me.
> 
> Thanks!
> 
> On Wed 14-12-16 11:17:43, Michal Hocko wrote:
> > On Tue 13-12-16 18:11:01, David Arendt wrote:
> > > Hi,
> > > 
> > > I receive the following page allocation stall while copying lots of
> > > large files from one btrfs hdd to another.
> > > 
> > > Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls for 
> > > 12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
> > > Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8 
> > > Tainted: P   O4.9.0 #1
> > [...]
> > > Dec 13 13:04:29 server kernel: Call Trace:
> > > Dec 13 13:04:29 server kernel:  [] ? 
> > > dump_stack+0x46/0x5d
> > > Dec 13 13:04:29 server kernel:  [] ? 
> > > warn_alloc+0x111/0x130
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > __alloc_pages_nodemask+0xbe8/0xd30
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > pagecache_get_page+0xe4/0x230
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > alloc_extent_buffer+0x10b/0x400
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > btrfs_alloc_tree_block+0x125/0x560
> > 
> > OK, so this is
> > find_or_create_page(mapping, index, GFP_NOFS|__GFP_NOFAIL)
> > 
> > The main question is whether this really needs to be NOFS request...
> > 
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > read_extent_buffer_pages+0x21f/0x280
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > __btrfs_cow_block+0x141/0x580
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > btrfs_cow_block+0x100/0x150
> > > Dec 13 13:04:33 server kernel:  [] ?  
> > > btrfs_search_slot+0x1e9/0x9c0
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > __set_extent_bit+0x512/0x550
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > lookup_inline_extent_backref+0xf5/0x5e0
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > set_extent_bit+0x24/0x30
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > update_block_group.isra.34+0x114/0x380
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > __btrfs_free_extent.isra.35+0xf4/0xd20
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > btrfs_merge_delayed_refs+0x61/0x5d0
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > __btrfs_run_delayed_refs+0x902/0x10a0
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > btrfs_run_delayed_refs+0x90/0x2a0
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > delayed_ref_async_start+0x84/0xa0
> > 
> > What would cause the reclaim recursion?
> > 
> > > Dec 13 13:04:34 server kernel: Mem-Info:
> > > Dec 13 13:04:34 server kernel: active_anon:20 inactive_anon:34
> > > isolated_anon:0\x0a active_file:7370032 inactive_file:450105
> > > isolated_file:320\x0a unevictable:0 dirty:522748 writeback:189
> > > unstable:0\x0a slab_reclaimable:178255 slab_unreclaimable:124617\x0a
> > > mapped:4236 shmem:0 pagetables:1163 bounce:0\x0a free:38224 free_pcp:241
> > > free_cma:0
> > 
> > This speaks for itself. There is a lot of dirty data, basically no
> > anonymous memory and GFP_NOFS cannot do much to reclaim obviously. This
> > is either a configuraion bug as somebody noted down the thread (setting
> > the dirty_ratio) or suboptimality of the btrfs code which might request
> > NOFS even though it is not strictly necessary. This would be more for
> > btrfs developers.
> > -- 
> > Michal Hocko
> > SUSE Labs
> 
> -- 
> Michal Hocko
> SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs random filesystem corruption in kernel 3.17

[admin]

Summarizing what I've seen on the threads...


First of all many thanks for summarizing the info.


1) The bug seems to be read-only snapshot related.  The connection to
send is that send creates read-only snapshots, but people creating 
read-
only snapshots for other purposes are now reporting the same problem, 
so

it's not send, it's the read-only snapshots.


In fact send does not create a read-only snapshot, snapshots are created 
manually prior to calling send.



2) Writable snapshots haven't been implicated yet, and the working set
from which the snapshots are taken doesn't seem to be affected, either.
So in that sense it's not affecting ordinary usage, only the read-only
snapshots themselves.

3) More problematic, however, is the fact that these apparently 
corrupted

read-only snapshots often are not listed properly and can't be deleted,
tho I'm not sure if that's /all/ the corrupted snapshots or only part 
of

them. So while it may not affect ordinary operation in the short term,
over time until there's a fix, people routinely doing read-only 
snapshots
are going to be getting more and more of these undeletable snapshots, 
and

depending on whether the eventual patch only prevents more or can
actually fix the bad ones (possibly via btrfs check or the like),
affected filesystems may ultimately have to be blown away and recreated
with a fresh mkfs, in ordered to kill the currently undeletable 
snapshots.


So the first thing to do would be to shut off whatever's making 
read-only

snapshots, so you don't make the problem worse while it's being
investigated.  For those who can do that without too big an 
interruption
to their normal routine (who don't depend on send/receive, for 
instance),

just keep it off for the time being.  For those who depend on read-only
snapshots (send-receive for backup and the data is too valuable to not 
do

the backups for a few days), consider switching back to 3.16-stable --
from 3.16.3 at least, the patch for the compress bug is there, so that
shouldn't be a problem.

And if you're affected, be aware that until we have a fix, we don't 
know

if it'll be possible to remove the affected and currently undeletable
snapshots.  If it's not, at some point you'll need to do a fresh
mkfs.btrfs, to get rid of the damage.  Since the bug doesn't appear to
affect writable snapshots or the head from which snapshots are made,
it's not urgent, and a full fix is likely to include a patch to detect
and fix the problem as well, but until we know what the problem is we
can't be sure of that, so be prepared to do that mkfs at some point, as
at this point it's possible that's the only way you'll be able to kill
the corrupted snapshots.


I don't agree with you concerning the not urgent part. In my opinion, 
any problem leading to filesystem or other data corruption should be 
considered as urgent, at least as long as it isn't known what exactly is 
affected and whether there is a simple way to salvage the corruption 
without going the backup/restore route.



4) Total speculation on my part, but given the wanted transid (aka
generation, in different contexts) is significantly lower than the 
found

transid, and the fact that the problem appears to be limited to
/read-only/ snapshots, my first suspicion is that something's getting
updated that would normally apply to all snapshots, but the read-only
nature of the snapshots is preventing the full update there.  The 
transid

of the block is updated, but the snapshot being read-only is preventing
update of the pointer in that snapshot accordingly.

What I do /not/ know is whether the bug is that something's getting
updated that should NOT be, and it's simply the read-only snapshots
letting us know about it since the writable snapshots are fully 
updated,

even if that breaks the snapshot (breaking writable snapshots in a
different and currently undetected way), or if instead, it's a 
legitimate

update, like a balance simply moving the snapshot around but not
affecting it otherwise, and the bug is that the read-only snapshots
aren't allowing the legitimate update.

Either way, this more or less developed over the weekend, and it's 
Monday

now, so the devs should be on it.  If it's anything like the 3.15/3.16
compression bug, it'll take some time for them to properly trace it, 
and
then to figure out an appropriate fix, but they will.  Chances are 
we'll
have at least some decent progress on a trace by Friday, and maybe even 
a

good-to-go patch. =:^)

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Your Unsubscribe Request to us...@tortoisesvn.tigris.org

[admin]

This is to inform you that your recent unsubscribe request was unsuccessful. 
This is probably because we could find no current subscription in your name.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Root fs on raid1

[admin]

Hi,

I have read that for using raid1 btrfs device scan must be run before mounting 
the fs. How do I proceed if root filesystem is mounted from btrfs ?

Thanks in advance
Bye,
David Arendt

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html