Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists

[Kai Krakow]

Am Fri, 10 Feb 2017 23:15:03 +0100
schrieb Marc Joliet :

> # btrfs filesystem df /media/MARCEC_BACKUP/   
> Data, single: total=851.00GiB, used=831.36GiB
> System, DUP: total=64.00MiB, used=120.00KiB
> Metadata, DUP: total=13.00GiB, used=10.38GiB
> Metadata, single: total=1.12GiB, used=0.00B
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> Hmm, I take it that the single metadata is a leftover from running
> --repair?

It's more probably a remnant of an incomplete balance operation or an
older mkfs version. I'd simply rebalance metadata to fix this.

I don't think that btrfs-repair would migrate missing metadata
duplicates back to single profile, it would more likely trigger
recreating the missing duplicates. But I'm not sure.

If it is a result of the repair operation, that could be an
interesting clue. Could it explain "error -17" from your logs? But that
would mean the duplicates were already missing before the repair
operation and triggered that problem. So the question is, why are those
duplicates missing in the first place as a result of normal operation?
From your logs:

---8<--- snip
Feb 02 22:49:14 thetick kernel: BTRFS: device label MARCEC_BACKUP devid
1 transid 283903 /dev/sdd2
Feb 02 22:49:19 thetick kernel: EXT4-fs (sdd1): mounted filesystem with 
ordered data mode. Opts: (null)
Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): use zlib
compression Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2):
disk space caching is enabled
Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): has skinny
extents Feb 03 00:20:09 thetick kernel: BTRFS info (device sdd2): The
free space cache file (3967375376384) is invalid. skip it
Feb 03 01:05:58 thetick kernel: [ cut here ]
Feb 03 01:05:58 thetick kernel: WARNING: CPU: 1 PID: 26544 at
fs/btrfs/extent- tree.c:2967 btrfs_run_delayed_refs+0x26c/0x290
Feb 03 01:05:58 thetick kernel: BTRFS: Transaction aborted (error -17)
--->8--- snap

"error -17" being "object already exists". My only theory would be this
has a direct connection to you finding the single metadata profile.
Like in "the kernel thinks the objects already exists when it really
didn't, and as a result the object is there only once now" aka "single
metadata".

But I'm no dev and no expert on the internals.

-- 
Regards,
Kai

Replies to list-only preferred.


pgp64OIVK8aRx.pgp
Description: Digitale Signatur von OpenPGP

Re: proper syslinux settings for ext4 boot, dm-crypt + btrfs subvol root?

[Hugo Mills]

On Fri, Feb 10, 2017 at 06:05:57PM -0600, John Hendy wrote:
> On Fri, Feb 10, 2017 at 5:49 PM, Hugo Mills  wrote:
> > On Fri, Feb 10, 2017 at 11:40:41AM -0600, John Hendy wrote:
> >> Greetings,
> 
> 
> >Just as an aside -- it doesn't affect you here, but you might want
> > to be aware -- syslinux doesn't support multi-device btrfs. I've been
> > bitten by that before, and it's a consideration for you should you
> > wish to add another device to this FS. Also, I haven't tried it, but I
> > suspect that it may not support compression in btrfs either.
> 
> Aside appreciated! I don't plan to add a second device, so that's alright.
> 
> I've run into the statement on compression as well... does this mean
> the device it's *on* or the device I want to mount as root?

   The FS that the kernel and initramfs live on.

   The bootloader is responsible for finding the two files containing
the kernel and the initramfs on the FS and loading/running them. (It
may also need to find some components if itself on there; GRUB looks
for config files, for example). Once the kernel and initramfs have
started, it's up to the kernel and the early userspace environment to
deal with, and it's not the bootloader's problem.

> I have to assume the former, and ran across this in my preparation,
> which is why I used ext4. Given that I've been able to specify
> rootflags=compress=lzo in my APPEND line, this is also in /etc/fstab
> and it booted... I'm using this as evidence that it must support
> compression on a root btrfs device? Any reason to think otherwise?
> Their wiki appears to talk about just the boot device:
> - http://www.syslinux.org/wiki/index.php?title=Filesystem#Btrfs

   rootflags=... is ignored by the bootloader and is passed to the
initramfs environment.

> Even with ext4, my current arch install is < May 2016 which is when
> mkfs.ext4 auto-implemented some sort of new 64bit feature so that
> caught me off guard. Initially syslinux just spit out a "can't find
> ldlinux.sys" or similar.
> 
> Given I'm early enough here to abort the plan without much fuss, this
> sort of aside is quite nice to consider if there's something
> inherently wrong with this! So far I have a booting arch and I'm using
> debootstrap to manually install Ubuntu since it's installer has ~zero
> customization options to prep your mount points if you've already got
> them ready...

   The main problem with multi-boot environments on btrfs is that most
distribution installers don't know about btrfs subvols, and won't
install on them. debootstrap is about the only way I know of that will
cope with it. (Although I suspect that some of the from-source
distributions like Gentoo will manage it given enough persuasion).

   A secondary problem is that every distribution assumes that it's
the only thing that's managing /boot and the corresponding bootloader,
kernel and initramfs. This has always been problematic -- it was a
problem when I was multi-booting back in 1999 -- and isn't restricted
to btrfs-based environments. I don't have a good solution for managing
it, I'm afraid.

   Hugo.

-- 
Hugo Mills | For months now, we have been making triumphant
hugo@... carfax.org.uk | retreats before a demoralised enemy who is advancing
http://carfax.org.uk/  | in utter disorder.
PGP: E2AB1DE4  |  Eric Frank Russell, Wasp


signature.asc
Description: Digital signature

Re: proper syslinux settings for ext4 boot, dm-crypt + btrfs subvol root?

[John Hendy]

On Fri, Feb 10, 2017 at 5:49 PM, Hugo Mills  wrote:
> On Fri, Feb 10, 2017 at 11:40:41AM -0600, John Hendy wrote:
>> Greetings,


>Just as an aside -- it doesn't affect you here, but you might want
> to be aware -- syslinux doesn't support multi-device btrfs. I've been
> bitten by that before, and it's a consideration for you should you
> wish to add another device to this FS. Also, I haven't tried it, but I
> suspect that it may not support compression in btrfs either.

Aside appreciated! I don't plan to add a second device, so that's alright.

I've run into the statement on compression as well... does this mean
the device it's *on* or the device I want to mount as root?

I have to assume the former, and ran across this in my preparation,
which is why I used ext4. Given that I've been able to specify
rootflags=compress=lzo in my APPEND line, this is also in /etc/fstab
and it booted... I'm using this as evidence that it must support
compression on a root btrfs device? Any reason to think otherwise?
Their wiki appears to talk about just the boot device:
- http://www.syslinux.org/wiki/index.php?title=Filesystem#Btrfs

Even with ext4, my current arch install is < May 2016 which is when
mkfs.ext4 auto-implemented some sort of new 64bit feature so that
caught me off guard. Initially syslinux just spit out a "can't find
ldlinux.sys" or similar.

Given I'm early enough here to abort the plan without much fuss, this
sort of aside is quite nice to consider if there's something
inherently wrong with this! So far I have a booting arch and I'm using
debootstrap to manually install Ubuntu since it's installer has ~zero
customization options to prep your mount points if you've already got
them ready...


Thanks,
John

>
>Hugo.
>
> --
> Hugo Mills | All mushrooms are edible, but some are only edible
> hugo@... carfax.org.uk | once.
> http://carfax.org.uk/  |
> PGP: E2AB1DE4  |
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: proper syslinux settings for ext4 boot, dm-crypt + btrfs subvol root?

[Hugo Mills]

On Fri, Feb 10, 2017 at 11:40:41AM -0600, John Hendy wrote:
> Greetings,
> 
> 
> I'm trying to utilize btrfs subvols to allow me to boot separate
> distros without having to create so many partitions. I'm on Arch Linux
> and not finding the wiki super helpful with specifics on dm-crypt/luks
> and btrfs.
> 
> Requested info:
> 
> $ uname -a
> Linux arch_zbook 4.9.8-1-ARCH #1 SMP PREEMPT Mon Feb 6 12:59:40 CET
> 2017 x86_64 GNU/Linux
> 
> $ btrfs --version
> btrfs-progs v4.9
> 
> $ sudo btrfs fi show
> Label: 'bigD'  uuid: e1284231-c264-4944-807d-5fcb1832ce47
> Total devices 1 FS bytes used 1.27GiB
> devid1 size 232.38GiB used 3.02GiB path /dev/mapper/btr
[snip] 
> My functioning syslinux.cfg:
[snip]
> Current behavior is I successfully get a syslinux boot menu (at least
> after I disabled the 64bit default option), but selecting the above
> entry just refreshes the menu and I'm stuck there with an automatic
> countdown from 5sec and then back to 5sec when it hits 0. It's like it
> just keeps pointing at the /dev/sdb MBR or can't find something and
> thus tries again?

   Just as an aside -- it doesn't affect you here, but you might want
to be aware -- syslinux doesn't support multi-device btrfs. I've been
bitten by that before, and it's a consideration for you should you
wish to add another device to this FS. Also, I haven't tried it, but I
suspect that it may not support compression in btrfs either.

   Hugo.

-- 
Hugo Mills | All mushrooms are edible, but some are only edible
hugo@... carfax.org.uk | once.
http://carfax.org.uk/  |
PGP: E2AB1DE4  |


signature.asc
Description: Digital signature

Re: proper syslinux settings for ext4 boot, dm-crypt + btrfs subvol root?

[John Hendy]

On Fri, Feb 10, 2017 at 2:03 PM, Chris Murphy  wrote:
> This is not a Btrfs problem yet. Syslinux isn't even getting the
> kernel and initramfs loaded, otherwise you'd get a failure during
> boot. But instead you're at the bootloader still.
>
> One thing to realize is that syslinux only finds kernels and initramfs
> on the local file system. So assuming this syslinux is on /dev/sda1:
> UUID="37441f68... ext2 volume; then it can only load kernels and
> initramfs from that file system. So I'm not sure what the purpose of
> /dev/sdb1: UUID="e046823f ext4 is for, but if you use it for the boot
> volume, you have to use extlinux install extlinux there *and* you have
> to make sure the MBR has sdb1's active bit set *and* you have to make
> sure the firmware boot manager is pointed to /dev/sdb.

For sure. The whole install is on /dev/sdb
- sdb1 is ext4 /boot partition
- sdb1 was mounted to /path/to/boot when I installed Arch
- I did extlinux --install on sdb1
- boot flag set on sdb1
- syslinux mbr dd'd to /dev/sdb

I think that part is ok.

>
> But I don't think that's necessary, mainly you just need a really big
> ext4 volume to hold all the distro kernels in one volume so that
> syslinux has access to them. If you want separate boot volumes for
> each distro then you have no choice but to use GRUB which can find and
> load kernel/initramfs on any drive, any volume, with pretty much any
> file system.

This is encouraging as I was particularly hoping to *avoid* separate
boot partitions.

>
> So the question is why won't that boot entry load the already working
> kernel and initramfs? I don't have a good explanation for that. What
> should happen if anything in APPEND is wrong, is you still get kernel
> and initramfs loading, but then end up at a dracut shell. Or maybe it
> crashes earlier due to some incompatibility by mixing kernel/initramfs
> with the wrong rootfs (different kernel versions for example since all
> the modules are version specific and those will be on Btrfs not on
> boot).
>

I did a couple of things after I sent this. For one, I ditched
arch/root. Now I just have arch. That let me get away from the call to
a nested subvol.

That on it's own didn't work, but I started playing with the
rootflags. I haven't tried one by one, but re-specifying my mount
options seems to work. So instead of just rootflags=subvol=arch, I
have:

rootflags=compress=lzo,subvol=arch,ssd,discard

Is there a reason that would make a difference? I started wondering
about the compression immediately and I admit I'm not savvy on how
that works. When I installed things were mounted with compress=lzo.
Would this mean syslinux was maybe trying to mount it uncompressed and
thus files are jumbled (like trying to run cat on a .zip)? Just wild
imagining here.

Short story is I can now boot! I just want to figure out what happened
and update the Arch wiki as I haven't seen much definitive material on
this.

> I'd say you need some syslinux debug option set to find out what it's
> tripping up on, because there's not enough information without that.
> Maybe there's something in APPEND that it doesn't like, but I don't
> think syslinux parses that line, it just adds it to the kernel boot
> param line handed off to the kernel. But it's not even getting that
> far.
>
> The UUID e1284231- is the correct one to use for Btrfs.
>
> I have not tried a double nested subvolume. I think you're better off
> with directories at the top level for each distro, and then use
> subvolumes in those directories for root and home. Note that the
> Ubuntu and opensuse GUI installers will make this impossible, they
> will not install to a user defined subvolume, they insist on creating
> things at the top level. Fedora's installer will let you do this
> except for rootfs, it requires that it be a new subvolume which you
> can name but the UI won't let you stick it in a directory, or create
> nested subvolumes. You could troubleshoot your existing APPEND by
> using rootflags=subvolid= and specifying the ID of the arch/root
> subvolume. But I'm suspicious whether that's a problem. You could try
> removing the leading / because the path to the subvolume is really
> subvol=arch/root. Hmmm.

As a separate question, I noticed that if I did:

mount -o subvol=arch/root /mnt/btr  ## note no leading slash

and then looked at `genfstab -p -U /mnt/btr`, I got both of these in
the options:

subvol=arch/root, subvol=/arch/root

So that tripped me up big time. In blogs/forums I don't tend to see
folks specifying a leading slash, but the system seemed to use it?


Thanks for the assistance!
John


>
> That's all I'm thinking of at the moment.
>
> Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] Btrfs: fix btrfs_decompress_buf2page()

[Omar Sandoval]

From: Omar Sandoval 

If btrfs_decompress_buf2page() is handed a bio with its page in the
middle of the working buffer, then we adjust the offset into the working
buffer. After we copy into the bio, we advance the iterator by the
number of bytes we copied. Then, we have some logic to handle the case
of discontiguous pages and adjust the offset into the working buffer
again. However, if we didn't advance the bio to a new page, we may enter
this case in error, essentially repeating the adjustment that we already
made when we entered the function. The end result is bogus data in the
bio.

Previously, we only checked for this case when we advanced to a new
page, but the conversion to bio iterators changed that. This restores
the old, correct behavior.

A case I saw when testing with zlib was:

buf_start = 42769
total_out = 46865
working_bytes = total_out - buf_start = 4096
start_byte = 45056

The condition (total_out > start_byte && buf_start < start_byte) is
true, so we adjust the offset:

buf_offset = start_byte - buf_start = 2287
working_bytes -= buf_offset = 1809
current_buf_start = buf_start = 42769

Then, we copy

bytes = min(bvec.bv_len, PAGE_SIZE - buf_offset, working_bytes) = 1809
buf_offset += bytes = 4096
working_bytes -= bytes = 0
current_buf_start += bytes = 44578

After bio_advance(), we are still in the same page, so start_byte is the
same. Then, we check (total_out > start_byte && current_buf_start < start_byte),
which is true! So, we adjust the values again:

buf_offset = start_byte - buf_start = 2287
working_bytes = total_out - start_byte = 1809
current_buf_start = buf_start + buf_offset = 45056

But note that working_bytes was already zero before this, so we should
have stopped copying.

Fixes: 974b1adc3b10 ("btrfs: use bio iterators for the decompression handlers")
Reported-by: Pat Erley 
Reviewed-by: Chris Mason 
Signed-off-by: Omar Sandoval 
---
Changed from v1:

- Check if start_byte changed instead of whether bv_offset == 0. This is
  a little more foolproof.

 fs/btrfs/compression.c | 37 -
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index 7f390849343b..25168852ccde 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -1024,6 +1024,7 @@ int btrfs_decompress_buf2page(char *buf, unsigned long 
buf_start,
unsigned long buf_offset;
unsigned long current_buf_start;
unsigned long start_byte;
+   unsigned long prev_start_byte;
unsigned long working_bytes = total_out - buf_start;
unsigned long bytes;
char *kaddr;
@@ -1071,26 +1072,28 @@ int btrfs_decompress_buf2page(char *buf, unsigned long 
buf_start,
if (!bio->bi_iter.bi_size)
return 0;
bvec = bio_iter_iovec(bio, bio->bi_iter);
-
+   prev_start_byte = start_byte;
start_byte = page_offset(bvec.bv_page) - disk_start;
 
-   /*
-* make sure our new page is covered by this
-* working buffer
-*/
-   if (total_out <= start_byte)
-   return 1;
+   if (start_byte != prev_start_byte) {
+   /*
+* make sure our new page is covered by this
+* working buffer
+*/
+   if (total_out <= start_byte)
+   return 1;
 
-   /*
-* the next page in the biovec might not be adjacent
-* to the last page, but it might still be found
-* inside this working buffer. bump our offset pointer
-*/
-   if (total_out > start_byte &&
-   current_buf_start < start_byte) {
-   buf_offset = start_byte - buf_start;
-   working_bytes = total_out - start_byte;
-   current_buf_start = buf_start + buf_offset;
+   /*
+* the next page in the biovec might not be adjacent
+* to the last page, but it might still be found
+* inside this working buffer. bump our offset pointer
+*/
+   if (total_out > start_byte &&
+   current_buf_start < start_byte) {
+   buf_offset = start_byte - buf_start;
+   working_bytes = total_out - start_byte;
+   current_buf_start = buf_start + buf_offset;
+   }
}
}
 
-- 
2.11.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More 

Re: [PATCH v2] Btrfs: fix btrfs_decompress_buf2page()

[Omar Sandoval]

On Fri, Feb 10, 2017 at 03:03:35PM -0800, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> If btrfs_decompress_buf2page() is handed a bio with its page in the
> middle of the working buffer, then we adjust the offset into the working
> buffer. After we copy into the bio, we advance the iterator by the
> number of bytes we copied. Then, we have some logic to handle the case
> of discontiguous pages and adjust the offset into the working buffer
> again. However, if we didn't advance the bio to a new page, we may enter
> this case in error, essentially repeating the adjustment that we already
> made when we entered the function. The end result is bogus data in the
> bio.
> 
> Previously, we only checked for this case when we advanced to a new
> page, but the conversion to bio iterators changed that. This restores
> the old, correct behavior.
> 
> A case I saw when testing with zlib was:
> 
> buf_start = 42769
> total_out = 46865
> working_bytes = total_out - buf_start = 4096
> start_byte = 45056
> 
> The condition (total_out > start_byte && buf_start < start_byte) is
> true, so we adjust the offset:
> 
> buf_offset = start_byte - buf_start = 2287
> working_bytes -= buf_offset = 1809
> current_buf_start = buf_start = 42769
> 
> Then, we copy
> 
> bytes = min(bvec.bv_len, PAGE_SIZE - buf_offset, working_bytes) = 1809
> buf_offset += bytes = 4096
> working_bytes -= bytes = 0
> current_buf_start += bytes = 44578
> 
> After bio_advance(), we are still in the same page, so start_byte is the
> same. Then, we check (total_out > start_byte && current_buf_start < 
> start_byte),
> which is true! So, we adjust the values again:
> 
> buf_offset = start_byte - buf_start = 2287
> working_bytes = total_out - start_byte = 1809
> current_buf_start = buf_start + buf_offset = 45056
> 
> But note that working_bytes was already zero before this, so we should
> have stopped copying.
> 
> Fixes: 974b1adc3b10 ("btrfs: use bio iterators for the decompression 
> handlers")
> Reported-by: Pat Erley 
> Reviewed-by: Chris Mason 
> Signed-off-by: Omar Sandoval 

Whoops, I sent this out before I got your other email, Bo, but I assume
it's still okay for Chris to add your reviewed-by and tested-by?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix btrfs_decompress_buf2page()

[Liu Bo]

On Fri, Feb 10, 2017 at 02:52:11PM -0800, Omar Sandoval wrote:
> On Fri, Feb 10, 2017 at 02:46:09PM -0800, Liu Bo wrote:
> > On Fri, Feb 10, 2017 at 12:15:11PM -0800, Omar Sandoval wrote:
> > > From: Omar Sandoval 
> > > 
> > > If btrfs_decompress_buf2page() is handed a bio with its page in the
> > > middle of the working buffer, then we adjust the offset into the working
> > > buffer. After we copy into the bio, we advance the iterator by the
> > > number of bytes we copied. Then, we have some logic to handle the case
> > > of discontiguous pages and adjust the offset into the working buffer
> > > again. However, if we didn't advance the bio to a new page, we may enter
> > > this case in error, essentially repeating the adjustment that we already
> > > made when we entered the function. The end result is bogus data in the
> > > bio.
> > > 
> > > Previously, we only checked for this case when we advanced to a new
> > > page, but the conversion to bio iterators changed that. This restores
> > > the old, correct behavior.
> > 
> > The fix looks good to me, just one comment below.
> > 
> > > 
> > > Fixes: 974b1adc3b10 ("btrfs: use bio iterators for the decompression 
> > > handlers")
> > > Reported-by: Pat Erley 
> > > Signed-off-by: Omar Sandoval 
> > > ---
> > > A case I saw when testing with zlib was:
> > > 
> > > buf_start = 42769
> > > total_out = 46865
> > > working_bytes = total_out - buf_start = 4096
> > > start_byte = 45056
> > > 
> > > 
> > > The condition (total_out > start_byte && buf_start < start_byte) is
> > > true, so we adjust the offset:
> > > 
> > > buf_offset = start_byte - buf_start = 2287
> > > working_bytes -= buf_offset = 1809
> > > current_buf_start = buf_start = 42769
> > > 
> > > Then, we copy
> > > 
> > > bytes = min(bvec.bv_len, PAGE_SIZE - buf_offset, working_bytes) = 1809
> > > buf_offset += bytes = 4096
> > > working_bytes -= bytes = 0
> > > current_buf_start += bytes = 44578
> > > 
> > > After bio_advance(), we are still in the same page, so start_byte is the
> > > same. Then, we check (total_out > start_byte && current_buf_start < 
> > > start_byte),
> > > which is true! So, we adjust the values again:
> > > 
> > > buf_offset = start_byte - buf_start = 2287
> > > working_bytes = total_out - start_byte = 1809
> > > current_buf_start = buf_start + buf_offset = 45056
> > > 
> > > But note that working_bytes was already zero before this, so we should
> > > have stopped copying.
> > > 
> > >  fs/btrfs/compression.c | 36 +++-
> > >  1 file changed, 19 insertions(+), 17 deletions(-)
> > > 
> > > diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
> > > index 7f390849343b..f9f22976d77d 100644
> > > --- a/fs/btrfs/compression.c
> > > +++ b/fs/btrfs/compression.c
> > > @@ -1072,25 +1072,27 @@ int btrfs_decompress_buf2page(char *buf, unsigned 
> > > long buf_start,
> > >   return 0;
> > >   bvec = bio_iter_iovec(bio, bio->bi_iter);
> > >  
> > > - start_byte = page_offset(bvec.bv_page) - disk_start;
> > > + if (bvec.bv_offset == 0) {
> > > + start_byte = page_offset(bvec.bv_page) - disk_start;
> > 
> > I'm not fully convinced that the next bvec's bv_offset is always
> > zero, since the pages are all locked, can we keep a orig_page and
> > check if (orig_page == bvec.bv_page)?
> 
> That's a good point, that's more foolproof. I'll send a v2.

With that, you can have

Reviewed-by: Liu Bo 
Tested-by: Liu Bo 

We may add ASSERT(bvec.bv_offset == 0) in the if statement of
(orig_page == bvec.bv_page), then we could know whether it's true :)

Thanks,

-liubo
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix btrfs_decompress_buf2page()

[Omar Sandoval]

On Fri, Feb 10, 2017 at 02:46:09PM -0800, Liu Bo wrote:
> On Fri, Feb 10, 2017 at 12:15:11PM -0800, Omar Sandoval wrote:
> > From: Omar Sandoval 
> > 
> > If btrfs_decompress_buf2page() is handed a bio with its page in the
> > middle of the working buffer, then we adjust the offset into the working
> > buffer. After we copy into the bio, we advance the iterator by the
> > number of bytes we copied. Then, we have some logic to handle the case
> > of discontiguous pages and adjust the offset into the working buffer
> > again. However, if we didn't advance the bio to a new page, we may enter
> > this case in error, essentially repeating the adjustment that we already
> > made when we entered the function. The end result is bogus data in the
> > bio.
> > 
> > Previously, we only checked for this case when we advanced to a new
> > page, but the conversion to bio iterators changed that. This restores
> > the old, correct behavior.
> 
> The fix looks good to me, just one comment below.
> 
> > 
> > Fixes: 974b1adc3b10 ("btrfs: use bio iterators for the decompression 
> > handlers")
> > Reported-by: Pat Erley 
> > Signed-off-by: Omar Sandoval 
> > ---
> > A case I saw when testing with zlib was:
> > 
> > buf_start = 42769
> > total_out = 46865
> > working_bytes = total_out - buf_start = 4096
> > start_byte = 45056
> > 
> > 
> > The condition (total_out > start_byte && buf_start < start_byte) is
> > true, so we adjust the offset:
> > 
> > buf_offset = start_byte - buf_start = 2287
> > working_bytes -= buf_offset = 1809
> > current_buf_start = buf_start = 42769
> > 
> > Then, we copy
> > 
> > bytes = min(bvec.bv_len, PAGE_SIZE - buf_offset, working_bytes) = 1809
> > buf_offset += bytes = 4096
> > working_bytes -= bytes = 0
> > current_buf_start += bytes = 44578
> > 
> > After bio_advance(), we are still in the same page, so start_byte is the
> > same. Then, we check (total_out > start_byte && current_buf_start < 
> > start_byte),
> > which is true! So, we adjust the values again:
> > 
> > buf_offset = start_byte - buf_start = 2287
> > working_bytes = total_out - start_byte = 1809
> > current_buf_start = buf_start + buf_offset = 45056
> > 
> > But note that working_bytes was already zero before this, so we should
> > have stopped copying.
> > 
> >  fs/btrfs/compression.c | 36 +++-
> >  1 file changed, 19 insertions(+), 17 deletions(-)
> > 
> > diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
> > index 7f390849343b..f9f22976d77d 100644
> > --- a/fs/btrfs/compression.c
> > +++ b/fs/btrfs/compression.c
> > @@ -1072,25 +1072,27 @@ int btrfs_decompress_buf2page(char *buf, unsigned 
> > long buf_start,
> > return 0;
> > bvec = bio_iter_iovec(bio, bio->bi_iter);
> >  
> > -   start_byte = page_offset(bvec.bv_page) - disk_start;
> > +   if (bvec.bv_offset == 0) {
> > +   start_byte = page_offset(bvec.bv_page) - disk_start;
> 
> I'm not fully convinced that the next bvec's bv_offset is always
> zero, since the pages are all locked, can we keep a orig_page and
> check if (orig_page == bvec.bv_page)?

That's a good point, that's more foolproof. I'll send a v2.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix btrfs_decompress_buf2page()

[Liu Bo]

On Fri, Feb 10, 2017 at 12:15:11PM -0800, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> If btrfs_decompress_buf2page() is handed a bio with its page in the
> middle of the working buffer, then we adjust the offset into the working
> buffer. After we copy into the bio, we advance the iterator by the
> number of bytes we copied. Then, we have some logic to handle the case
> of discontiguous pages and adjust the offset into the working buffer
> again. However, if we didn't advance the bio to a new page, we may enter
> this case in error, essentially repeating the adjustment that we already
> made when we entered the function. The end result is bogus data in the
> bio.
> 
> Previously, we only checked for this case when we advanced to a new
> page, but the conversion to bio iterators changed that. This restores
> the old, correct behavior.

The fix looks good to me, just one comment below.

> 
> Fixes: 974b1adc3b10 ("btrfs: use bio iterators for the decompression 
> handlers")
> Reported-by: Pat Erley 
> Signed-off-by: Omar Sandoval 
> ---
> A case I saw when testing with zlib was:
> 
> buf_start = 42769
> total_out = 46865
> working_bytes = total_out - buf_start = 4096
> start_byte = 45056
> 
> 
> The condition (total_out > start_byte && buf_start < start_byte) is
> true, so we adjust the offset:
> 
> buf_offset = start_byte - buf_start = 2287
> working_bytes -= buf_offset = 1809
> current_buf_start = buf_start = 42769
> 
> Then, we copy
> 
> bytes = min(bvec.bv_len, PAGE_SIZE - buf_offset, working_bytes) = 1809
> buf_offset += bytes = 4096
> working_bytes -= bytes = 0
> current_buf_start += bytes = 44578
> 
> After bio_advance(), we are still in the same page, so start_byte is the
> same. Then, we check (total_out > start_byte && current_buf_start < 
> start_byte),
> which is true! So, we adjust the values again:
> 
> buf_offset = start_byte - buf_start = 2287
> working_bytes = total_out - start_byte = 1809
> current_buf_start = buf_start + buf_offset = 45056
> 
> But note that working_bytes was already zero before this, so we should
> have stopped copying.
> 
>  fs/btrfs/compression.c | 36 +++-
>  1 file changed, 19 insertions(+), 17 deletions(-)
> 
> diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
> index 7f390849343b..f9f22976d77d 100644
> --- a/fs/btrfs/compression.c
> +++ b/fs/btrfs/compression.c
> @@ -1072,25 +1072,27 @@ int btrfs_decompress_buf2page(char *buf, unsigned 
> long buf_start,
>   return 0;
>   bvec = bio_iter_iovec(bio, bio->bi_iter);
>  
> - start_byte = page_offset(bvec.bv_page) - disk_start;
> + if (bvec.bv_offset == 0) {
> + start_byte = page_offset(bvec.bv_page) - disk_start;

I'm not fully convinced that the next bvec's bv_offset is always
zero, since the pages are all locked, can we keep a orig_page and
check if (orig_page == bvec.bv_page)?

Thanks,

-liubo
>  
> - /*
> -  * make sure our new page is covered by this
> -  * working buffer
> -  */
> - if (total_out <= start_byte)
> - return 1;
> + /*
> +  * make sure our new page is covered by this
> +  * working buffer
> +  */
> + if (total_out <= start_byte)
> + return 1;
>  
> - /*
> -  * the next page in the biovec might not be adjacent
> -  * to the last page, but it might still be found
> -  * inside this working buffer. bump our offset pointer
> -  */
> - if (total_out > start_byte &&
> - current_buf_start < start_byte) {
> - buf_offset = start_byte - buf_start;
> - working_bytes = total_out - start_byte;
> - current_buf_start = buf_start + buf_offset;
> + /*
> +  * the next page in the biovec might not be adjacent
> +  * to the last page, but it might still be found
> +  * inside this working buffer. bump our offset pointer
> +  */
> + if (total_out > start_byte &&
> + current_buf_start < start_byte) {
> + buf_offset = start_byte - buf_start;
> + working_bytes = total_out - start_byte;
> + current_buf_start = buf_start + buf_offset;
> + }
>   }
>   }
>  
> -- 
> 2.11.1
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists

[Marc Joliet]

Sorry for the late reply, see below for why :) .

On Friday 03 February 2017 23:44:10 Kai Krakow wrote:
> Am Thu, 02 Feb 2017 13:01:03 +0100
> 
> schrieb Marc Joliet :
[...]
> > > Btrfs --repair refused to repair the filesystem telling me something
> > > about compressed extents and an unsupported case, wanting me to
> > > take an image and send it to the devs. *sigh*
> > 
> > I haven't tried a repair yet; it's a big file system, and btrfs-check
> > is still running:
> > 
> > # btrfs check -p /dev/sdd2
> > Checking filesystem on /dev/sdd2
> > UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
> > parent transid verify failed on 3829276291072 wanted 224274 found
> > 283858 parent transid verify failed on 3829276291072 wanted 224274
> > found 283858 parent transid verify failed on 3829276291072 wanted
> > 224274 found 283858 parent transid verify failed on 3829276291072
> > wanted 224274 found 283858 Ignoring transid failure
> > leaf parent key incorrect 3829276291072
> > bad block 3829276291072
> > 
> > ERROR: errors found in extent allocation tree or chunk allocation
> > block group 4722282987520 has wrong amount of free space
> > failed to load free space cache for block group 4722282987520
> > checking free space cache [O]
> > root 32018 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32089 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32091 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32092 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32107 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32189 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32190 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32191 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32265 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32266 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32409 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32410 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32411 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32412 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32413 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32631 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32632 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32633 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32634 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32635 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32636 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32718 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > root 32732 inode 95066 errors 100, file extent discount
> > 
> > Found file extent holes:
> > start: 413696, len: 4096
> > 
> > checking fs roots [o]
> > 
> > I know that the "file extend discount" errors are fixable from my
> > previous email to this ML, but what about the rest?  From looking
> > through the ML archives it seems that --repair won't be able to fix
> > the transid failures.  It seems that one person had success with the
> > "usebackuproot" mount option, which I haven't tried yet.

Well, it turns out that the problem fixed itself.  After check finished, I 
tried mounting the FS normally, which succeeded.  Shortly after, btrfs-cleaner 
started removing deleted snapshots.  About 40 minutes later I got the error 
again:

[PATCH] Btrfs: fix btrfs_decompress_buf2page()

[Omar Sandoval]

From: Omar Sandoval 

If btrfs_decompress_buf2page() is handed a bio with its page in the
middle of the working buffer, then we adjust the offset into the working
buffer. After we copy into the bio, we advance the iterator by the
number of bytes we copied. Then, we have some logic to handle the case
of discontiguous pages and adjust the offset into the working buffer
again. However, if we didn't advance the bio to a new page, we may enter
this case in error, essentially repeating the adjustment that we already
made when we entered the function. The end result is bogus data in the
bio.

Previously, we only checked for this case when we advanced to a new
page, but the conversion to bio iterators changed that. This restores
the old, correct behavior.

Fixes: 974b1adc3b10 ("btrfs: use bio iterators for the decompression handlers")
Reported-by: Pat Erley 
Signed-off-by: Omar Sandoval 
---
A case I saw when testing with zlib was:

buf_start = 42769
total_out = 46865
working_bytes = total_out - buf_start = 4096
start_byte = 45056


The condition (total_out > start_byte && buf_start < start_byte) is
true, so we adjust the offset:

buf_offset = start_byte - buf_start = 2287
working_bytes -= buf_offset = 1809
current_buf_start = buf_start = 42769

Then, we copy

bytes = min(bvec.bv_len, PAGE_SIZE - buf_offset, working_bytes) = 1809
buf_offset += bytes = 4096
working_bytes -= bytes = 0
current_buf_start += bytes = 44578

After bio_advance(), we are still in the same page, so start_byte is the
same. Then, we check (total_out > start_byte && current_buf_start < start_byte),
which is true! So, we adjust the values again:

buf_offset = start_byte - buf_start = 2287
working_bytes = total_out - start_byte = 1809
current_buf_start = buf_start + buf_offset = 45056

But note that working_bytes was already zero before this, so we should
have stopped copying.

 fs/btrfs/compression.c | 36 +++-
 1 file changed, 19 insertions(+), 17 deletions(-)

diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index 7f390849343b..f9f22976d77d 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -1072,25 +1072,27 @@ int btrfs_decompress_buf2page(char *buf, unsigned long 
buf_start,
return 0;
bvec = bio_iter_iovec(bio, bio->bi_iter);
 
-   start_byte = page_offset(bvec.bv_page) - disk_start;
+   if (bvec.bv_offset == 0) {
+   start_byte = page_offset(bvec.bv_page) - disk_start;
 
-   /*
-* make sure our new page is covered by this
-* working buffer
-*/
-   if (total_out <= start_byte)
-   return 1;
+   /*
+* make sure our new page is covered by this
+* working buffer
+*/
+   if (total_out <= start_byte)
+   return 1;
 
-   /*
-* the next page in the biovec might not be adjacent
-* to the last page, but it might still be found
-* inside this working buffer. bump our offset pointer
-*/
-   if (total_out > start_byte &&
-   current_buf_start < start_byte) {
-   buf_offset = start_byte - buf_start;
-   working_bytes = total_out - start_byte;
-   current_buf_start = buf_start + buf_offset;
+   /*
+* the next page in the biovec might not be adjacent
+* to the last page, but it might still be found
+* inside this working buffer. bump our offset pointer
+*/
+   if (total_out > start_byte &&
+   current_buf_start < start_byte) {
+   buf_offset = start_byte - buf_start;
+   working_bytes = total_out - start_byte;
+   current_buf_start = buf_start + buf_offset;
+   }
}
}
 
-- 
2.11.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: proper syslinux settings for ext4 boot, dm-crypt + btrfs subvol root?

[Chris Murphy]

This is not a Btrfs problem yet. Syslinux isn't even getting the
kernel and initramfs loaded, otherwise you'd get a failure during
boot. But instead you're at the bootloader still.

One thing to realize is that syslinux only finds kernels and initramfs
on the local file system. So assuming this syslinux is on /dev/sda1:
UUID="37441f68... ext2 volume; then it can only load kernels and
initramfs from that file system. So I'm not sure what the purpose of
/dev/sdb1: UUID="e046823f ext4 is for, but if you use it for the boot
volume, you have to use extlinux install extlinux there *and* you have
to make sure the MBR has sdb1's active bit set *and* you have to make
sure the firmware boot manager is pointed to /dev/sdb.

But I don't think that's necessary, mainly you just need a really big
ext4 volume to hold all the distro kernels in one volume so that
syslinux has access to them. If you want separate boot volumes for
each distro then you have no choice but to use GRUB which can find and
load kernel/initramfs on any drive, any volume, with pretty much any
file system.

So the question is why won't that boot entry load the already working
kernel and initramfs? I don't have a good explanation for that. What
should happen if anything in APPEND is wrong, is you still get kernel
and initramfs loading, but then end up at a dracut shell. Or maybe it
crashes earlier due to some incompatibility by mixing kernel/initramfs
with the wrong rootfs (different kernel versions for example since all
the modules are version specific and those will be on Btrfs not on
boot).

I'd say you need some syslinux debug option set to find out what it's
tripping up on, because there's not enough information without that.
Maybe there's something in APPEND that it doesn't like, but I don't
think syslinux parses that line, it just adds it to the kernel boot
param line handed off to the kernel. But it's not even getting that
far.

The UUID e1284231- is the correct one to use for Btrfs.

I have not tried a double nested subvolume. I think you're better off
with directories at the top level for each distro, and then use
subvolumes in those directories for root and home. Note that the
Ubuntu and opensuse GUI installers will make this impossible, they
will not install to a user defined subvolume, they insist on creating
things at the top level. Fedora's installer will let you do this
except for rootfs, it requires that it be a new subvolume which you
can name but the UI won't let you stick it in a directory, or create
nested subvolumes. You could troubleshoot your existing APPEND by
using rootflags=subvolid= and specifying the ID of the arch/root
subvolume. But I'm suspicious whether that's a problem. You could try
removing the leading / because the path to the subvolume is really
subvol=arch/root. Hmmm.

That's all I'm thinking of at the moment.

Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: remove unused ulist members

[Liu Bo]

On Thu, Feb 09, 2017 at 04:47:43PM +0100, David Sterba wrote:
> Commit "btrfs: ulist: Add ulist_del() function" (d4b804045924d7f8)
> removed some debugging code but left the structure defintions.
> 

Reviewed-by: Liu Bo 

> Signed-off-by: David Sterba 
> ---
>  fs/btrfs/ulist.h | 7 ---
>  1 file changed, 7 deletions(-)
> 
> diff --git a/fs/btrfs/ulist.h b/fs/btrfs/ulist.h
> index a01a2c45825f..007b22fff3f9 100644
> --- a/fs/btrfs/ulist.h
> +++ b/fs/btrfs/ulist.h
> @@ -19,9 +19,6 @@
>   *
>   */
>  struct ulist_iterator {
> -#ifdef CONFIG_BTRFS_DEBUG
> - int i;
> -#endif
>   struct list_head *cur_list;  /* hint to start search */
>  };
>  
> @@ -32,10 +29,6 @@ struct ulist_node {
>   u64 val;/* value to store */
>   u64 aux;/* auxiliary value saved along with the val */
>  
> -#ifdef CONFIG_BTRFS_DEBUG
> - int seqnum; /* sequence number this node is added */
> -#endif
> -
>   struct list_head list;  /* used to link node */
>   struct rb_node rb_node; /* used to speed up search */
>  };
> -- 
> 2.10.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

proper syslinux settings for ext4 boot, dm-crypt + btrfs subvol root?

[John Hendy]

Greetings,


I'm trying to utilize btrfs subvols to allow me to boot separate
distros without having to create so many partitions. I'm on Arch Linux
and not finding the wiki super helpful with specifics on dm-crypt/luks
and btrfs.

Requested info:

$ uname -a
Linux arch_zbook 4.9.8-1-ARCH #1 SMP PREEMPT Mon Feb 6 12:59:40 CET
2017 x86_64 GNU/Linux

$ btrfs --version
btrfs-progs v4.9

$ sudo btrfs fi show
Label: 'bigD'  uuid: e1284231-c264-4944-807d-5fcb1832ce47
Total devices 1 FS bytes used 1.27GiB
devid1 size 232.38GiB used 3.02GiB path /dev/mapper/btr

$ btrfs fi df /mnt/btr/
Data, single: total=2.01GiB, used=1.21GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=1.01GiB, used=56.45MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

/dev/sdb1 is ext4 for /boot
/dev/sdb2 is a luks/dm-crypt device which I've opened as /dev/mapper/btr

I have a top level arch subvol with root and home undeneath. I'll
create a similar set for an Ubuntu install I need for work, a separate
shared data subvol, and then one for snapshots. I'm ignoring those and
would just like to get arch linux working properly.

Here's the setup I'm looking for once booted, with the /mnt/btf becoming /

$ mount
/dev/mapper/btr on /mnt/btr type btrfs
(rw,relatime,ssd,space_cache,subvolid=259,subvol=/arch/root)
/dev/mapper/btr on /mnt/btr/home type btrfs
(rw,relatime,ssd,space_cache,subvolid=260,subvol=/arch/home)
/dev/sdb1 on /mnt/btr/boot type ext4 (rw,relatime,data=ordered)

My functioning syslinux.cfg:

LABEL arch-ssd
MENU LABEL arch-ssd-uuid
LINUX ../vmlinuz-linux
APPEND luks.uuid=7101e83b-31c0-4cdf-bc07-678e00e19c32
root=UUID=eb20c219-0df8-4051-bad2-39d57aed7b59 luks.allow-discards rw
INITRD ../intel-ucode.img,../initramfs-linux.img

I tried this:

LABEL arch-btrfs
MENU LABEL arch-btrfs-uuid
LINUX ../vmlinuz-linux
APPEND luks.uuid=bcacb6d5-2874-4652-a25d-88b0bf3bbce8
root=UUID=e1284231-c264-4944-807d-5fcb1832ce47
rootflags=subvol=/arch/root luks.allow-discards rw
INITRD ../intel-ucode.img,../initramfs-linux.img

Current behavior is I successfully get a syslinux boot menu (at least
after I disabled the 64bit default option), but selecting the above
entry just refreshes the menu and I'm stuck there with an automatic
countdown from 5sec and then back to 5sec when it hits 0. It's like it
just keeps pointing at the /dev/sdb MBR or can't find something and
thus tries again?

Output of blkid:

$ sudo blkid
/dev/sda1: UUID="37441f68-9d76-45bc-b98c-996e68a3555c" TYPE="ext2"
PARTUUID="81053e11-01"
/dev/sda2: UUID="7101e83b-31c0-4cdf-bc07-678e00e19c32"
TYPE="crypto_LUKS" PARTUUID="81053e11-02"
/dev/sdb1: UUID="e046823f-c7bc-4ff0-8d76-307e6353d324" TYPE="ext4"
PARTUUID="45dbad3a-01"
/dev/sdb2: UUID="bcacb6d5-2874-4652-a25d-88b0bf3bbce8"
TYPE="crypto_LUKS" PARTUUID="45dbad3a-02"
/dev/mapper/luks-7101e83b-31c0-4cdf-bc07-678e00e19c32:
UUID="eb20c219-0df8-4051-bad2-39d57aed7b59" TYPE="ext4"
/dev/mapper/btr: LABEL="bigD"
UUID="e1284231-c264-4944-807d-5fcb1832ce47"
UUID_SUB="123f46c6-1b3a-4273-a84c-fa2c257652d0" TYPE="btrfs"

Am I specifying the wrong uuid to use, incorrect subvol, other? I
haven't been able to find anyone specifying a 2nd level subvol for the
root in their kernel parameters and wondered if that's the issue. I
just find "rootflags=subvol=__active" or similar in various google
hits.

Thanks for any suggestions; my apologies is asking about this to the
btrfs list isn't correct. I wasn't sure if this should be directed at
syslinux or here.


Best regards,
John
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: Better csum error message for data csum mismatch

[David Sterba]

On Thu, Feb 09, 2017 at 10:45:06AM +0800, Qu Wenruo wrote:
> The original csum error message only outputs inode number, offset, check
> sum and expected check sum.
> 
> However no root objectid is outputted, which sometimes makes debugging
> quite painful under multi-subvolume case (including relocation).
> 
> Also the checksum output is decimal, which seldom makes sense for
> users/developers and is hard to read in most time.
> 
> This patch will add root objectid, which will be %lld for rootid larger
> than LAST_FREE_OBJECTID, and hex csum output for better readability.
> 
> Signed-off-by: Qu Wenruo 
> ---
> v2:
>   Output mirror number in both inode.c and compression.c

Reviewed-by: David Sterba 

Added to 4.11 queue.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] Btrfs: add another missing end_page_writeback on submit_extent_page failure

[David Sterba]

On Thu, Feb 09, 2017 at 05:24:33PM +0900, Takafumi Kubota wrote:
> If btrfs_bio_alloc fails in submit_extent_page, submit_extent_page returns
> without clearing the writeback bit of the failed page.
> 
> __extent_writepage_io, that is a caller of submit_extent_page,
> does not clear the remaining writeback bit anywhere.
> As a result, this will cause the hang at filemap_fdatawait_range,
> because it waits the writeback bit to be cleared from the failed page.
> So, we have to call end_page_writeback to clear the writeback bit.
> 
> For reproducing the hang, we inject a fault like
> 
>if (should_failtest()) { // I define should_failtest()
> bio = NULL;
>}
>else {
> bio = btrfs_bio_alloc(...);
>}
> 
> in submit_extent_page.
> 
> We should also check whether page has the bit before end_page_writeback,
> to avoid the conflict against the other end_page_writeback in bio_endio.
> Thus, we add PageWriteback checks not only in __extent_writepage_io,
> but also in write_one_eb too, because it misses the check.
> 
> Signed-off-by: Takafumi Kubota 
> Reviewed-by: Liu Bo 
> Cc: David Sterba 

Added to 4.11 queue, thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] Btrfs: use helper to simplify lock/unlock pages

[David Sterba]

On Thu, Feb 02, 2017 at 05:49:23PM -0800, Liu Bo wrote:
> Since we have a helper to set page bits, let lock_delalloc_pages and
> __unlock_for_delalloc use it.
> 
> Signed-off-by: Liu Bo 

Reviewed-by: David Sterba 

I've separated the changes of __process_pages_contig to another patch,
just moving hunks.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] Btrfs: create helper for processing bits on contiguous pages

[David Sterba]

On Thu, Feb 02, 2017 at 05:49:22PM -0800, Liu Bo wrote:
> This introduces a new helper which can be used to process pages bits.
> 
> Signed-off-by: Liu Bo 

Reviewed-by: David Sterba 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: partial quota rescan

[Marat Khalili]

Suddenly discovered undocumented (in man or anywhere) -i parameter of 
'btrfs subvolume snapshot' that adds snapshot to qgroup without 
invalidating statistics. Amazing!


--

With Best Regards,
Marat Khalili

On 08/02/17 18:46, Marat Khalili wrote:
I'm using  trying to use qgroups to keep track of storage occupied by 
snapshots. I noticed that:
a) no two rescans can run in parallel, and there's no way to schedule 
another rescan while one is running;
b) seems like it's a whole-disk operation regardless of path specified 
in CLI.


I only just started to fill my new 24Tb btrfs volume using qgroups, 
but rescans already take a long time, and due to (a) above I each time 
have to wait for previous rescan to finish in my scripts. Can anything 
be done about it, like trashing and recomputing only statistics for 
specific qgroup?


Linux host 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux

btrfs-progs v4.4



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Help understanding autodefrag details

[Peter Zaitsev]

Hi,

As I have been reading btrfs whitepaper  it speaks about autodefrag in very
generic terms - once random write in the file is detected it is put in the
queue to be defragmented.   Yet I could not find any specifics about this
process described anywhere.

My use case is databases and as such large files (100GB+)so my
questions are

- is my understanding what defrag queue is based on files not parts of
files which got fragmented correct ?

- Is single random write is enough to schedule file for defrag or is there
some more elaborate math to consider file fragmented and needing
optimization  ?

- Is this queue FIFO or is it priority queue where files in more need of
fragmentation jump in front (or is there some other mechanics ?

- Will file to be attempted to be defragmented completely or does defrag
focuses on the most fragmented areas of the file first ?

- Is there any way to view this defrag queue ?

- How are resources allocated to background autodefrag vs resources serving
foreground user load are controlled

- What are space requirements for defrag ? is it required for the space to
be available for complete file copy or is it not required ?

- Can defrag handle file which is being constantly written to or is it
based on the concept what file should be idle for some time and when it is
going to be defragmented

Let me know if you have any information on these

-- 
Peter Zaitsev, CEO, Percona
Tel: +1 888 401 3401 ext 7360   Skype:  peter_zaitsev
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html