Re: [BUG] receive not seeing file that exists

[Benedikt Morbach]

Hi all,

On Thu, Jun 2, 2016 at 9:26 AM, Benedikt Morbach
 wrote:
> I've encountered a bug in btrfs-receive. When receiving a certain
> incremental send, it will error with:
>
> ERROR: cannot open
> backup/detritus/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b/user-1000.journal:
> No such file or directory
>
> even though that path exists and the parent subvolume is identical on
> both ends (I checked manually).
>
> I've noticed this happen before on the same directory (and google
> confirms it has also happened to others) and /var/log/journal/ and its
> children are the only directories with 'chattr +C' on this system, so
> it might be related to that?
>
> This was reported on IRC a week or so ago and Josef requested a tree
> --inode of the file/the dirs leading to it and the incremental send,
> so here you go:
>
>
> send side:
> /mnt
> [256]  btrfs_pool_ssd
>
> /mnt/btrfs_pool_ssd
> [256]  backup
>
> /mnt/btrfs_pool_ssd/backup
> [256]  root
>
> /mnt/btrfs_pool_ssd/backup/root
> [256]  root.20160524T1800
> [256]  root.20160524T1900
>
> /mnt/btrfs_pool_ssd/backup/root/root.20160524T1800
> [268]  var
>
> /mnt/btrfs_pool_ssd/backup/root/root.20160524T1800/var
> [   9035]  log
>
> /mnt/btrfs_pool_ssd/backup/root/root.20160524T1800/var/log
> [35122105]  journal
>
> /mnt/btrfs_pool_ssd/backup/root/root.20160524T1800/var/log/journal
> [35122136]  9cbb44cf160f4c1089f77e32ed376a0b
>
> 
> /mnt/btrfs_pool_ssd/backup/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b
> [53198460]  user-1000.journal
>
>
> receive side:
> /backup
> [256]  detritus
>
> /backup/detritus
> [256]  root
>
> /backup/detritus/root
> [256]  root.20160524T1800
>
> /backup/detritus/root/root.20160524T1800
> [267]  var
>
> /backup/detritus/root/root.20160524T1800/var
> [856]  log
>
> /backup/detritus/root/root.20160524T1800/var/log
> [ 316157]  journal
>
> /backup/detritus/root/root.20160524T1800/var/log/journal
> [ 316158]  9cbb44cf160f4c1089f77e32ed376a0b
>
> 
> /backup/detritus/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b
> [ 738979]  user-1000.journal
>
> both trimmed down to only the relevant path.
>
> I don't know how the ML handles attachments, so incremental send
> stream (with --no-data) is here:
> http://dev.exherbo.org/~moben/send-receive_incremental.stream
>
> Let me know if you need anything else or if I misunderstood the tree
> thing. (I _think_ I can also provide the with-data send, but I'd like
> to take a look at that first ;) )


I just re-tested this with kernel 4.6.2 and progs 4.6 and the error is the same.

I also ran a scrub on both the send and receive fs and both came out
with 0 errors.

Another thing I noticed: the +C attr doesn't seem to be preserved by
send/receive:

send # lsattr
/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b/user-1000.journal
C--
/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b/user-1000.journal

receive # lsattr
/backup/detritus/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b/user-1000.journal
---
/backup/detritus/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b/user-1000.journal

Might be related, given that /var/log/journal (and its children) are
the only dirs/files with +C on this system afaik.

Cheers
Benedikt
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] receive not seeing file that exists

[Henk Slager]

On Thu, Jun 2, 2016 at 9:26 AM, Benedikt Morbach
 wrote:
> Hi all,
>
> I've encountered a bug in btrfs-receive. When receiving a certain
> incremental send, it will error with:
>
> ERROR: cannot open
> backup/detritus/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b/user-1000.journal:
> No such file or directory
>
> even though that path exists and the parent subvolume is identical on
> both ends (I checked manually).
>
> I've noticed this happen before on the same directory (and google
> confirms it has also happened to others) and /var/log/journal/ and its
> children are the only directories with 'chattr +C' on this system, so
> it might be related to that?

Now that I see this report, I realize that I also hit this issue. I
was compiling a kernel with 'make -j4 all'. Under some circumstances,
this leads to 'package temp too high' and throtlling speed of CPU;
with -j1 or -j2 I haven't seen it (root cause is the power supply I
think).

Anyhow, while this compile was running, my nightly snapshotting and
incremental send|receive was started. I saw a mce HW error in kernel
log also at that point, so I did restart. Also the inc send had failed
so I thought, it was due to mce issue. But also with no mce-HW issues
logged, tools-4.5.3 + kernel-4.5.4 and also tools-4.5.3 + kernel-4.6.0
had the same issue.

I run send and receive on same PC in this case, but splitting the
stream to a file in addition. The file was already corrupt (too short)
I noticed, so I concluded the issue was in send. I set up a hourly
extra backup crontask for this problem subvol and it failed almost
every hour. For another subvolume on the new 3-day young fs, it was
not a problem. The fs is a few TB, has default mkfs settings +noholes.
Nodesize increased from 4k to 16k, that was a reason to re-create it.

For the problem subvol and also others that I do not inc backup, I set
the subvol to ro on old fs, send the stream-file to temp storage,
received it back on new fs and set it to rw and created initial backup
snapshot of it and send it over to backup fs. That all worked fine.
Several programs write and delete roughly 10 files/hour so not very
active part of the fs. It was quite random at which file the
incremental stream got corrupted.

My best guess was that the use of  btrfs property set  might be the
issue, so I rsynced the data in the subvol into a new subvol and did
initial backup snapshot transfer. This was with tools-4.5.3 +
kernel-4.5.4 and it runs now fine for 10 days.

I had limited time to research this issue for the subvol and also
cannot provide send-stream data for the subvol. But I have still a 12G
btrfs-stream of a .git kernelbuild tree that also got this btrfs
property set ro=true treatment. So I might try to reproduce the bug
with that one.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] receive not seeing file that exists

[Benedikt Morbach]

On Thu, Jun 2, 2016 at 6:35 PM, Chris Murphy  wrote:
> kernel and btrfs-profs versions?

kernel:
send:4.5.5
receive: 4.5.4
btrfs-progs (both): 4.5.3

--
Benedikt
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] receive not seeing file that exists

[Chris Murphy]

On Thu, Jun 2, 2016 at 1:26 AM, Benedikt Morbach
 wrote:
>
> Let me know if you need anything else or if I misunderstood the tree
> thing. (I _think_ I can also provide the with-data send, but I'd like
> to take a look at that first ;) )

kernel and btrfs-profs versions?

-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[BUG] receive not seeing file that exists

[Benedikt Morbach]

Hi all,

I've encountered a bug in btrfs-receive. When receiving a certain
incremental send, it will error with:

ERROR: cannot open
backup/detritus/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b/user-1000.journal:
No such file or directory

even though that path exists and the parent subvolume is identical on
both ends (I checked manually).

I've noticed this happen before on the same directory (and google
confirms it has also happened to others) and /var/log/journal/ and its
children are the only directories with 'chattr +C' on this system, so
it might be related to that?

This was reported on IRC a week or so ago and Josef requested a tree
--inode of the file/the dirs leading to it and the incremental send,
so here you go:


send side:
/mnt
[256]  btrfs_pool_ssd

/mnt/btrfs_pool_ssd
[256]  backup

/mnt/btrfs_pool_ssd/backup
[256]  root

/mnt/btrfs_pool_ssd/backup/root
[256]  root.20160524T1800
[256]  root.20160524T1900

/mnt/btrfs_pool_ssd/backup/root/root.20160524T1800
[268]  var

/mnt/btrfs_pool_ssd/backup/root/root.20160524T1800/var
[   9035]  log

/mnt/btrfs_pool_ssd/backup/root/root.20160524T1800/var/log
[35122105]  journal

/mnt/btrfs_pool_ssd/backup/root/root.20160524T1800/var/log/journal
[35122136]  9cbb44cf160f4c1089f77e32ed376a0b


/mnt/btrfs_pool_ssd/backup/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b
[53198460]  user-1000.journal


receive side:
/backup
[256]  detritus

/backup/detritus
[256]  root

/backup/detritus/root
[256]  root.20160524T1800

/backup/detritus/root/root.20160524T1800
[267]  var

/backup/detritus/root/root.20160524T1800/var
[856]  log

/backup/detritus/root/root.20160524T1800/var/log
[ 316157]  journal

/backup/detritus/root/root.20160524T1800/var/log/journal
[ 316158]  9cbb44cf160f4c1089f77e32ed376a0b


/backup/detritus/root/root.20160524T1800/var/log/journal/9cbb44cf160f4c1089f77e32ed376a0b
[ 738979]  user-1000.journal

both trimmed down to only the relevant path.

I don't know how the ML handles attachments, so incremental send
stream (with --no-data) is here:
http://dev.exherbo.org/~moben/send-receive_incremental.stream

Let me know if you need anything else or if I misunderstood the tree
thing. (I _think_ I can also provide the with-data send, but I'd like
to take a look at that first ;) )


Cheers
Benedikt
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html