Re: send | receive: received snapshot is missing recent files

[Dave]

On Mon, Sep 11, 2017 at 11:19 PM, Andrei Borzenkov  wrote:
> 11.09.2017 20:53, Axel Burri пишет:
>> On 2017-09-08 06:44, Dave wrote:
>>> I'm referring to the link below. Using "btrfs subvolume snapshot -r"
>>> copies the Received UUID from the source into the new snapshot. The
>>> btrbk FAQ entry suggests otherwise. Has something changed?
>>
>> I don't think something has changed, the description for the read-only
>> subvolumes on the btrbk FAQ was just wrong (fixed now).
>>
>>> The only way I see to remove a Received UUID is to create a rw
>>> snapshot (above command without the "-r"), which is not ideal in this
>>> situation when cleaning up readonly source snapshots.
>>>
>>> Any suggestions? Thanks
>>
>> No suggestions from my part, as far as I know there is no way to easily
>> remove/change a received_uuid from a subvolume.
>>
>
> There is BTRFS_IOC_SET_RECEIVED_SUBVOL IOCTL which is used by "btrfs
> received". My understanding is that it can also be set to empty (this
> clearing it). You could write small program to do it.
>
> In general it sounds like a bug - removing read-only flag from subvolume
> by any means should also clear Received UUID as we cannot anymore
> guarantee that subvolume content is the same.

Yes! That makes a great deal of sense.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: send | receive: received snapshot is missing recent files

[Andrei Borzenkov]

11.09.2017 20:53, Axel Burri пишет:
> On 2017-09-08 06:44, Dave wrote:
>> I'm referring to the link below. Using "btrfs subvolume snapshot -r"
>> copies the Received UUID from the source into the new snapshot. The
>> btrbk FAQ entry suggests otherwise. Has something changed?
> 
> I don't think something has changed, the description for the read-only
> subvolumes on the btrbk FAQ was just wrong (fixed now).
> 
>> The only way I see to remove a Received UUID is to create a rw
>> snapshot (above command without the "-r"), which is not ideal in this
>> situation when cleaning up readonly source snapshots.
>>
>> Any suggestions? Thanks
> 
> No suggestions from my part, as far as I know there is no way to easily
> remove/change a received_uuid from a subvolume.
> 

There is BTRFS_IOC_SET_RECEIVED_SUBVOL IOCTL which is used by "btrfs
received". My understanding is that it can also be set to empty (this
clearing it). You could write small program to do it.

In general it sounds like a bug - removing read-only flag from subvolume
by any means should also clear Received UUID as we cannot anymore
guarantee that subvolume content is the same.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: send | receive: received snapshot is missing recent files

[Axel Burri]

On 2017-09-08 06:44, Dave wrote:
> I'm referring to the link below. Using "btrfs subvolume snapshot -r"
> copies the Received UUID from the source into the new snapshot. The
> btrbk FAQ entry suggests otherwise. Has something changed?

I don't think something has changed, the description for the read-only
subvolumes on the btrbk FAQ was just wrong (fixed now).

> The only way I see to remove a Received UUID is to create a rw
> snapshot (above command without the "-r"), which is not ideal in this
> situation when cleaning up readonly source snapshots.
> 
> Any suggestions? Thanks

No suggestions from my part, as far as I know there is no way to easily
remove/change a received_uuid from a subvolume.

As you mentioned, you can snapshot it twice:

# btrfs subvolume snapshot mysubvol mysubvol.rw
# btrfs subvolume delete mysubvol
# btrfs subvolume snapshot -r mysubvol.rw mysubvol
# btrfs subvolume delete mysubvol.rw

Instead of the second snapshot operation, this time you could also use
the (evil) command: "btrfs btrfs property set -ts mysnapshot ro true"

> On Thu, Sep 7, 2017 at 10:33 AM, Axel Burri  wrote:
>>
>> Having a received_uuid set on the source volume ("/home" in your case)
>> is indeed a bad thing when it comes to send/receive. You probably
>> restored a backup with send/receive, and made it read/write using "btrfs
>> property set -ts /home ro false". This is a an evil thing, as it leaves
>> received_uuid intact. In order to make a subvolume read-write, I
>> recommend to use "btrfs subvolume snapshot  ".
>>
>> There is a FAQ entry on btrbk on how to fix this:
>>
>> https://github.com/digint/btrbk/blob/master/doc/FAQ.md#im-getting-an-error-aborted-received-uuid-is-set
>>
>>
>> On 2017-09-07 15:34, Dave wrote:
>>> I just ran a test. The btrfs send - receive problem I described is
>>> indeed fully resolved by removing the "problematic" snapshot on the
>>> target device. I did not make any changes to the source volume. I did
>>> not make any other changes in my steps (see earlier message for my
>>> exact steps).
>>>
>>> Therefore, the problem I described in my earlier message is not due
>>> exclusively to having a Received UUID on the source volume (or to any
>>> other feature of the source volume). It is not related to any feature
>>> of the directly specified parent volume either. More details are
>>> included in my earlier email.
>>>
>>> Thanks for any further feedback, including answers to my questions and
>>> comments about whether this is a known issue.
>>>
>>>
>>> On Thu, Sep 7, 2017 at 8:39 AM, Dave  wrote:

 Hello. Can anyone further explain this issue ("you have a Received UUID on 
 the source volume")?

 How does it happen?
 How does one remove a Received UUID from the source volume?

 And how does that explain my results where I showed that the problem
 is not dependent upon the source volume but is instead dependent upon
 some existing snapshot on the target volume?

 My results do not appear to be fully explained by a Received UUID on the 
 source volume, as my prior message hopefully shows clearly.

 Thank you.

 On Thu, Sep 7, 2017 at 2:24 AM, A L  wrote:
> The problem can be that you have a Received UUID on the source volume. 
> This breaks send-receive.
>
>  From: Dave  -- Sent: 2017-09-07 - 06:43 
>
>> Here is more info and a possible (shocking) explanation. This
>> aggregates my prior messages and it provides an almost complete set of
>> steps to reproduce this problem.
>>
>> Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 
>> GNU/Linux
>> btrfs-progs v4.12
>>
>> My steps:
>>
>> [root@srv]# sync
>> [root@srv]# mkdir /home/.snapshots/test1
>> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/
>> Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home'
>> [root@srv]# sync
>> [root@srv]# mkdir /mnt/x5a/home/test1
>> [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive
>> /mnt/x5a/home/test1/
>> At subvol /home/.snapshots/test1/home/
>> At subvol home
>> [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/
>> NOTE: all recent files are present
>> [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/
>> NOTE: all recent files are present
>> [root@srv]# mkdir /home/.snapshots/test2
>> [root@srv]# mkdir /mnt/x5a/home/test2
>> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/
>> Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home'
>> [root@srv]# sync
>> [root@srv]# btrfs send -p /home/.snapshots/test1/home/
>> /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/
>> At subvol /home/.snapshots/test2/home/
>> At snapshot home
>> [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/

Re: send | receive: received snapshot is missing recent files

[Dave]

I'm referring to the link below. Using "btrfs subvolume snapshot -r"
copies the Received UUID from the source into the new snapshot. The
btrbk FAQ entry suggests otherwise. Has something changed?

The only way I see to remove a Received UUID is to create a rw
snapshot (above command without the "-r"), which is not ideal in this
situation when cleaning up readonly source snapshots.

Any suggestions? Thanks

On Thu, Sep 7, 2017 at 10:33 AM, Axel Burri  wrote:
>
> Having a received_uuid set on the source volume ("/home" in your case)
> is indeed a bad thing when it comes to send/receive. You probably
> restored a backup with send/receive, and made it read/write using "btrfs
> property set -ts /home ro false". This is a an evil thing, as it leaves
> received_uuid intact. In order to make a subvolume read-write, I
> recommend to use "btrfs subvolume snapshot  ".
>
> There is a FAQ entry on btrbk on how to fix this:
>
> https://github.com/digint/btrbk/blob/master/doc/FAQ.md#im-getting-an-error-aborted-received-uuid-is-set
>
>
> On 2017-09-07 15:34, Dave wrote:
> > I just ran a test. The btrfs send - receive problem I described is
> > indeed fully resolved by removing the "problematic" snapshot on the
> > target device. I did not make any changes to the source volume. I did
> > not make any other changes in my steps (see earlier message for my
> > exact steps).
> >
> > Therefore, the problem I described in my earlier message is not due
> > exclusively to having a Received UUID on the source volume (or to any
> > other feature of the source volume). It is not related to any feature
> > of the directly specified parent volume either. More details are
> > included in my earlier email.
> >
> > Thanks for any further feedback, including answers to my questions and
> > comments about whether this is a known issue.
> >
> >
> > On Thu, Sep 7, 2017 at 8:39 AM, Dave  wrote:
> >>
> >> Hello. Can anyone further explain this issue ("you have a Received UUID on 
> >> the source volume")?
> >>
> >> How does it happen?
> >> How does one remove a Received UUID from the source volume?
> >>
> >> And how does that explain my results where I showed that the problem
> >> is not dependent upon the source volume but is instead dependent upon
> >> some existing snapshot on the target volume?
> >>
> >> My results do not appear to be fully explained by a Received UUID on the 
> >> source volume, as my prior message hopefully shows clearly.
> >>
> >> Thank you.
> >>
> >> On Thu, Sep 7, 2017 at 2:24 AM, A L  wrote:
> >>> The problem can be that you have a Received UUID on the source volume. 
> >>> This breaks send-receive.
> >>>
> >>>  From: Dave  -- Sent: 2017-09-07 - 06:43 
> >>>
>  Here is more info and a possible (shocking) explanation. This
>  aggregates my prior messages and it provides an almost complete set of
>  steps to reproduce this problem.
> 
>  Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 
>  GNU/Linux
>  btrfs-progs v4.12
> 
>  My steps:
> 
>  [root@srv]# sync
>  [root@srv]# mkdir /home/.snapshots/test1
>  [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/
>  Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home'
>  [root@srv]# sync
>  [root@srv]# mkdir /mnt/x5a/home/test1
>  [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive
>  /mnt/x5a/home/test1/
>  At subvol /home/.snapshots/test1/home/
>  At subvol home
>  [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/
>  NOTE: all recent files are present
>  [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/
>  NOTE: all recent files are present
>  [root@srv]# mkdir /home/.snapshots/test2
>  [root@srv]# mkdir /mnt/x5a/home/test2
>  [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/
>  Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home'
>  [root@srv]# sync
>  [root@srv]# btrfs send -p /home/.snapshots/test1/home/
>  /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/
>  At subvol /home/.snapshots/test2/home/
>  At snapshot home
>  [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/
>  NOTE: all recent files are MISSING
>  [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/
>  NOTE: all recent files are MISSING
> 
>  Below I am including some rsync output to illustrate when a snapshot
>  is missing files (or not):
> 
>  [root@srv]# rsync -aniv /home/.snapshots/test1/home/
>  /home/.snapshots/test2/home/
>  sending incremental file list
> 
>  sent 1,143,286 bytes  received 1,123 bytes  762,939.33 bytes/sec
>  total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)
> 
>  This indicates that these two subvolumes contain the same files, which
>  they should because test2 

Re: send | receive: received snapshot is missing recent files

[Axel Burri]

Having a received_uuid set on the source volume ("/home" in your case)
is indeed a bad thing when it comes to send/receive. You probably
restored a backup with send/receive, and made it read/write using "btrfs
property set -ts /home ro false". This is a an evil thing, as it leaves
received_uuid intact. In order to make a subvolume read-write, I
recommend to use "btrfs subvolume snapshot  ".

There is a FAQ entry on btrbk on how to fix this:

https://github.com/digint/btrbk/blob/master/doc/FAQ.md#im-getting-an-error-aborted-received-uuid-is-set


On 2017-09-07 15:34, Dave wrote:
> I just ran a test. The btrfs send - receive problem I described is
> indeed fully resolved by removing the "problematic" snapshot on the
> target device. I did not make any changes to the source volume. I did
> not make any other changes in my steps (see earlier message for my
> exact steps).
> 
> Therefore, the problem I described in my earlier message is not due
> exclusively to having a Received UUID on the source volume (or to any
> other feature of the source volume). It is not related to any feature
> of the directly specified parent volume either. More details are
> included in my earlier email.
> 
> Thanks for any further feedback, including answers to my questions and
> comments about whether this is a known issue.
> 
> 
> On Thu, Sep 7, 2017 at 8:39 AM, Dave  wrote:
>>
>> Hello. Can anyone further explain this issue ("you have a Received UUID on 
>> the source volume")?
>>
>> How does it happen?
>> How does one remove a Received UUID from the source volume?
>>
>> And how does that explain my results where I showed that the problem
>> is not dependent upon the source volume but is instead dependent upon
>> some existing snapshot on the target volume?
>>
>> My results do not appear to be fully explained by a Received UUID on the 
>> source volume, as my prior message hopefully shows clearly.
>>
>> Thank you.
>>
>> On Thu, Sep 7, 2017 at 2:24 AM, A L  wrote:
>>> The problem can be that you have a Received UUID on the source volume. This 
>>> breaks send-receive.
>>>
>>>  From: Dave  -- Sent: 2017-09-07 - 06:43 
>>>
 Here is more info and a possible (shocking) explanation. This
 aggregates my prior messages and it provides an almost complete set of
 steps to reproduce this problem.

 Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux
 btrfs-progs v4.12

 My steps:

 [root@srv]# sync
 [root@srv]# mkdir /home/.snapshots/test1
 [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/
 Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home'
 [root@srv]# sync
 [root@srv]# mkdir /mnt/x5a/home/test1
 [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive
 /mnt/x5a/home/test1/
 At subvol /home/.snapshots/test1/home/
 At subvol home
 [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/
 NOTE: all recent files are present
 [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/
 NOTE: all recent files are present
 [root@srv]# mkdir /home/.snapshots/test2
 [root@srv]# mkdir /mnt/x5a/home/test2
 [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/
 Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home'
 [root@srv]# sync
 [root@srv]# btrfs send -p /home/.snapshots/test1/home/
 /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/
 At subvol /home/.snapshots/test2/home/
 At snapshot home
 [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/
 NOTE: all recent files are MISSING
 [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/
 NOTE: all recent files are MISSING

 Below I am including some rsync output to illustrate when a snapshot
 is missing files (or not):

 [root@srv]# rsync -aniv /home/.snapshots/test1/home/
 /home/.snapshots/test2/home/
 sending incremental file list

 sent 1,143,286 bytes  received 1,123 bytes  762,939.33 bytes/sec
 total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)

 This indicates that these two subvolumes contain the same files, which
 they should because test2 is a snapshot of test1 without any changes
 to files, and it was not sent to another physical device.

 The problem is when test2 is sent to another device as shown by the
 rsync results below.

 [root@srv]# rsync -aniv /home/.snapshots/test2/home/ 
 /mnt/x5a/home/test2/home/
 sending incremental file list
 .d..t.. ./
 .d..t.. user1/
> f.st.. user1/.bash_history
> f.st.. user1/.bashrc
> f+ user1/test2017-09-06.txt
 ...
 and a long list of other missing files

 The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is
 missing all recent files (any files from the month of 

Re: send | receive: received snapshot is missing recent files

[Dave]

I just ran a test. The btrfs send - receive problem I described is
indeed fully resolved by removing the "problematic" snapshot on the
target device. I did not make any changes to the source volume. I did
not make any other changes in my steps (see earlier message for my
exact steps).

Therefore, the problem I described in my earlier message is not due
exclusively to having a Received UUID on the source volume (or to any
other feature of the source volume). It is not related to any feature
of the directly specified parent volume either. More details are
included in my earlier email.

Thanks for any further feedback, including answers to my questions and
comments about whether this is a known issue.


On Thu, Sep 7, 2017 at 8:39 AM, Dave  wrote:
>
> Hello. Can anyone further explain this issue ("you have a Received UUID on 
> the source volume")?
>
> How does it happen?
> How does one remove a Received UUID from the source volume?
>
> And how does that explain my results where I showed that the problem
> is not dependent upon the source volume but is instead dependent upon
> some existing snapshot on the target volume?
>
> My results do not appear to be fully explained by a Received UUID on the 
> source volume, as my prior message hopefully shows clearly.
>
> Thank you.
>
> On Thu, Sep 7, 2017 at 2:24 AM, A L  wrote:
> > The problem can be that you have a Received UUID on the source volume. This 
> > breaks send-receive.
> >
> >  From: Dave  -- Sent: 2017-09-07 - 06:43 
> >
> >> Here is more info and a possible (shocking) explanation. This
> >> aggregates my prior messages and it provides an almost complete set of
> >> steps to reproduce this problem.
> >>
> >> Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux
> >> btrfs-progs v4.12
> >>
> >> My steps:
> >>
> >> [root@srv]# sync
> >> [root@srv]# mkdir /home/.snapshots/test1
> >> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/
> >> Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home'
> >> [root@srv]# sync
> >> [root@srv]# mkdir /mnt/x5a/home/test1
> >> [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive
> >> /mnt/x5a/home/test1/
> >> At subvol /home/.snapshots/test1/home/
> >> At subvol home
> >> [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/
> >> NOTE: all recent files are present
> >> [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/
> >> NOTE: all recent files are present
> >> [root@srv]# mkdir /home/.snapshots/test2
> >> [root@srv]# mkdir /mnt/x5a/home/test2
> >> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/
> >> Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home'
> >> [root@srv]# sync
> >> [root@srv]# btrfs send -p /home/.snapshots/test1/home/
> >> /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/
> >> At subvol /home/.snapshots/test2/home/
> >> At snapshot home
> >> [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/
> >> NOTE: all recent files are MISSING
> >> [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/
> >> NOTE: all recent files are MISSING
> >>
> >> Below I am including some rsync output to illustrate when a snapshot
> >> is missing files (or not):
> >>
> >> [root@srv]# rsync -aniv /home/.snapshots/test1/home/
> >> /home/.snapshots/test2/home/
> >> sending incremental file list
> >>
> >> sent 1,143,286 bytes  received 1,123 bytes  762,939.33 bytes/sec
> >> total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)
> >>
> >> This indicates that these two subvolumes contain the same files, which
> >> they should because test2 is a snapshot of test1 without any changes
> >> to files, and it was not sent to another physical device.
> >>
> >> The problem is when test2 is sent to another device as shown by the
> >> rsync results below.
> >>
> >> [root@srv]# rsync -aniv /home/.snapshots/test2/home/ 
> >> /mnt/x5a/home/test2/home/
> >> sending incremental file list
> >> .d..t.. ./
> >> .d..t.. user1/
> >>>f.st.. user1/.bash_history
> >>>f.st.. user1/.bashrc
> >>>f+ user1/test2017-09-06.txt
> >> ...
> >> and a long list of other missing files
> >>
> >> The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is
> >> missing all recent files (any files from the month of August or
> >> September), as my prior visual inspections had indicated. The same
> >> files are missing every time. There is no randomness to the missing
> >> data.
> >>
> >> The problem does not happen for me if the receive command target is
> >> located on the same physical device as shown next. (However, I suspect
> >> there's more to it than that, as explained further below.)
> >>
> >> [root@srv]# mkdir /home/.snapshots/test2rec
> >> [root@srv]# btrfs send -p /home/.snapshots/test1/home/
> >> /home/.snapshots/test2/home/ | btrfs receive
> >> /home/.snapshots/test2rec/
> >> At subvol /home/.snapshots/test2/home/
> >>
> >> # 

Re: send | receive: received snapshot is missing recent files

[Dave]

Hello. Can anyone further explain this issue ("you have a Received
UUID on the source volume")?

How does it happen?
How does one remove a Received UUID from the source volume?

And how does that explain my results where I showed that the problem
is not dependent upon the source volume but is instead dependent upon
some existing snapshot on the target volume?

My results do not appear to be fully explained by a Received UUID on
the source volume, as my prior message hopefully shows clearly.

Thank you.

On Thu, Sep 7, 2017 at 2:24 AM, A L  wrote:
> The problem can be that you have a Received UUID on the source volume. This 
> breaks send-receive.
>
>  From: Dave  -- Sent: 2017-09-07 - 06:43 
>
>> Here is more info and a possible (shocking) explanation. This
>> aggregates my prior messages and it provides an almost complete set of
>> steps to reproduce this problem.
>>
>> Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux
>> btrfs-progs v4.12
>>
>> My steps:
>>
>> [root@srv]# sync
>> [root@srv]# mkdir /home/.snapshots/test1
>> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/
>> Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home'
>> [root@srv]# sync
>> [root@srv]# mkdir /mnt/x5a/home/test1
>> [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive
>> /mnt/x5a/home/test1/
>> At subvol /home/.snapshots/test1/home/
>> At subvol home
>> [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/
>> NOTE: all recent files are present
>> [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/
>> NOTE: all recent files are present
>> [root@srv]# mkdir /home/.snapshots/test2
>> [root@srv]# mkdir /mnt/x5a/home/test2
>> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/
>> Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home'
>> [root@srv]# sync
>> [root@srv]# btrfs send -p /home/.snapshots/test1/home/
>> /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/
>> At subvol /home/.snapshots/test2/home/
>> At snapshot home
>> [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/
>> NOTE: all recent files are MISSING
>> [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/
>> NOTE: all recent files are MISSING
>>
>> Below I am including some rsync output to illustrate when a snapshot
>> is missing files (or not):
>>
>> [root@srv]# rsync -aniv /home/.snapshots/test1/home/
>> /home/.snapshots/test2/home/
>> sending incremental file list
>>
>> sent 1,143,286 bytes  received 1,123 bytes  762,939.33 bytes/sec
>> total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)
>>
>> This indicates that these two subvolumes contain the same files, which
>> they should because test2 is a snapshot of test1 without any changes
>> to files, and it was not sent to another physical device.
>>
>> The problem is when test2 is sent to another device as shown by the
>> rsync results below.
>>
>> [root@srv]# rsync -aniv /home/.snapshots/test2/home/ 
>> /mnt/x5a/home/test2/home/
>> sending incremental file list
>> .d..t.. ./
>> .d..t.. user1/
>>>f.st.. user1/.bash_history
>>>f.st.. user1/.bashrc
>>>f+ user1/test2017-09-06.txt
>> ...
>> and a long list of other missing files
>>
>> The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is
>> missing all recent files (any files from the month of August or
>> September), as my prior visual inspections had indicated. The same
>> files are missing every time. There is no randomness to the missing
>> data.
>>
>> The problem does not happen for me if the receive command target is
>> located on the same physical device as shown next. (However, I suspect
>> there's more to it than that, as explained further below.)
>>
>> [root@srv]# mkdir /home/.snapshots/test2rec
>> [root@srv]# btrfs send -p /home/.snapshots/test1/home/
>> /home/.snapshots/test2/home/ | btrfs receive
>> /home/.snapshots/test2rec/
>> At subvol /home/.snapshots/test2/home/
>>
>> # rsync -aniv /home/.snapshots/test2/home/ /home/.snapshots/test2rec/home/
>> sending incremental file list
>>
>> sent 1,143,286 bytes  received 1,123 bytes  2,288,818.00 bytes/sec
>> total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)
>>
>> The above (as well as visual inspection of files) indicates that these
>> two subvolumes contain the same files, which was not the case when the
>> same command had a target located on another physical device. Of
>> course, a snapshot which resides on the same physical device is not a
>> very good backup. So I do need to send it to another device, but that
>> results in missing files when the -p or -c options are used with btrfs
>> send. (Non-incremental sending to another physical device does work.)
>>
>> I can think of a couple possible explanations.
>>
>> One is that there is a problem when using the -p or -c options with
>> btrfs send when the target is another physical device. I suspect this
>> is the actual explanation, 

Re: send | receive: received snapshot is missing recent files

[A L]

The problem can be that you have a Received UUID on the source volume. This 
breaks send-receive.

 From: Dave  -- Sent: 2017-09-07 - 06:43 

> Here is more info and a possible (shocking) explanation. This
> aggregates my prior messages and it provides an almost complete set of
> steps to reproduce this problem.
> 
> Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux
> btrfs-progs v4.12
> 
> My steps:
> 
> [root@srv]# sync
> [root@srv]# mkdir /home/.snapshots/test1
> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/
> Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home'
> [root@srv]# sync
> [root@srv]# mkdir /mnt/x5a/home/test1
> [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive
> /mnt/x5a/home/test1/
> At subvol /home/.snapshots/test1/home/
> At subvol home
> [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/
> NOTE: all recent files are present
> [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/
> NOTE: all recent files are present
> [root@srv]# mkdir /home/.snapshots/test2
> [root@srv]# mkdir /mnt/x5a/home/test2
> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/
> Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home'
> [root@srv]# sync
> [root@srv]# btrfs send -p /home/.snapshots/test1/home/
> /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/
> At subvol /home/.snapshots/test2/home/
> At snapshot home
> [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/
> NOTE: all recent files are MISSING
> [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/
> NOTE: all recent files are MISSING
> 
> Below I am including some rsync output to illustrate when a snapshot
> is missing files (or not):
> 
> [root@srv]# rsync -aniv /home/.snapshots/test1/home/
> /home/.snapshots/test2/home/
> sending incremental file list
> 
> sent 1,143,286 bytes  received 1,123 bytes  762,939.33 bytes/sec
> total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)
> 
> This indicates that these two subvolumes contain the same files, which
> they should because test2 is a snapshot of test1 without any changes
> to files, and it was not sent to another physical device.
> 
> The problem is when test2 is sent to another device as shown by the
> rsync results below.
> 
> [root@srv]# rsync -aniv /home/.snapshots/test2/home/ /mnt/x5a/home/test2/home/
> sending incremental file list
> .d..t.. ./
> .d..t.. user1/
>>f.st.. user1/.bash_history
>>f.st.. user1/.bashrc
>>f+ user1/test2017-09-06.txt
> ...
> and a long list of other missing files
> 
> The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is
> missing all recent files (any files from the month of August or
> September), as my prior visual inspections had indicated. The same
> files are missing every time. There is no randomness to the missing
> data.
> 
> The problem does not happen for me if the receive command target is
> located on the same physical device as shown next. (However, I suspect
> there's more to it than that, as explained further below.)
> 
> [root@srv]# mkdir /home/.snapshots/test2rec
> [root@srv]# btrfs send -p /home/.snapshots/test1/home/
> /home/.snapshots/test2/home/ | btrfs receive
> /home/.snapshots/test2rec/
> At subvol /home/.snapshots/test2/home/
> 
> # rsync -aniv /home/.snapshots/test2/home/ /home/.snapshots/test2rec/home/
> sending incremental file list
> 
> sent 1,143,286 bytes  received 1,123 bytes  2,288,818.00 bytes/sec
> total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)
> 
> The above (as well as visual inspection of files) indicates that these
> two subvolumes contain the same files, which was not the case when the
> same command had a target located on another physical device. Of
> course, a snapshot which resides on the same physical device is not a
> very good backup. So I do need to send it to another device, but that
> results in missing files when the -p or -c options are used with btrfs
> send. (Non-incremental sending to another physical device does work.)
> 
> I can think of a couple possible explanations.
> 
> One is that there is a problem when using the -p or -c options with
> btrfs send when the target is another physical device. I suspect this
> is the actual explanation, however.
> 
> A second possibility is that the presence of prior existing snapshots
> at the target location (even if old and not referenced in any current
> btrfs command), can determine the outcome and final contents of an
> incremental send operation. I believe the info below suggests this to
> be the case.
> 
> [root@srv]# btrfs su show /home/.snapshots/test2/home/
> test2/home
> Name:   home
> UUID:   292e8bbf-a95f-2a4e-8280-129202d389dc
> Parent UUID:62418df6-a1f8-d74a-a152-11f519593053
> Received UUID:  e00d5318-6efd-824e-ac91-f25efa5c2a74
> Creation time:  

Re: send | receive: received snapshot is missing recent files

[Dave]

Here is more info and a possible (shocking) explanation. This
aggregates my prior messages and it provides an almost complete set of
steps to reproduce this problem.

Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux
btrfs-progs v4.12

My steps:

[root@srv]# sync
[root@srv]# mkdir /home/.snapshots/test1
[root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/
Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home'
[root@srv]# sync
[root@srv]# mkdir /mnt/x5a/home/test1
[root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive
/mnt/x5a/home/test1/
At subvol /home/.snapshots/test1/home/
At subvol home
[root@srv]# ls -la /mnt/x5a/home/test1/home/user1/
NOTE: all recent files are present
[root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/
NOTE: all recent files are present
[root@srv]# mkdir /home/.snapshots/test2
[root@srv]# mkdir /mnt/x5a/home/test2
[root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/
Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home'
[root@srv]# sync
[root@srv]# btrfs send -p /home/.snapshots/test1/home/
/home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/
At subvol /home/.snapshots/test2/home/
At snapshot home
[root@srv]# ls -la /mnt/x5a/home/test2/home/user1/
NOTE: all recent files are MISSING
[root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/
NOTE: all recent files are MISSING

Below I am including some rsync output to illustrate when a snapshot
is missing files (or not):

[root@srv]# rsync -aniv /home/.snapshots/test1/home/
/home/.snapshots/test2/home/
sending incremental file list

sent 1,143,286 bytes  received 1,123 bytes  762,939.33 bytes/sec
total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)

This indicates that these two subvolumes contain the same files, which
they should because test2 is a snapshot of test1 without any changes
to files, and it was not sent to another physical device.

The problem is when test2 is sent to another device as shown by the
rsync results below.

[root@srv]# rsync -aniv /home/.snapshots/test2/home/ /mnt/x5a/home/test2/home/
sending incremental file list
.d..t.. ./
.d..t.. user1/
>f.st.. user1/.bash_history
>f.st.. user1/.bashrc
>f+ user1/test2017-09-06.txt
...
and a long list of other missing files

The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is
missing all recent files (any files from the month of August or
September), as my prior visual inspections had indicated. The same
files are missing every time. There is no randomness to the missing
data.

The problem does not happen for me if the receive command target is
located on the same physical device as shown next. (However, I suspect
there's more to it than that, as explained further below.)

[root@srv]# mkdir /home/.snapshots/test2rec
[root@srv]# btrfs send -p /home/.snapshots/test1/home/
/home/.snapshots/test2/home/ | btrfs receive
/home/.snapshots/test2rec/
At subvol /home/.snapshots/test2/home/

# rsync -aniv /home/.snapshots/test2/home/ /home/.snapshots/test2rec/home/
sending incremental file list

sent 1,143,286 bytes  received 1,123 bytes  2,288,818.00 bytes/sec
total size is 3,642,972,271  speedup is 3,183.28 (DRY RUN)

The above (as well as visual inspection of files) indicates that these
two subvolumes contain the same files, which was not the case when the
same command had a target located on another physical device. Of
course, a snapshot which resides on the same physical device is not a
very good backup. So I do need to send it to another device, but that
results in missing files when the -p or -c options are used with btrfs
send. (Non-incremental sending to another physical device does work.)

I can think of a couple possible explanations.

One is that there is a problem when using the -p or -c options with
btrfs send when the target is another physical device. I suspect this
is the actual explanation, however.

A second possibility is that the presence of prior existing snapshots
at the target location (even if old and not referenced in any current
btrfs command), can determine the outcome and final contents of an
incremental send operation. I believe the info below suggests this to
be the case.

[root@srv]# btrfs su show /home/.snapshots/test2/home/
test2/home
Name:   home
UUID:   292e8bbf-a95f-2a4e-8280-129202d389dc
Parent UUID:62418df6-a1f8-d74a-a152-11f519593053
Received UUID:  e00d5318-6efd-824e-ac91-f25efa5c2a74
Creation time:  2017-09-06 15:38:16 -0400
Subvolume ID:   2000
Generation: 5020
Gen at creation:5020
Parent ID:  257
Top level ID:   257
Flags:  readonly
Snapshot(s):

[root@srv]# btrfs su show /mnt/x5a/home/test1/home
home/test1/home
Name:   home
UUID: 

Re: send | receive: received snapshot is missing recent files

[Dave]

This is an even better set of steps for reproducing the problem.

[root@srv]# sync
[root@srv]# mkdir /home/.snapshots/test1
[root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/
Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home'
[root@srv]# sync
[root@srv]# mkdir /mnt/x5a/home/test1
[root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive
/mnt/x5a/home/test1/
At subvol /home/.snapshots/test1/home/
At subvol home
[root@srv]# ls -la /mnt/x5a/home/test1/home/user1/
NOTE: all recent files are present
[root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/
NOTE: all recent files are present
[root@srv]# mkdir /home/.snapshots/test2
[root@srv]# mkdir /mnt/x5a/home/test2
[root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/
Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home'
[root@srv]# sync
[root@srv]# btrfs send -p /home/.snapshots/test1/home/
/home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/
At subvol /home/.snapshots/test2/home/
At snapshot home
[root@srv]# ls -la /mnt/x5a/home/test2/home/user1/
NOTE: all recent files are MISSING
[root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/
NOTE: all recent files are MISSING

Any ideas what could be causing this problem with incremental backups?


On Wed, Sep 6, 2017 at 3:23 PM, Dave  wrote:
>
> Here is more info on this problem. I can reproduce this without using my 
> script. Simple btrfs commands will reproduce the problem every time. The same 
> files are missing every time. There is no randomness to the missing data.
>
> Here are my steps:
>
> 1. snapper -c home create
> result is a valid snapshot at /home/.snapshots/1704/snapshot
> 2. btrfs send /home/.snapshots/1704/snapshot | btrfs receive 
> /mnt/x5a/home/1704
> 3. snapper -c home create
> result is a valid snapshot at /home/.snapshots/1716/snapshot
> 4. btrfs send -c /home/.snapshots/1704/snapshot/ 
> /home/.snapshots/1716/snapshot/ | btrfs receive /mnt/x5a/home/1716/
>
> I expect /mnt/x5a/home/1716/snapshot to be identical to 
> /home/.snapshots/1716/snapshot. However, it is not.
> The result is that /mnt/x5a/home/1716/snapshot is missing all recent files.
>
> Next step was to delete snapshot 1716 (in both locations) and repeat the send 
> | receive using -p
>
> btrfs su del /mnt/x5a/home/1716/snapshot
> snapper -c home delete 1716
> snapper -c home create
> btrfs send -p /home/.snapshots/1704/snapshot/ /home/.snapshots/1716/snapshot/ 
> | btrfs receive /mnt/x5a/home/1716/
>
> The result is once again that /mnt/x5a/home/1716/snapshot is missing all 
> recent files. However, the other snapshots are all valid:
> /home/.snapshots/1704/snapshot is valid & complete
> /mnt/x5a/home/1704/snapshot -- non-incremental send: snapshot is valid & 
> complete
> /home/.snapshots/1716/snapshot is valid & complete
> /mnt/x5a/home/1716/snapshot -- incrementally sent snapshot is missing all 
> recent files whether sent with -c or -p
>
> The incrementally sent snapshot is even missing files that are present in the 
> reference snapshot /mnt/x5a/home/1704/snapshot.
>
>
>
> On Wed, Sep 6, 2017 at 1:37 AM, Dave  wrote:
>>
>> I'm running Arch Linux on BTRFS. I use Snapper to take hourly
>> snapshots and it works without any issues.
>>
>> I have a bash script that uses send | receive to transfer snapshots to
>> a couple external HDD's. The script runs daily on a systemd timer. I
>> set all this up recently and I first noticed that it runs every day
>> and that the expected snapshots are received.
>>
>> At a glance, everything looked correct. However, today was my day to
>> drill down and really make sure everything was working.
>>
>> To my surprise, the newest received incremental snapshots are missing
>> all recent files. These new snapshots reflect the system state from
>> weeks ago and no files more recent than a certain date are in the
>> snapshots.
>>
>> However, the snapshots are newly created and newly received. The work
>> is being done fresh each day when my script runs, but the results are
>> anchored back in time at this earlier date. Weird.
>>
>> I'm not really sure where to start troubleshooting, so I'll start by
>> sharing part of my script. I'm sure the problem is in my script, and
>> is not related to BTRFS or snapper functionality. (As I said, the
>> Snapper snapshots are totally OK before being sent | received.
>>
>> These are the key lines of the script I'm using to send | receive a snapshot:
>>
>> old_num=$(snapper -c "$config" list -t single | awk
>> '/'"$selected_uuid"'/ {print $1}')
>> old_snap=$SUBVOLUME/.snapshots/$old_num/snapshot
>> new_num=$(snapper -c "$config" create --print-number)
>> new_snap=$SUBVOLUME/.snapshots/$new_num/snapshot
>> btrfs send -c "$old_snap" "$new_snap" | $ssh btrfs receive
>> "$backup_location"
>>
>> I have to admit that even after reading the following page half a
>> dozen times, I barely understand the difference between