send | receive: received snapshot is missing recent files
I'm running Arch Linux on BTRFS. I use Snapper to take hourly snapshots and it works without any issues. I have a bash script that uses send | receive to transfer snapshots to a couple external HDD's. The script runs daily on a systemd timer. I set all this up recently and I first noticed that it runs every day and that the expected snapshots are received. At a glance, everything looked correct. However, today was my day to drill down and really make sure everything was working. To my surprise, the newest received incremental snapshots are missing all recent files. These new snapshots reflect the system state from weeks ago and no files more recent than a certain date are in the snapshots. However, the snapshots are newly created and newly received. The work is being done fresh each day when my script runs, but the results are anchored back in time at this earlier date. Weird. I'm not really sure where to start troubleshooting, so I'll start by sharing part of my script. I'm sure the problem is in my script, and is not related to BTRFS or snapper functionality. (As I said, the Snapper snapshots are totally OK before being sent | received. These are the key lines of the script I'm using to send | receive a snapshot: old_num=$(snapper -c "$config" list -t single | awk '/'"$selected_uuid"'/ {print $1}') old_snap=$SUBVOLUME/.snapshots/$old_num/snapshot new_num=$(snapper -c "$config" create --print-number) new_snap=$SUBVOLUME/.snapshots/$new_num/snapshot btrfs send -c "$old_snap" "$new_snap" | $ssh btrfs receive "$backup_location" I have to admit that even after reading the following page half a dozen times, I barely understand the difference between -c and -p. https://btrfs.wiki.kernel.org/index.php/FAQ#What_is_the_difference_between_-c_and_-p_in_send.3F After reading that page again today, I feel like I should switch to -p (maybe). However, the -c vs -p choice probably isn't my problem. Any ideas what my problem could be? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send | receive: received snapshot is missing recent files
This is an even better set of steps for reproducing the problem. [root@srv]# sync [root@srv]# mkdir /home/.snapshots/test1 [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/ Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home' [root@srv]# sync [root@srv]# mkdir /mnt/x5a/home/test1 [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive /mnt/x5a/home/test1/ At subvol /home/.snapshots/test1/home/ At subvol home [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/ NOTE: all recent files are present [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/ NOTE: all recent files are present [root@srv]# mkdir /home/.snapshots/test2 [root@srv]# mkdir /mnt/x5a/home/test2 [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/ Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home' [root@srv]# sync [root@srv]# btrfs send -p /home/.snapshots/test1/home/ /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/ At subvol /home/.snapshots/test2/home/ At snapshot home [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/ NOTE: all recent files are MISSING [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/ NOTE: all recent files are MISSING Any ideas what could be causing this problem with incremental backups? On Wed, Sep 6, 2017 at 3:23 PM, Dave wrote: > > Here is more info on this problem. I can reproduce this without using my > script. Simple btrfs commands will reproduce the problem every time. The same > files are missing every time. There is no randomness to the missing data. > > Here are my steps: > > 1. snapper -c home create > result is a valid snapshot at /home/.snapshots/1704/snapshot > 2. btrfs send /home/.snapshots/1704/snapshot | btrfs receive > /mnt/x5a/home/1704 > 3. snapper -c home create > result is a valid snapshot at /home/.snapshots/1716/snapshot > 4. btrfs send -c /home/.snapshots/1704/snapshot/ > /home/.snapshots/1716/snapshot/ | btrfs receive /mnt/x5a/home/1716/ > > I expect /mnt/x5a/home/1716/snapshot to be identical to > /home/.snapshots/1716/snapshot. However, it is not. > The result is that /mnt/x5a/home/1716/snapshot is missing all recent files. > > Next step was to delete snapshot 1716 (in both locations) and repeat the send > | receive using -p > > btrfs su del /mnt/x5a/home/1716/snapshot > snapper -c home delete 1716 > snapper -c home create > btrfs send -p /home/.snapshots/1704/snapshot/ /home/.snapshots/1716/snapshot/ > | btrfs receive /mnt/x5a/home/1716/ > > The result is once again that /mnt/x5a/home/1716/snapshot is missing all > recent files. However, the other snapshots are all valid: > /home/.snapshots/1704/snapshot is valid & complete > /mnt/x5a/home/1704/snapshot -- non-incremental send: snapshot is valid & > complete > /home/.snapshots/1716/snapshot is valid & complete > /mnt/x5a/home/1716/snapshot -- incrementally sent snapshot is missing all > recent files whether sent with -c or -p > > The incrementally sent snapshot is even missing files that are present in the > reference snapshot /mnt/x5a/home/1704/snapshot. > > > > On Wed, Sep 6, 2017 at 1:37 AM, Dave wrote: >> >> I'm running Arch Linux on BTRFS. I use Snapper to take hourly >> snapshots and it works without any issues. >> >> I have a bash script that uses send | receive to transfer snapshots to >> a couple external HDD's. The script runs daily on a systemd timer. I >> set all this up recently and I first noticed that it runs every day >> and that the expected snapshots are received. >> >> At a glance, everything looked correct. However, today was my day to >> drill down and really make sure everything was working. >> >> To my surprise, the newest received incremental snapshots are missing >> all recent files. These new snapshots reflect the system state from >> weeks ago and no files more recent than a certain date are in the >> snapshots. >> >> However, the snapshots are newly created and newly received. The work >> is being done fresh each day when my script runs, but the results are >> anchored back in time at this earlier date. Weird. >> >> I'm not really sure where to start troubleshooting, so I'll start by >> sharing part of my script. I'm sure the problem is in my script, and >> is not related to BTRFS or snapper functionality. (As I said, the >> Snapper snapshots are totally OK before being sent | received. >> >> These are the key lines of the script I'm using to send | receive a snapshot: >> >> old_num=$(snapper -c "$config" list -t single | awk >> '/'"$selected_uuid"'/ {print $1}') >> old_snap=$SUBVOLUME/.snapshots/$old_num/snapshot >> new_num=$(snapper -c "$config" create --print-number) >> new_snap=$SUBVOLUME/.snapshots/$new_num/snapshot >> btrfs send -c "$old_snap" "$new_snap" | $ssh btrfs receive >> "$backup_location" >> >> I have to admit that even after reading the following page half a >> dozen times, I barely understand the difference between -c and -p. >> https://btrfs.wiki.kernel.org/index.
Re: send | receive: received snapshot is missing recent files
Here is more info and a possible (shocking) explanation. This aggregates my prior messages and it provides an almost complete set of steps to reproduce this problem. Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux btrfs-progs v4.12 My steps: [root@srv]# sync [root@srv]# mkdir /home/.snapshots/test1 [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/ Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home' [root@srv]# sync [root@srv]# mkdir /mnt/x5a/home/test1 [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive /mnt/x5a/home/test1/ At subvol /home/.snapshots/test1/home/ At subvol home [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/ NOTE: all recent files are present [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/ NOTE: all recent files are present [root@srv]# mkdir /home/.snapshots/test2 [root@srv]# mkdir /mnt/x5a/home/test2 [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/ Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home' [root@srv]# sync [root@srv]# btrfs send -p /home/.snapshots/test1/home/ /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/ At subvol /home/.snapshots/test2/home/ At snapshot home [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/ NOTE: all recent files are MISSING [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/ NOTE: all recent files are MISSING Below I am including some rsync output to illustrate when a snapshot is missing files (or not): [root@srv]# rsync -aniv /home/.snapshots/test1/home/ /home/.snapshots/test2/home/ sending incremental file list sent 1,143,286 bytes received 1,123 bytes 762,939.33 bytes/sec total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) This indicates that these two subvolumes contain the same files, which they should because test2 is a snapshot of test1 without any changes to files, and it was not sent to another physical device. The problem is when test2 is sent to another device as shown by the rsync results below. [root@srv]# rsync -aniv /home/.snapshots/test2/home/ /mnt/x5a/home/test2/home/ sending incremental file list .d..t.. ./ .d..t.. user1/ >f.st.. user1/.bash_history >f.st.. user1/.bashrc >f+ user1/test2017-09-06.txt ... and a long list of other missing files The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is missing all recent files (any files from the month of August or September), as my prior visual inspections had indicated. The same files are missing every time. There is no randomness to the missing data. The problem does not happen for me if the receive command target is located on the same physical device as shown next. (However, I suspect there's more to it than that, as explained further below.) [root@srv]# mkdir /home/.snapshots/test2rec [root@srv]# btrfs send -p /home/.snapshots/test1/home/ /home/.snapshots/test2/home/ | btrfs receive /home/.snapshots/test2rec/ At subvol /home/.snapshots/test2/home/ # rsync -aniv /home/.snapshots/test2/home/ /home/.snapshots/test2rec/home/ sending incremental file list sent 1,143,286 bytes received 1,123 bytes 2,288,818.00 bytes/sec total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) The above (as well as visual inspection of files) indicates that these two subvolumes contain the same files, which was not the case when the same command had a target located on another physical device. Of course, a snapshot which resides on the same physical device is not a very good backup. So I do need to send it to another device, but that results in missing files when the -p or -c options are used with btrfs send. (Non-incremental sending to another physical device does work.) I can think of a couple possible explanations. One is that there is a problem when using the -p or -c options with btrfs send when the target is another physical device. I suspect this is the actual explanation, however. A second possibility is that the presence of prior existing snapshots at the target location (even if old and not referenced in any current btrfs command), can determine the outcome and final contents of an incremental send operation. I believe the info below suggests this to be the case. [root@srv]# btrfs su show /home/.snapshots/test2/home/ test2/home Name: home UUID: 292e8bbf-a95f-2a4e-8280-129202d389dc Parent UUID:62418df6-a1f8-d74a-a152-11f519593053 Received UUID: e00d5318-6efd-824e-ac91-f25efa5c2a74 Creation time: 2017-09-06 15:38:16 -0400 Subvolume ID: 2000 Generation: 5020 Gen at creation:5020 Parent ID: 257 Top level ID: 257 Flags: readonly Snapshot(s): [root@srv]# btrfs su show /mnt/x5a/home/test1/home home/test1/home Name: home UUID:
Re: send | receive: received snapshot is missing recent files
The problem can be that you have a Received UUID on the source volume. This breaks send-receive. From: Dave -- Sent: 2017-09-07 - 06:43 > Here is more info and a possible (shocking) explanation. This > aggregates my prior messages and it provides an almost complete set of > steps to reproduce this problem. > > Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux > btrfs-progs v4.12 > > My steps: > > [root@srv]# sync > [root@srv]# mkdir /home/.snapshots/test1 > [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/ > Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home' > [root@srv]# sync > [root@srv]# mkdir /mnt/x5a/home/test1 > [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive > /mnt/x5a/home/test1/ > At subvol /home/.snapshots/test1/home/ > At subvol home > [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/ > NOTE: all recent files are present > [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/ > NOTE: all recent files are present > [root@srv]# mkdir /home/.snapshots/test2 > [root@srv]# mkdir /mnt/x5a/home/test2 > [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/ > Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home' > [root@srv]# sync > [root@srv]# btrfs send -p /home/.snapshots/test1/home/ > /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/ > At subvol /home/.snapshots/test2/home/ > At snapshot home > [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/ > NOTE: all recent files are MISSING > [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/ > NOTE: all recent files are MISSING > > Below I am including some rsync output to illustrate when a snapshot > is missing files (or not): > > [root@srv]# rsync -aniv /home/.snapshots/test1/home/ > /home/.snapshots/test2/home/ > sending incremental file list > > sent 1,143,286 bytes received 1,123 bytes 762,939.33 bytes/sec > total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) > > This indicates that these two subvolumes contain the same files, which > they should because test2 is a snapshot of test1 without any changes > to files, and it was not sent to another physical device. > > The problem is when test2 is sent to another device as shown by the > rsync results below. > > [root@srv]# rsync -aniv /home/.snapshots/test2/home/ /mnt/x5a/home/test2/home/ > sending incremental file list > .d..t.. ./ > .d..t.. user1/ >>f.st.. user1/.bash_history >>f.st.. user1/.bashrc >>f+ user1/test2017-09-06.txt > ... > and a long list of other missing files > > The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is > missing all recent files (any files from the month of August or > September), as my prior visual inspections had indicated. The same > files are missing every time. There is no randomness to the missing > data. > > The problem does not happen for me if the receive command target is > located on the same physical device as shown next. (However, I suspect > there's more to it than that, as explained further below.) > > [root@srv]# mkdir /home/.snapshots/test2rec > [root@srv]# btrfs send -p /home/.snapshots/test1/home/ > /home/.snapshots/test2/home/ | btrfs receive > /home/.snapshots/test2rec/ > At subvol /home/.snapshots/test2/home/ > > # rsync -aniv /home/.snapshots/test2/home/ /home/.snapshots/test2rec/home/ > sending incremental file list > > sent 1,143,286 bytes received 1,123 bytes 2,288,818.00 bytes/sec > total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) > > The above (as well as visual inspection of files) indicates that these > two subvolumes contain the same files, which was not the case when the > same command had a target located on another physical device. Of > course, a snapshot which resides on the same physical device is not a > very good backup. So I do need to send it to another device, but that > results in missing files when the -p or -c options are used with btrfs > send. (Non-incremental sending to another physical device does work.) > > I can think of a couple possible explanations. > > One is that there is a problem when using the -p or -c options with > btrfs send when the target is another physical device. I suspect this > is the actual explanation, however. > > A second possibility is that the presence of prior existing snapshots > at the target location (even if old and not referenced in any current > btrfs command), can determine the outcome and final contents of an > incremental send operation. I believe the info below suggests this to > be the case. > > [root@srv]# btrfs su show /home/.snapshots/test2/home/ > test2/home > Name: home > UUID: 292e8bbf-a95f-2a4e-8280-129202d389dc > Parent UUID:62418df6-a1f8-d74a-a152-11f519593053 > Received UUID: e00d5318-6efd-824e-ac91-f25efa5c2a74 > Creation time: 2017-09-06 15:38:16 -0400 >
Re: send | receive: received snapshot is missing recent files
Hello. Can anyone further explain this issue ("you have a Received UUID on the source volume")? How does it happen? How does one remove a Received UUID from the source volume? And how does that explain my results where I showed that the problem is not dependent upon the source volume but is instead dependent upon some existing snapshot on the target volume? My results do not appear to be fully explained by a Received UUID on the source volume, as my prior message hopefully shows clearly. Thank you. On Thu, Sep 7, 2017 at 2:24 AM, A L wrote: > The problem can be that you have a Received UUID on the source volume. This > breaks send-receive. > > From: Dave -- Sent: 2017-09-07 - 06:43 > >> Here is more info and a possible (shocking) explanation. This >> aggregates my prior messages and it provides an almost complete set of >> steps to reproduce this problem. >> >> Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux >> btrfs-progs v4.12 >> >> My steps: >> >> [root@srv]# sync >> [root@srv]# mkdir /home/.snapshots/test1 >> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/ >> Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home' >> [root@srv]# sync >> [root@srv]# mkdir /mnt/x5a/home/test1 >> [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive >> /mnt/x5a/home/test1/ >> At subvol /home/.snapshots/test1/home/ >> At subvol home >> [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/ >> NOTE: all recent files are present >> [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/ >> NOTE: all recent files are present >> [root@srv]# mkdir /home/.snapshots/test2 >> [root@srv]# mkdir /mnt/x5a/home/test2 >> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/ >> Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home' >> [root@srv]# sync >> [root@srv]# btrfs send -p /home/.snapshots/test1/home/ >> /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/ >> At subvol /home/.snapshots/test2/home/ >> At snapshot home >> [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/ >> NOTE: all recent files are MISSING >> [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/ >> NOTE: all recent files are MISSING >> >> Below I am including some rsync output to illustrate when a snapshot >> is missing files (or not): >> >> [root@srv]# rsync -aniv /home/.snapshots/test1/home/ >> /home/.snapshots/test2/home/ >> sending incremental file list >> >> sent 1,143,286 bytes received 1,123 bytes 762,939.33 bytes/sec >> total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) >> >> This indicates that these two subvolumes contain the same files, which >> they should because test2 is a snapshot of test1 without any changes >> to files, and it was not sent to another physical device. >> >> The problem is when test2 is sent to another device as shown by the >> rsync results below. >> >> [root@srv]# rsync -aniv /home/.snapshots/test2/home/ >> /mnt/x5a/home/test2/home/ >> sending incremental file list >> .d..t.. ./ >> .d..t.. user1/ >>>f.st.. user1/.bash_history >>>f.st.. user1/.bashrc >>>f+ user1/test2017-09-06.txt >> ... >> and a long list of other missing files >> >> The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is >> missing all recent files (any files from the month of August or >> September), as my prior visual inspections had indicated. The same >> files are missing every time. There is no randomness to the missing >> data. >> >> The problem does not happen for me if the receive command target is >> located on the same physical device as shown next. (However, I suspect >> there's more to it than that, as explained further below.) >> >> [root@srv]# mkdir /home/.snapshots/test2rec >> [root@srv]# btrfs send -p /home/.snapshots/test1/home/ >> /home/.snapshots/test2/home/ | btrfs receive >> /home/.snapshots/test2rec/ >> At subvol /home/.snapshots/test2/home/ >> >> # rsync -aniv /home/.snapshots/test2/home/ /home/.snapshots/test2rec/home/ >> sending incremental file list >> >> sent 1,143,286 bytes received 1,123 bytes 2,288,818.00 bytes/sec >> total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) >> >> The above (as well as visual inspection of files) indicates that these >> two subvolumes contain the same files, which was not the case when the >> same command had a target located on another physical device. Of >> course, a snapshot which resides on the same physical device is not a >> very good backup. So I do need to send it to another device, but that >> results in missing files when the -p or -c options are used with btrfs >> send. (Non-incremental sending to another physical device does work.) >> >> I can think of a couple possible explanations. >> >> One is that there is a problem when using the -p or -c options with >> btrfs send when the target is another physical device. I suspect this >> is the actual explanation, however. >> >> A second possibility is that the presence
Re: send | receive: received snapshot is missing recent files
I just ran a test. The btrfs send - receive problem I described is indeed fully resolved by removing the "problematic" snapshot on the target device. I did not make any changes to the source volume. I did not make any other changes in my steps (see earlier message for my exact steps). Therefore, the problem I described in my earlier message is not due exclusively to having a Received UUID on the source volume (or to any other feature of the source volume). It is not related to any feature of the directly specified parent volume either. More details are included in my earlier email. Thanks for any further feedback, including answers to my questions and comments about whether this is a known issue. On Thu, Sep 7, 2017 at 8:39 AM, Dave wrote: > > Hello. Can anyone further explain this issue ("you have a Received UUID on > the source volume")? > > How does it happen? > How does one remove a Received UUID from the source volume? > > And how does that explain my results where I showed that the problem > is not dependent upon the source volume but is instead dependent upon > some existing snapshot on the target volume? > > My results do not appear to be fully explained by a Received UUID on the > source volume, as my prior message hopefully shows clearly. > > Thank you. > > On Thu, Sep 7, 2017 at 2:24 AM, A L wrote: > > The problem can be that you have a Received UUID on the source volume. This > > breaks send-receive. > > > > From: Dave -- Sent: 2017-09-07 - 06:43 > > > >> Here is more info and a possible (shocking) explanation. This > >> aggregates my prior messages and it provides an almost complete set of > >> steps to reproduce this problem. > >> > >> Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux > >> btrfs-progs v4.12 > >> > >> My steps: > >> > >> [root@srv]# sync > >> [root@srv]# mkdir /home/.snapshots/test1 > >> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/ > >> Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home' > >> [root@srv]# sync > >> [root@srv]# mkdir /mnt/x5a/home/test1 > >> [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive > >> /mnt/x5a/home/test1/ > >> At subvol /home/.snapshots/test1/home/ > >> At subvol home > >> [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/ > >> NOTE: all recent files are present > >> [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/ > >> NOTE: all recent files are present > >> [root@srv]# mkdir /home/.snapshots/test2 > >> [root@srv]# mkdir /mnt/x5a/home/test2 > >> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/ > >> Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home' > >> [root@srv]# sync > >> [root@srv]# btrfs send -p /home/.snapshots/test1/home/ > >> /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/ > >> At subvol /home/.snapshots/test2/home/ > >> At snapshot home > >> [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/ > >> NOTE: all recent files are MISSING > >> [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/ > >> NOTE: all recent files are MISSING > >> > >> Below I am including some rsync output to illustrate when a snapshot > >> is missing files (or not): > >> > >> [root@srv]# rsync -aniv /home/.snapshots/test1/home/ > >> /home/.snapshots/test2/home/ > >> sending incremental file list > >> > >> sent 1,143,286 bytes received 1,123 bytes 762,939.33 bytes/sec > >> total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) > >> > >> This indicates that these two subvolumes contain the same files, which > >> they should because test2 is a snapshot of test1 without any changes > >> to files, and it was not sent to another physical device. > >> > >> The problem is when test2 is sent to another device as shown by the > >> rsync results below. > >> > >> [root@srv]# rsync -aniv /home/.snapshots/test2/home/ > >> /mnt/x5a/home/test2/home/ > >> sending incremental file list > >> .d..t.. ./ > >> .d..t.. user1/ > >>>f.st.. user1/.bash_history > >>>f.st.. user1/.bashrc > >>>f+ user1/test2017-09-06.txt > >> ... > >> and a long list of other missing files > >> > >> The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is > >> missing all recent files (any files from the month of August or > >> September), as my prior visual inspections had indicated. The same > >> files are missing every time. There is no randomness to the missing > >> data. > >> > >> The problem does not happen for me if the receive command target is > >> located on the same physical device as shown next. (However, I suspect > >> there's more to it than that, as explained further below.) > >> > >> [root@srv]# mkdir /home/.snapshots/test2rec > >> [root@srv]# btrfs send -p /home/.snapshots/test1/home/ > >> /home/.snapshots/test2/home/ | btrfs receive > >> /home/.snapshots/test2rec/ > >> At subvol /home/.snapshots/test2/home/ > >> > >> # rsync -aniv /home/.snapshots/test2/home/ /home/.snapshots/test2rec/home/ > >> se
Re: send | receive: received snapshot is missing recent files
Having a received_uuid set on the source volume ("/home" in your case) is indeed a bad thing when it comes to send/receive. You probably restored a backup with send/receive, and made it read/write using "btrfs property set -ts /home ro false". This is a an evil thing, as it leaves received_uuid intact. In order to make a subvolume read-write, I recommend to use "btrfs subvolume snapshot ". There is a FAQ entry on btrbk on how to fix this: https://github.com/digint/btrbk/blob/master/doc/FAQ.md#im-getting-an-error-aborted-received-uuid-is-set On 2017-09-07 15:34, Dave wrote: > I just ran a test. The btrfs send - receive problem I described is > indeed fully resolved by removing the "problematic" snapshot on the > target device. I did not make any changes to the source volume. I did > not make any other changes in my steps (see earlier message for my > exact steps). > > Therefore, the problem I described in my earlier message is not due > exclusively to having a Received UUID on the source volume (or to any > other feature of the source volume). It is not related to any feature > of the directly specified parent volume either. More details are > included in my earlier email. > > Thanks for any further feedback, including answers to my questions and > comments about whether this is a known issue. > > > On Thu, Sep 7, 2017 at 8:39 AM, Dave wrote: >> >> Hello. Can anyone further explain this issue ("you have a Received UUID on >> the source volume")? >> >> How does it happen? >> How does one remove a Received UUID from the source volume? >> >> And how does that explain my results where I showed that the problem >> is not dependent upon the source volume but is instead dependent upon >> some existing snapshot on the target volume? >> >> My results do not appear to be fully explained by a Received UUID on the >> source volume, as my prior message hopefully shows clearly. >> >> Thank you. >> >> On Thu, Sep 7, 2017 at 2:24 AM, A L wrote: >>> The problem can be that you have a Received UUID on the source volume. This >>> breaks send-receive. >>> >>> From: Dave -- Sent: 2017-09-07 - 06:43 >>> Here is more info and a possible (shocking) explanation. This aggregates my prior messages and it provides an almost complete set of steps to reproduce this problem. Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux btrfs-progs v4.12 My steps: [root@srv]# sync [root@srv]# mkdir /home/.snapshots/test1 [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/ Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home' [root@srv]# sync [root@srv]# mkdir /mnt/x5a/home/test1 [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive /mnt/x5a/home/test1/ At subvol /home/.snapshots/test1/home/ At subvol home [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/ NOTE: all recent files are present [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/ NOTE: all recent files are present [root@srv]# mkdir /home/.snapshots/test2 [root@srv]# mkdir /mnt/x5a/home/test2 [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/ Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home' [root@srv]# sync [root@srv]# btrfs send -p /home/.snapshots/test1/home/ /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/ At subvol /home/.snapshots/test2/home/ At snapshot home [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/ NOTE: all recent files are MISSING [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/ NOTE: all recent files are MISSING Below I am including some rsync output to illustrate when a snapshot is missing files (or not): [root@srv]# rsync -aniv /home/.snapshots/test1/home/ /home/.snapshots/test2/home/ sending incremental file list sent 1,143,286 bytes received 1,123 bytes 762,939.33 bytes/sec total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) This indicates that these two subvolumes contain the same files, which they should because test2 is a snapshot of test1 without any changes to files, and it was not sent to another physical device. The problem is when test2 is sent to another device as shown by the rsync results below. [root@srv]# rsync -aniv /home/.snapshots/test2/home/ /mnt/x5a/home/test2/home/ sending incremental file list .d..t.. ./ .d..t.. user1/ > f.st.. user1/.bash_history > f.st.. user1/.bashrc > f+ user1/test2017-09-06.txt ... and a long list of other missing files The incrementally sent snapshot at /mnt/x5a/home/test2/home/ is missing all recent files (any files from the month of August or September), as my prior visual inspections had indicated. The
Re: send | receive: received snapshot is missing recent files
I'm referring to the link below. Using "btrfs subvolume snapshot -r" copies the Received UUID from the source into the new snapshot. The btrbk FAQ entry suggests otherwise. Has something changed? The only way I see to remove a Received UUID is to create a rw snapshot (above command without the "-r"), which is not ideal in this situation when cleaning up readonly source snapshots. Any suggestions? Thanks On Thu, Sep 7, 2017 at 10:33 AM, Axel Burri wrote: > > Having a received_uuid set on the source volume ("/home" in your case) > is indeed a bad thing when it comes to send/receive. You probably > restored a backup with send/receive, and made it read/write using "btrfs > property set -ts /home ro false". This is a an evil thing, as it leaves > received_uuid intact. In order to make a subvolume read-write, I > recommend to use "btrfs subvolume snapshot ". > > There is a FAQ entry on btrbk on how to fix this: > > https://github.com/digint/btrbk/blob/master/doc/FAQ.md#im-getting-an-error-aborted-received-uuid-is-set > > > On 2017-09-07 15:34, Dave wrote: > > I just ran a test. The btrfs send - receive problem I described is > > indeed fully resolved by removing the "problematic" snapshot on the > > target device. I did not make any changes to the source volume. I did > > not make any other changes in my steps (see earlier message for my > > exact steps). > > > > Therefore, the problem I described in my earlier message is not due > > exclusively to having a Received UUID on the source volume (or to any > > other feature of the source volume). It is not related to any feature > > of the directly specified parent volume either. More details are > > included in my earlier email. > > > > Thanks for any further feedback, including answers to my questions and > > comments about whether this is a known issue. > > > > > > On Thu, Sep 7, 2017 at 8:39 AM, Dave wrote: > >> > >> Hello. Can anyone further explain this issue ("you have a Received UUID on > >> the source volume")? > >> > >> How does it happen? > >> How does one remove a Received UUID from the source volume? > >> > >> And how does that explain my results where I showed that the problem > >> is not dependent upon the source volume but is instead dependent upon > >> some existing snapshot on the target volume? > >> > >> My results do not appear to be fully explained by a Received UUID on the > >> source volume, as my prior message hopefully shows clearly. > >> > >> Thank you. > >> > >> On Thu, Sep 7, 2017 at 2:24 AM, A L wrote: > >>> The problem can be that you have a Received UUID on the source volume. > >>> This breaks send-receive. > >>> > >>> From: Dave -- Sent: 2017-09-07 - 06:43 > >>> > Here is more info and a possible (shocking) explanation. This > aggregates my prior messages and it provides an almost complete set of > steps to reproduce this problem. > > Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 > GNU/Linux > btrfs-progs v4.12 > > My steps: > > [root@srv]# sync > [root@srv]# mkdir /home/.snapshots/test1 > [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/ > Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home' > [root@srv]# sync > [root@srv]# mkdir /mnt/x5a/home/test1 > [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive > /mnt/x5a/home/test1/ > At subvol /home/.snapshots/test1/home/ > At subvol home > [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/ > NOTE: all recent files are present > [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/ > NOTE: all recent files are present > [root@srv]# mkdir /home/.snapshots/test2 > [root@srv]# mkdir /mnt/x5a/home/test2 > [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/ > Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home' > [root@srv]# sync > [root@srv]# btrfs send -p /home/.snapshots/test1/home/ > /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/ > At subvol /home/.snapshots/test2/home/ > At snapshot home > [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/ > NOTE: all recent files are MISSING > [root@srv]# ls -la /mnt/x5a/home/test2/home/user2/Documents/ > NOTE: all recent files are MISSING > > Below I am including some rsync output to illustrate when a snapshot > is missing files (or not): > > [root@srv]# rsync -aniv /home/.snapshots/test1/home/ > /home/.snapshots/test2/home/ > sending incremental file list > > sent 1,143,286 bytes received 1,123 bytes 762,939.33 bytes/sec > total size is 3,642,972,271 speedup is 3,183.28 (DRY RUN) > > This indicates that these two subvolumes contain the same files, which > they should because test2 is a snapshot of test1 without any changes > to files, and it was not sent to another
Re: send | receive: received snapshot is missing recent files
On 2017-09-08 06:44, Dave wrote: > I'm referring to the link below. Using "btrfs subvolume snapshot -r" > copies the Received UUID from the source into the new snapshot. The > btrbk FAQ entry suggests otherwise. Has something changed? I don't think something has changed, the description for the read-only subvolumes on the btrbk FAQ was just wrong (fixed now). > The only way I see to remove a Received UUID is to create a rw > snapshot (above command without the "-r"), which is not ideal in this > situation when cleaning up readonly source snapshots. > > Any suggestions? Thanks No suggestions from my part, as far as I know there is no way to easily remove/change a received_uuid from a subvolume. As you mentioned, you can snapshot it twice: # btrfs subvolume snapshot mysubvol mysubvol.rw # btrfs subvolume delete mysubvol # btrfs subvolume snapshot -r mysubvol.rw mysubvol # btrfs subvolume delete mysubvol.rw Instead of the second snapshot operation, this time you could also use the (evil) command: "btrfs btrfs property set -ts mysnapshot ro true" > On Thu, Sep 7, 2017 at 10:33 AM, Axel Burri wrote: >> >> Having a received_uuid set on the source volume ("/home" in your case) >> is indeed a bad thing when it comes to send/receive. You probably >> restored a backup with send/receive, and made it read/write using "btrfs >> property set -ts /home ro false". This is a an evil thing, as it leaves >> received_uuid intact. In order to make a subvolume read-write, I >> recommend to use "btrfs subvolume snapshot ". >> >> There is a FAQ entry on btrbk on how to fix this: >> >> https://github.com/digint/btrbk/blob/master/doc/FAQ.md#im-getting-an-error-aborted-received-uuid-is-set >> >> >> On 2017-09-07 15:34, Dave wrote: >>> I just ran a test. The btrfs send - receive problem I described is >>> indeed fully resolved by removing the "problematic" snapshot on the >>> target device. I did not make any changes to the source volume. I did >>> not make any other changes in my steps (see earlier message for my >>> exact steps). >>> >>> Therefore, the problem I described in my earlier message is not due >>> exclusively to having a Received UUID on the source volume (or to any >>> other feature of the source volume). It is not related to any feature >>> of the directly specified parent volume either. More details are >>> included in my earlier email. >>> >>> Thanks for any further feedback, including answers to my questions and >>> comments about whether this is a known issue. >>> >>> >>> On Thu, Sep 7, 2017 at 8:39 AM, Dave wrote: Hello. Can anyone further explain this issue ("you have a Received UUID on the source volume")? How does it happen? How does one remove a Received UUID from the source volume? And how does that explain my results where I showed that the problem is not dependent upon the source volume but is instead dependent upon some existing snapshot on the target volume? My results do not appear to be fully explained by a Received UUID on the source volume, as my prior message hopefully shows clearly. Thank you. On Thu, Sep 7, 2017 at 2:24 AM, A L wrote: > The problem can be that you have a Received UUID on the source volume. > This breaks send-receive. > > From: Dave -- Sent: 2017-09-07 - 06:43 > >> Here is more info and a possible (shocking) explanation. This >> aggregates my prior messages and it provides an almost complete set of >> steps to reproduce this problem. >> >> Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 >> GNU/Linux >> btrfs-progs v4.12 >> >> My steps: >> >> [root@srv]# sync >> [root@srv]# mkdir /home/.snapshots/test1 >> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test1/ >> Create a readonly snapshot of '/home/' in '/home/.snapshots/test1//home' >> [root@srv]# sync >> [root@srv]# mkdir /mnt/x5a/home/test1 >> [root@srv]# btrfs send /home/.snapshots/test1/home/ | btrfs receive >> /mnt/x5a/home/test1/ >> At subvol /home/.snapshots/test1/home/ >> At subvol home >> [root@srv]# ls -la /mnt/x5a/home/test1/home/user1/ >> NOTE: all recent files are present >> [root@srv]# ls -la /mnt/x5a/home/test1/home/user2/Documents/ >> NOTE: all recent files are present >> [root@srv]# mkdir /home/.snapshots/test2 >> [root@srv]# mkdir /mnt/x5a/home/test2 >> [root@srv]# btrfs su sn -r /home/ /home/.snapshots/test2/ >> Create a readonly snapshot of '/home/' in '/home/.snapshots/test2//home' >> [root@srv]# sync >> [root@srv]# btrfs send -p /home/.snapshots/test1/home/ >> /home/.snapshots/test2/home/ | btrfs receive /mnt/x5a/home/test2/ >> At subvol /home/.snapshots/test2/home/ >> At snapshot home >> [root@srv]# ls -la /mnt/x5a/home/test2/home/user1/ >> NOTE: all recent files are MISSING >> [root@srv]# ls -la /mnt/x5a/home/test2/ho
Re: send | receive: received snapshot is missing recent files
11.09.2017 20:53, Axel Burri пишет: > On 2017-09-08 06:44, Dave wrote: >> I'm referring to the link below. Using "btrfs subvolume snapshot -r" >> copies the Received UUID from the source into the new snapshot. The >> btrbk FAQ entry suggests otherwise. Has something changed? > > I don't think something has changed, the description for the read-only > subvolumes on the btrbk FAQ was just wrong (fixed now). > >> The only way I see to remove a Received UUID is to create a rw >> snapshot (above command without the "-r"), which is not ideal in this >> situation when cleaning up readonly source snapshots. >> >> Any suggestions? Thanks > > No suggestions from my part, as far as I know there is no way to easily > remove/change a received_uuid from a subvolume. > There is BTRFS_IOC_SET_RECEIVED_SUBVOL IOCTL which is used by "btrfs received". My understanding is that it can also be set to empty (this clearing it). You could write small program to do it. In general it sounds like a bug - removing read-only flag from subvolume by any means should also clear Received UUID as we cannot anymore guarantee that subvolume content is the same. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send | receive: received snapshot is missing recent files
On Mon, Sep 11, 2017 at 11:19 PM, Andrei Borzenkov wrote: > 11.09.2017 20:53, Axel Burri пишет: >> On 2017-09-08 06:44, Dave wrote: >>> I'm referring to the link below. Using "btrfs subvolume snapshot -r" >>> copies the Received UUID from the source into the new snapshot. The >>> btrbk FAQ entry suggests otherwise. Has something changed? >> >> I don't think something has changed, the description for the read-only >> subvolumes on the btrbk FAQ was just wrong (fixed now). >> >>> The only way I see to remove a Received UUID is to create a rw >>> snapshot (above command without the "-r"), which is not ideal in this >>> situation when cleaning up readonly source snapshots. >>> >>> Any suggestions? Thanks >> >> No suggestions from my part, as far as I know there is no way to easily >> remove/change a received_uuid from a subvolume. >> > > There is BTRFS_IOC_SET_RECEIVED_SUBVOL IOCTL which is used by "btrfs > received". My understanding is that it can also be set to empty (this > clearing it). You could write small program to do it. > > In general it sounds like a bug - removing read-only flag from subvolume > by any means should also clear Received UUID as we cannot anymore > guarantee that subvolume content is the same. Yes! That makes a great deal of sense. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html