Re: [resend] btrfs-send -c fails: reproduction case
Zachary Vance posted on Sun, 17 Apr 2016 18:46:13 -0700 as excerpted: > I was already making sure all -c references were both present and > unmodified, I think the confusion is mostly around whether the parent > required to use -c, and whether it's an implicit reference volume in > particular. If it's required, it's impossible to preserve de-duplication > after deleting the original parent which would be really bad. AFAIK, send -c does require a parent, but if there's clones and no parent given, it will try to pick its own from the list of clones given. Which explains the error complaining about being unable to find a parent, when given only clone sources that (based on your reproducer) had nothing in common with the snapshots listed as clones. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [resend] btrfs-send -c fails: reproduction case
>>> Reproduction case after running into the same problem as Paride >>> Legovini: >>> http://article.gmane.org/gmane.comp.file-systems.btrfs/48706/match=send Your case is not the same as in this thread from Paride IMO. The error message is the same, but that doesn't mean the call tree leading to it is the same. My first impression from your 1st and also your 2nd (in resend thread) example was that btrfs' error message is correct and usage of -c is wrong. More important is that the problem that Paride describes, is solved for kernel/tools v4.5.0/v4.5.1 [...] > Every time I start asking about snapshots people tell me how I could > set up my process differently which is annoying because: > 1) They assume I'm doing a single, incremental backup process, which is wrong > 2) I already have years/terabytes pf data I need to transfer, so I > can't change what I did in the past > 3) They inevitably recommend their favorite method, which doesn't work > for me, because I have different requirements You would have to write a clear set of requirements, as an extension to what btrfs send|receive currently can do. >From the examples it is simply not clear what you want. [...] > Anyway, I'm doing a one-time transfer of a large number of snapshots > which are currently stuck on a .img file. Over the years they've been So, this is yet another problem; What is the background here? > re-organized repeatedly and I've snapshotted many from writable to > read-only (most of the cases of the parent being deleted, I think). My > goal is to get them into a new filesystem, but I can't do a full send > in every case, because then it would balloon up to a petabyte or so. This gives some hint what the situation is. How many ro snapshots is it? How much data in total as uncompressed diskblocks? >> Does that make more sense now, or were you actually already doing it that >> way and still getting the errors, and just didn't say so, or did I just >> confuse you even further (hopefully not!)? > > I was already making sure all -c references were both present and > unmodified, I think the confusion is mostly around whether the parent > required to use -c, and whether it's an implicit reference volume in > particular. If it's required, it's impossible to preserve > de-duplication after deleting the original parent which would be > really bad. Can you give examples for your real data how bad it is? And how you expect or want it to be ? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [resend] btrfs-send -c fails: reproduction case
+linux-btrfs and with new policy On 04/16/2016 08:37 PM, Duncan wrote: > Zachary Vance posted on Sat, 16 Apr 2016 13:08:17 -0700 as excerpted: > >> Reproduction case after running into the same problem as Paride >> Legovini: >> http://article.gmane.org/gmane.comp.file-systems.btrfs/48706/match=send > > [As a btrfs user and list-regular with a personal use-case that doesn't > include send/receive, I generally skip posts on that topic, leaving them > for those with more direct experience. Except... the above thread didn't > seem to get /any/ replies except from others with the same problem, and > while you just posted yours a few hours ago, yours hasn't either so far. > But while I don't follow send/receive /that/ closely and could be wrong, > I did happen to read this one and think I know the problem, so...] So... I may need to re-send the original post, basically, Chris Murphy privately pointed a problem with my mail hosting which causes it to be classified as spam by gmail. (Mailing lists can't validly format mail as me with my restrictive SPF policy). > OK, now you delete the parent subvolumes and the subvolume that contained > them, leaving: > > /root/ (pwd and there originally) > | > + a/ (/root/a, ro snap of (now deleted) /root/m/a, no parent left) > | > + b/ (/root/b, ro snap of (now deleted) /root/m/b, no parent left) > >> [root@burn ~]# sudo btrfs send -c a b >/dev/null >> At subvol b ERROR: parent determination failed for 9622 > > AFAIK, that's entirely expected, because there /is/ no parent now to > incrementally send against -- you deleted the parents and can no longer > incrementally send against them. I'm not incrementally sending against the parent, as I understand -c. Failure here was surprising. Also, I'd like to flag that there are two notions of "parent" -- a parent ID and a parent_uuid, and it would be great if that error message specified which more clearly since both are involved in the reproduction case. The exact same steps, keeping 'm' as a folder rather than a subvolume, also fails, which clarifies what's happening. The message is related to parent_uuid. > Instead, at this point you have to do a full (non-incremental) send, > since there's nothing to match against as the parent for an incremental > send. > > > However, what I believe you /wanted/ to do is something like this, > assuming monthly snapshots to make it easy: > [...] Every time I start asking about snapshots people tell me how I could set up my process differently which is annoying because: 1) They assume I'm doing a single, incremental backup process, which is wrong 2) I already have years/terabytes pf data I need to transfer, so I can't change what I did in the past 3) They inevitably recommend their favorite method, which doesn't work for me, because I have different requirements I realize you're also trying to provide basic explanation of -p and -c here but it's EVERY time I ask about snapshots... Anyway, I'm doing a one-time transfer of a large number of snapshots which are currently stuck on a .img file. Over the years they've been re-organized repeatedly and I've snapshotted many from writable to read-only (most of the cases of the parent being deleted, I think). My goal is to get them into a new filesystem, but I can't do a full send in every case, because then it would balloon up to a petabyte or so. > Does that make more sense now, or were you actually already doing it that > way and still getting the errors, and just didn't say so, or did I just > confuse you even further (hopefully not!)? I was already making sure all -c references were both present and unmodified, I think the confusion is mostly around whether the parent required to use -c, and whether it's an implicit reference volume in particular. If it's required, it's impossible to preserve de-duplication after deleting the original parent which would be really bad. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [resend] btrfs-send -c fails: reproduction case
On Sun, Apr 17, 2016 at 3:55 PM, Zachary Vancewrote: > To recap, the basic problem is that you can get into a state where > "btrfs-send -c" fails for a subvolume, with "ERROR: parent > determination failed for 9622". It appears the problem happens when > the 'parent_uuid' does not exist. > > I'm pretty sure this isn't the intended behavior, and it's certainly > preventing me from copying my backups right now. Duncan replied to the original post. http://www.spinics.net/lists/linux-btrfs/msg54275.html -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[resend] btrfs-send -c fails: reproduction case
Apologies for the repeat send for anyone that gets this twice; I was informed there was a problem with my mail setup and many ISPs would block the first message. I have also simplified the reproduction case. --- Reproduction case after running into the same problem as Paride Legovini: http://article.gmane.org/gmane.comp.file-systems.btrfs/48706/match=send To recap, the basic problem is that you can get into a state where "btrfs-send -c" fails for a subvolume, with "ERROR: parent determination failed for 9622". It appears the problem happens when the 'parent_uuid' does not exist. I'm pretty sure this isn't the intended behavior, and it's certainly preventing me from copying my backups right now. burn% sudo btrfs subvol create a Create subvolume './a' burn% sudo btrfs subvol snapshot a -r a-r Create a readonly snapshot of 'a' in './a-r' burn% sudo btrfs subvol create b1 Create subvolume './b1' burn% sudo btrfs subvol snapshot b1 -r b2 Create a readonly snapshot of 'b1' in './b2' burn% sudo btrfs subvol del b1 Delete subvolume (no-commit): '/data/b1' burn% sudo btrfs send -c b2 a-r >/dev/null At subvol a-r ERROR: parent determination failed for 9637 /data/a-r Name: a-r UUID: 819415e9-ea57-4a4d-9745-78a65f19b614 Parent UUID:e63b985a-3463-be4c-8ad5-900cd147ef7d Received UUID: - Creation time: 2016-02-20 10:19:59 -0800 Subvolume ID: 9638 Generation: 18524 Gen at creation:18524 Parent ID: 5 Top level ID: 5 Flags: readonly Snapshot(s): burn% sudo btrfs subvolume show b2 /data/b2 Name: b2 UUID: e1d0314d-9c2a-5240-9f4c-1716b7e87741 Parent UUID:00920504-c78b-ee42-af9f-05f4600321bb Received UUID: - Creation time: 2016-02-20 10:19:01 -0800 Subvolume ID: 9637 Generation: 18522 Gen at creation:18522 Parent ID: 5 Top level ID: 5 Flags: readonly Snapshot(s): [root@burn ~]# btrfs --version btrfs-progs v4.4.1 [root@burn ~]# uname -a Linux burn 4.5.0-1-ARCH #1 SMP PREEMPT Tue Mar 15 09:41:03 CET 2016 x86_64 GNU/Linux --- Zachary Vance -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html