On Wed, May 16, 2018 at 01:40:56PM -0600, Martin Fick wrote:
> > In theory the fetch means that it's safe to actually prune
> > in the mother repo, but in practice there are still
> > races. They don't come up often, but if you have enough
> > repositories, they do eventually. :)
>
> Peff,
>
> I would be very curious to hear what you think of this
> approach to mitigating the effect of those races?
>
> https://git.eclipse.org/r/c/122288/2
The crux of the problem is that we have no way to atomically mark an
object as "I am using this -- do not delete" with respect to the actual
deletion.
So if I'm reading your approach correctly, you put objects into a
purgatory rather than delete them, and let some operations rescue them
from purgatory if we had a race. That's certainly a direction we've
considered, but I think there are some open questions, like:
1. When do you rescue from purgatory? Any time the object is
referenced? Do you then pull in all of its reachable objects too?
2. How do you decide when to drop an object from purgatory? And
specifically, how do you avoid racing with somebody using the
object as you're pruning purgatory?
3. How do you know that an operation has been run that will actually
rescue the object, as opposed to silently having a corrupted state
on disk?
E.g., imagine this sequence:
a. git-prune computes reachability and finds that commit X is
ready to be pruned
b. another process sees that commit X exists and builds a commit
that references it as a parent
c. git-prune drops the object into purgatory
Now we have a corrupt state created by the process in (b), since we
have a reachable object in purgatory. But what if nobody goes back
and tries to read those commits in the meantime?
I think this might be solvable by using the purgatory as a kind of
"lock", where prune does something like:
1. compute reachability
2. move candidate objects into purgatory; nobody can look into
purgatory except us
3. compute reachability _again_, making sure that no purgatory objects
are used (if so, rollback the deletion and try again)
But even that's not quite there, because you need to have some
consistent atomic view of what's "used". Just checking refs isn't
enough, because some other process may be planning to reference a
purgatory object but not yet have updated the ref. So you need some
atomic way of saying "I am interested in using this object".
-Peff
