As I understand it, the problem is that mirrors want to mirror Fedora using generic rsync, and ostree repositories have huge numbers of small files. To use rsync requires, at a minimum, a full directory walk, calling stat() every file on both the source and destination.
From the little I know about casync, I expect it to be better handling lots of small files, but I don't think it can avoid the full directory walk. If we can get mirrors to run custom software, then a really good protocol for mirroring OSTree repositories is OSTree - especially if the source repository is hosted with HTTP/2 support. Because of the content-addressed-storage nature of OSTree, no matter how large some portion of the repository is, if it hasn't changed, it takes no time to transfer it. I'd have to study your code to really know how it would compare - it looks like what it basically saves is that the source repository doesn't have to do the stat walk for *every* client, just once, but the clients still do a full walk each time. If it's more palatable to mirrors to run something that is just standard software + a zsh script, then it could be a big improvement, but I wouldn't expect it to be nearly efficient as mirroring via OSTree. The proof would be in the pudding - in testing how these different mechanisms actually compare in practice. Owen ----- Original Message ----- > >>>>> "OT" == Owen Taylor <otay...@redhat.com> writes: > > OT> The issue with this is that nobody has yet figured out how to handle > OT> OSTree repositories within the Fedora mirror infrastructure. While > OT> OSTree repositories can be mirrored efficiently, they can't be > OT> mirrored efficiently by rsync. > > Is the problem simply quantity of files? Because I have been doing work > related to that in the form of https://pagure.io/quick-fedora-mirror and > it's worked pretty well so far. > > If the problem is simply the volume of churn or something else, then > that's a separate problem which may or may not be alleviated by > quick-fedora-mirror. But I recall that ostree stuff was moved out of > the main repository mostly due to the huge file count. > > - J< > _______________________________________________ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org
