On Mon, 26 Jun 2023, Mathias De Mar? wrote: > Hello, > > > > I?d like to give a little bit of additional information about a Mercurial > feature that might provide some help here. My apologies if this is old news > and doesn't add anything useful. > > (Small context: I?m a Mercurial contributor and I maintain a set of > repositories at our company that are cloned a lot at the same point in time) > > > > By default, when you do a clone, Mercurial will take the repository, > compress it and send it across the wire to the client. > > I simulated similar behaviour for the gmp repository by creating a bundle: > > - it takes about 8 seconds (so approximately 1 core occupied for < 8 > seconds, as there is a bit of overhead involved) > - it generates a bundle of about 14 MB > > That?s not much, but it of course becomes problematic when there are 700 > clones at nearly the exact same moment, like you experienced. > > > > I cannot help in reducing the bandwidth, but assuming the cpu usage is the > problem, ?clonebundles? would help resolve the issue. Clonebundles are a > Mercurial feature that allows the Mercurial server to reuse an existing > bundle instead of generating a new one for every user. They would basically > get rid of the cpu issue, the server simply points to the pre-generated > bundle and does not actually need to do any work. More context from Gregory > Szorc, who initially implemented this feature: > https://gregoryszorc.com/blog/2015/05/29/faster-cloning-from-hg.mozilla.org-with-server-provided-bundles/ > > > > To use clonebundles for a repository (let me know if there?s anything > missing, I have been known to make mistakes): > > 1. Make sure you?re running a fairly recent Mercurial version. > 2. Put up a cronjob that generates weekly bundles of your repository (hg > bundle ?all /path/to/gmp.bundle). > 3. Make sure the bundle is placed in a location accessible to clients, > for example: https://gmplib.org/clonebundles/gmp.bundle. > 4. In your repository, create a file .hg/clonebundles.manifest that > refers to the bundle. > To check what the ?BUNDLESPEC? parameter should be, you can run: hg > debugbundle ?spec /path/to/gmp.bundle > Contents could for example be: > https://gmplib.org/clonebundles/gmp.bundle BUNDLESPEC=bzip2-v2 > 5. In your hgrc (either $HOME/.hgrc or /path/to/repository/.hg/hgrc), > you will need to enable clonebundles: > [extensions] > clonebundles = > > > The next version of Mercurial (6.5) will contain a feature (implemented by > Mercurial developer Pierre-Yves David) to auto-generate clonebundles when > changes are pushed to a repository. This means you don't need the above > steps, you can purely use configuration options, as described in the help > of the clonebundles extension: > https://foss.heptapod.net/mercurial/mercurial-devel/-/blob/branch/default/hgext/clonebundles.py > > Some additional notes: > 1. Clonebundles reduce server load massively, in exchange for additional > disk space. Make sure you have enough if you want to enable this. > 2. Mercurial clients don't need to do anything to enable this feature, it > only needs to be enabled on the server side. > 3. I think on the Mercurial development side, we should try to make this > feature more visible/known, as it could make the situation a lot better for > people hosting Mercurial repositories.
I think it would catch 99% of the cases when mercurial would cache the bundle from the "HEAD" (I don't know mercurial or bundles aka whether each branch or revision would create a different bundle than the complete repository state) and do that on-demand, so not create a bundle for each commit. Richard. -- Richard Biener <rguent...@suse.de> SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg, Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman; HRB 36809 (AG Nuernberg) _______________________________________________ gmp-devel mailing list gmp-devel@gmplib.org https://gmplib.org/mailman/listinfo/gmp-devel
