On Tue, Jun 10, 2014 at 6:59 PM, Patrick Lauer <patr...@gentoo.org> wrote: > The first migration attempt failed after consuming nearly 100GB of RAM! > When it did work it took obscene amounts of time, and the result was > unusably large (e.g. initial checkout would take 16GB RAM on the server, > thus not allowing a few hundred devs to do checkouts the same day).
Well, effort required to do the migration isn't too big of a deal since it is a one-time event. As long as it can be done correctly, it is good enough. Ferringb was routinely crunching it in something like an hour at the end of last year, granted on a system with an obscene number of cores. > The current state is almost usable, but it is still obscenely slow (e.g. > initial clone taking ~10 CPU-minutes just to figure out what to do), but > we can just throw more hardware at it. > (10 minutes @ 3.6Ghz, so on my notebook it'll take about 4h to just > clone the friggin repository. Too awesome!) Hmm, can't say I've tried it on a variety of systems but I never had trouble cloning it. You are cloning from a bundle right, and not just hitting the server for the whole tree, right? The thought is to have a hook that will block anybody from actually doing a full clone from the server. Instead users would clone from a bundle (either full or with shallow history) and then do pulls from the server vs that. The bundle would just be distributed on the mirrors. > > Another part: Few people thought about the difficult problems in the > migration. For example the rsync mirrors, which will still be used - the > scripts that generate those will need to be changed, tested, and then > replaced if a migration ever happens. Well, I wouldn't say that few have thought about it. The bigger issue is that I don't think any of the existing code is published anywhere, so it is a bit hard to contribute to it. If that isn't the case I'm happy to be corrected. But, yes, that is the bigger block right now. Rich
