Re: Make `git fetch --all` parallel?

On Tue, Oct 11, 2016 at 6:52 PM, Jeff King wrote: > On Tue, Oct 11, 2016 at 09:34:28PM -0400, Jeff King wrote: > >> > Ok, time to present data... Let's assume a degenerate case first: >> > "up-to-date with all remotes" because that is easy to reproduce. >> > >> > I have 14 remotes currently: >> >

Re: Make `git fetch --all` parallel?

On Tue, Oct 11, 2016 at 09:34:28PM -0400, Jeff King wrote: > > Ok, time to present data... Let's assume a degenerate case first: > > "up-to-date with all remotes" because that is easy to reproduce. > > > > I have 14 remotes currently: > > > > $ time git fetch --all > > real 0m18.016s > > user 0m

Re: Make `git fetch --all` parallel?

On Tue, Oct 11, 2016 at 04:18:15PM -0700, Stefan Beller wrote: > >> At the very least we would need a similar thing as Jeff recently sent for > >> the > >> push case with objects quarantined and then made available in one go? > > > > I don't think so. The object database is perfectly happy with m

Re: Make `git fetch --all` parallel?

On Tue, Oct 11, 2016 at 3:59 PM, Jeff King wrote: > On Tue, Oct 11, 2016 at 03:50:36PM -0700, Stefan Beller wrote: > >> I agree. Though even for implementing the "dumb" case of fetching >> objects twice we'd have to take care of some racing issues, I would assume. >> >> Why did you put a "sleep 2"

Re: Make `git fetch --all` parallel?

On Wed, Oct 12, 2016 at 12:59 AM, Jeff King wrote: > I'm not altogether convinced that parallel fetch would be that much > faster, though. I have local aliases to use GNU parallel for stuff like this, on my git.git which has accumulated 17 remotes: $ time parallel -j1 'git fetch {}' ::: $(g

Re: Make `git fetch --all` parallel?

On Tue, Oct 11, 2016 at 03:50:36PM -0700, Stefan Beller wrote: > I agree. Though even for implementing the "dumb" case of fetching > objects twice we'd have to take care of some racing issues, I would assume. > > Why did you put a "sleep 2" below? > * a slow start to better spread load locally? (

Re: Make `git fetch --all` parallel?

> I dunno, if documented though. http://stackoverflow.com/questions/26373995/how-to-control-the-order-of-fetching-when-fetching-all-remotes-by-git-fetch-al We do not give promises about the order of --all (checked with our documentation as well), however there seems to be a grouping scheme for re

Re: Make `git fetch --all` parallel?

Stefan Beller writes: > Why did you put a "sleep 2" below? The assumption was to fetch from faster and near the center of the project universe early, so by giving them head-start, fetches that start in later rounds may have chance to see newly updated remote tracking refs when telling the poorer

Re: Make `git fetch --all` parallel?

On Tue, Oct 11, 2016 at 3:37 PM, Junio C Hamano wrote: > Stefan Beller writes: > >> So I do think it would be much faster, but I also think patches for this >> would >> require some thought and a lot of refactoring of the fetch code. >> ... >> During the negotiation phase a client would have to

Re: Make `git fetch --all` parallel?

Stefan Beller writes: > So I do think it would be much faster, but I also think patches for this would > require some thought and a lot of refactoring of the fetch code. > ... > During the negotiation phase a client would have to be able to change its > mind (add more "haves", or in case of the p

Re: Make `git fetch --all` parallel?

On Tue, Oct 11, 2016 at 1:12 PM, Ram Rachum wrote: > Hi everyone! > > I have a repo that has a bunch of different remotes, and I noticed > slowness when doing `git fetch --all`. Is it currently made > sequentially? Do you think that maybe it could be done in parallel so > it could be much faster?

Make `git fetch --all` parallel?

Hi everyone! I have a repo that has a bunch of different remotes, and I noticed slowness when doing `git fetch --all`. Is it currently made sequentially? Do you think that maybe it could be done in parallel so it could be much faster? Thanks, Ram.