Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-29 Thread Duy Nguyen
On Fri, Jul 27, 2018 at 07:52:33PM +0200, Duy Nguyen wrote: > Just FYI I'm still trying to reduce execution time further and this > change happens to half traverse_trees() time (which is a huge deal) > > diff --git a/unpack-trees.c b/unpack-trees.c > index f0be9f298d..a2e63ad5bf 100644 > ---

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-27 Thread Duy Nguyen
On Fri, Jul 27, 2018 at 6:22 PM Ben Peart wrote: > > > > On 7/27/2018 11:42 AM, Duy Nguyen wrote: > > On Thu, Jul 26, 2018 at 12:40:05PM -0700, Junio C Hamano wrote: > >> Duy Nguyen writes: > >> > >>> I'm excited so I decided to try out anyway. This is what I've come up > >>> with. Switching

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-27 Thread Duy Nguyen
On Fri, Jul 27, 2018 at 7:14 PM Junio C Hamano wrote: > > Duy Nguyen writes: > > > diff --git a/unpack-trees.c b/unpack-trees.c > > index 66741130ae..9c791b55b2 100644 > > --- a/unpack-trees.c > > +++ b/unpack-trees.c > > @@ -642,6 +642,110 @@ static inline int are_same_oid(struct name_entry >

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-27 Thread Junio C Hamano
Duy Nguyen writes: > diff --git a/unpack-trees.c b/unpack-trees.c > index 66741130ae..9c791b55b2 100644 > --- a/unpack-trees.c > +++ b/unpack-trees.c > @@ -642,6 +642,110 @@ static inline int are_same_oid(struct name_entry > *name_j, struct name_entry *nam > return name_j->oid &&

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-27 Thread Ben Peart
On 7/27/2018 11:42 AM, Duy Nguyen wrote: On Thu, Jul 26, 2018 at 12:40:05PM -0700, Junio C Hamano wrote: Duy Nguyen writes: I'm excited so I decided to try out anyway. This is what I've come up with. Switching trees on git.git shows it could skip plenty entries, so promising. It's ugly

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-27 Thread Ben Peart
On 7/26/2018 3:40 PM, Junio C Hamano wrote: Duy Nguyen writes: I'm excited so I decided to try out anyway. This is what I've come up with. Switching trees on git.git shows it could skip plenty entries, so promising. It's ugly and it fails at t6020 though, there's still work ahead. But I

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-27 Thread Duy Nguyen
On Thu, Jul 26, 2018 at 12:40:05PM -0700, Junio C Hamano wrote: > Duy Nguyen writes: > > > I'm excited so I decided to try out anyway. This is what I've come up > > with. Switching trees on git.git shows it could skip plenty entries, > > so promising. It's ugly and it fails at t6020 though,

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-26 Thread Junio C Hamano
Duy Nguyen writes: > I'm excited so I decided to try out anyway. This is what I've come up > with. Switching trees on git.git shows it could skip plenty entries, > so promising. It's ugly and it fails at t6020 though, there's still > work ahead. But I think it'll stop here. We are extremely

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-26 Thread Duy Nguyen
On Wed, Jul 25, 2018 at 10:56 PM Ben Peart wrote: > > > > On 7/24/2018 11:33 AM, Duy Nguyen wrote: > > On Tue, Jul 24, 2018 at 6:20 AM Jeff King wrote: > >> At least that's my view of it. unpack_trees() has always been a > >> terrifying beast that I've avoided looking too closely at. > > > > /me

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-26 Thread Duy Nguyen
On Thu, Jul 26, 2018 at 07:30:20AM +0200, Duy Nguyen wrote: > Let's get back to two-way merge. I suggest you read the two-way merge > in git-read-tree.txt. That table could give you a pretty good idea > what's going on. twoway_merge() will be given a tuple of three entries > (I, H, M) of the same

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-25 Thread Duy Nguyen
On Wed, Jul 25, 2018 at 10:56 PM Ben Peart wrote: > I'm still very new to this part of the code so am trying to figure out > what you're suggesting. I've read your description a few times and what > I'm getting out of it is that with some additional checks (ie verify > it's a twoway_merge,

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-25 Thread Ben Peart
On 7/24/2018 11:33 AM, Duy Nguyen wrote: On Tue, Jul 24, 2018 at 6:20 AM Jeff King wrote: At least that's my view of it. unpack_trees() has always been a terrifying beast that I've avoided looking too closely at. /me nods on the terrifying part. After a quick look at the code, the only

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-25 Thread Ben Peart
On 7/24/2018 11:13 AM, Duy Nguyen wrote: On Mon, Jul 23, 2018 at 04:51:38PM -0400, Ben Peart wrote: What's the current state of the index before this checkout? This was after running "git checkout" multiple times so there was really nothing for git to do. Hmm.. this means cache-tree is

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-24 Thread Jeff King
On Tue, Jul 24, 2018 at 05:13:36PM +0200, Duy Nguyen wrote: > I guess you have the starting points you need after Jeff's and Junio's > explanation (and it would be great if cache-tree could actually be for > for this two-way merge). But to make it easier for new people in > future, maybe we

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-24 Thread Duy Nguyen
On Tue, Jul 24, 2018 at 6:20 AM Jeff King wrote: > At least that's my view of it. unpack_trees() has always been a > terrifying beast that I've avoided looking too closely at. /me nods on the terrifying part. > > After a quick look at the code, the only place I can find that tries to use > >

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-24 Thread Duy Nguyen
On Mon, Jul 23, 2018 at 04:51:38PM -0400, Ben Peart wrote: > >>> What's the current state of the index before this checkout? > >> > >> This was after running "git checkout" multiple times so there was really > >> nothing for git to do. > > > > Hmm.. this means cache-tree is fully valid, unless

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-23 Thread Junio C Hamano
Ben Peart writes: >> Hmm.. this means cache-tree is fully valid, unless you have changes in >> index. We're quite aggressive in repairing cache-tree since aecf567cbf >> (cache-tree: create/update cache-tree on checkout - 2014-07-05). If we >> have very good cache-tree records and still spend 33s

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-23 Thread Jeff King
On Mon, Jul 23, 2018 at 07:03:16PM +0200, Duy Nguyen wrote: > > > Try bumping core.deltaBaseCacheLimit to see if that has any impact. It's > > > 96MB by default. > > > > > > There may also be some possible work in making it more aggressive about > > > storing the intermediate results. I seem to

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-23 Thread Jeff King
On Mon, Jul 23, 2018 at 04:51:38PM -0400, Ben Peart wrote: > > Hmm.. this means cache-tree is fully valid, unless you have changes in > > index. We're quite aggressive in repairing cache-tree since aecf567cbf > > (cache-tree: create/update cache-tree on checkout - 2014-07-05). If we > > have very

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-23 Thread Ben Peart
On 7/23/2018 1:03 PM, Duy Nguyen wrote: On Mon, Jul 23, 2018 at 5:50 PM Ben Peart wrote: Anyway, on to the actual discussion: Here is a checkout command with tracing turned on to demonstrate where the time is spent. Note, this is somewhat of a �best case� as I�m simply checking out the

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-23 Thread Duy Nguyen
On Mon, Jul 23, 2018 at 5:50 PM Ben Peart wrote: > > Anyway, on to the actual discussion: > > > >> Here is a checkout command with tracing turned on to demonstrate where the > >> time is spent. Note, this is somewhat of a �best case� as I�m simply > >> checking out the current commit: > >> > >>

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-23 Thread Ben Peart
On 7/18/2018 5:34 PM, Jeff King wrote: On Wed, Jul 18, 2018 at 08:45:14PM +, Ben Peart wrote: When working directories get big, checkout times start to suffer. Even with GVFS virtualization (which limits git to only having to update those files that have been changed locally) we�re

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-18 Thread Jeff King
On Wed, Jul 18, 2018 at 08:45:14PM +, Ben Peart wrote: > When working directories get big, checkout times start to suffer. Even with > GVFS virtualization (which limits git to only having to update those files > that have been changed locally) we�re seeing P50 times for checkout of 31 >

Re: [PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-18 Thread Stefan Beller
> don�t The encoding seems to be broken somehow (also on) https://public-inbox.org/git/20180718204458.20936-1-benpe...@microsoft.com/ > When I brought up this idea with some other git contributors they mentioned > that multi threading unpack_trees() had been discussed a few years ago on

[PATCH v1 0/3] [RFC] Speeding up checkout (and merge, rebase, etc)

2018-07-18 Thread Ben Peart
When working directories get big, checkout times start to suffer. Even with GVFS virtualization (which limits git to only having to update those files that have been changed locally) we�re seeing P50 times for checkout of 31 seconds and the P80 time is 43 seconds. Here is a checkout command with