RE: [PATCH] [RFC] Making use of bitmaps for thin objects

2013-12-22 Thread Ben Maurer
One issue with this approach is that it seems git-pack-index doesn't perform as well with thin packs. git-index-pack uses a multi-threaded approach to resolving the deltas. However, the multithreading only works on deltas that are exclusively in the pack. After the multi-threaded phase, it incre

Re: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-06 Thread Jeff King
On Sun, Dec 22, 2013 at 09:55:23PM +, Ben Maurer wrote: > One issue with this approach is that it seems git-pack-index doesn't > perform as well with thin packs. git-index-pack uses a multi-threaded > approach to resolving the deltas. However, the multithreading only > works on deltas that are

Re: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-06 Thread Jeff King
On Sun, Dec 22, 2013 at 11:47:34AM -0800, Ben Maurer wrote: > Jeff King's bitmap branch appears to give a very substantial speedup. > After applying this branch, the "counting objects" phase is basically > free. However, I found that the compression phase still takes a > non-trivial amount of time

Re: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-06 Thread Junio C Hamano
Jeff King writes: > We could probably teach index-pack an "--assume-refs-are-thin" > option to optimize for this case, and have fetch-pack/receive-pack pass > it whenever they know that delta-base-offset was negotiated. I thought the existing negotiation merely means "I understand offset encoded

Re: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-06 Thread Jeff King
On Mon, Jan 06, 2014 at 08:37:49AM -0800, Junio C Hamano wrote: > Jeff King writes: > > > We could probably teach index-pack an "--assume-refs-are-thin" > > option to optimize for this case, and have fetch-pack/receive-pack pass > > it whenever they know that delta-base-offset was negotiated. >

Re: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-06 Thread Jeff King
On Mon, Jan 06, 2014 at 09:57:23AM -0500, Jeff King wrote: > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > index c733379..0cff874 100644 > --- a/builtin/pack-objects.c > +++ b/builtin/pack-objects.c > @@ -1402,6 +1402,19 @@ static void check_object(struct object_entry *entry) >

RE: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-06 Thread Ben Maurer
> Sorry for the slow reply; I've been on vacation. No worries. > When you build your bitmaps, do you set the pack.writeBitmapHashCache > option? We found that it makes a significant difference during the > compression phase, as otherwise git attempts deltas between random files > based on size. H

Re: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-06 Thread Jeff King
On Mon, Jan 06, 2014 at 09:15:04PM +, Ben Maurer wrote: > > Let me know how this patch does for you. My testing has been fairly > > limited so far. > > This patch looks like a much cleaner version :-). Works well for me, > but my test setup may not be great since I didn't get any errors from

RE: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-06 Thread Ben Maurer
58825), 81.35 MiB | 0 bytes/s, done. Total 158825 (delta 135730), reused 158825 (delta 141479) real 2.70 user 2.28 sys 0.65 From: Jeff King [p...@peff.net] Sent: Monday, January 06, 2014 1:57 PM To: Ben Maurer Cc: git@vger.kernel.org Subject: Re: [PATCH] [

Re: [PATCH] [RFC] Making use of bitmaps for thin objects

2014-01-07 Thread Jeff King
On Mon, Jan 06, 2014 at 10:14:30PM +, Ben Maurer wrote: > It looks like for my repo the size win wasn't as big (~10%). Is it > possible that with the kernel test you got extremely lucky and there > was some huge binary blob that thin packing turned into a tiny delta? I don't think so. When I