Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-23 Thread Jeff King
On Sat, Jul 21, 2018 at 06:23:32AM +0200, Duy Nguyen wrote: > > I'm not sure I completely agree with this. While 4GB deltas should be > > pretty rare, the nice thing about 64-bit is that you never have to even > > think about whether the variable is large enough. I think the 2^32 > > limitations o

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-23 Thread Duy Nguyen
On Mon, Jul 23, 2018 at 2:34 PM Elijah Newren wrote: > This patch provides a stop-gap improvement for maint that increases the > delta size back up to 32 bits (still an improvement over the 64 bit size > of the original). The "still an improvement" is actually not true. Due to padding and stuff,

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-23 Thread Elijah Newren
On Fri, Jul 20, 2018 at 10:43 AM, Elijah Newren wrote: > On Fri, Jul 20, 2018 at 8:39 AM, Nguyễn Thái Ngọc Duy > wrote: >> If we want a quick fix for 2.18.1. I suggest bumping up >> OE_DELTA_SIZE_BITS like Elijah did in his second option. I don't >> think it's safe to fast track this patch.

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-21 Thread Duy Nguyen
On Sun, Jul 22, 2018 at 8:22 AM Elijah Newren wrote: > > On Fri, Jul 20, 2018 at 9:47 PM, Duy Nguyen wrote: > > On Fri, Jul 20, 2018 at 10:43:25AM -0700, Elijah Newren wrote: > >> Out of curiosity, would it be possible to use the delta_size_ field > >> for deltas that are small enough, and only u

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-21 Thread Elijah Newren
On Fri, Jul 20, 2018 at 9:47 PM, Duy Nguyen wrote: > On Fri, Jul 20, 2018 at 10:43:25AM -0700, Elijah Newren wrote: >> Out of curiosity, would it be possible to use the delta_size_ field >> for deltas that are small enough, and only use an external data >> structure (perhaps a hash rather than an

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-21 Thread Duy Nguyen
On Sat, Jul 21, 2018 at 8:56 AM Elijah Newren wrote: > > -- 8< -- > ... > > diff --git a/pack-objects.h b/pack-objects.h > > index 9f977ae800..11890e7217 100644 > > --- a/pack-objects.h > > +++ b/pack-objects.h > > @@ -15,7 +15,7 @@ > ... > > @@ -353,37 +354,26 @@ static inline void oe_set_size(st

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-21 Thread Duy Nguyen
On Sat, Jul 21, 2018 at 6:07 AM Duy Nguyen wrote: > > However, I'm just concentrating on a beefy machine; it may be that v6 > > drastically outperforms v2 on weaker hardware? Can others measure a > > lower memory usage for v6 than v2? > > I'll try it with massif on linux.git, but this is a very w

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-20 Thread Elijah Newren
On Fri, Jul 20, 2018 at 9:47 PM, Duy Nguyen wrote: > On Fri, Jul 20, 2018 at 10:43:25AM -0700, Elijah Newren wrote: >> Out of curiosity, would it be possible to use the delta_size_ field >> for deltas that are small enough, and only use an external data >> structure (perhaps a hash rather than an

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-20 Thread Duy Nguyen
On Fri, Jul 20, 2018 at 10:43:25AM -0700, Elijah Newren wrote: > > This patch provides a better fallback that is > > > > - cheaper in terms of cpu and io because we won't have to read > > existing pack files as much > > > > - better in terms of pack size because the pack heuristics is back to > >

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-20 Thread Duy Nguyen
On Fri, Jul 20, 2018 at 7:40 PM Jeff King wrote: > > On Fri, Jul 20, 2018 at 05:39:43PM +0200, Nguyễn Thái Ngọc Duy wrote: > > > Let's start with some background about oe_delta_size() and > > oe_set_delta_size(). If you already know, skip the next paragraph. > > > > These two are added in 0aca34e8

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-20 Thread Duy Nguyen
On Sat, Jul 21, 2018 at 1:52 AM Elijah Newren wrote: > > On Fri, Jul 20, 2018 at 10:43 AM, Elijah Newren wrote: > > On Fri, Jul 20, 2018 at 8:39 AM, Nguyễn Thái Ngọc Duy > > wrote: > > >> This patch provides a better fallback that is > >> > >> - cheaper in terms of cpu and io because we won't h

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-20 Thread Elijah Newren
On Fri, Jul 20, 2018 at 10:43 AM, Elijah Newren wrote: > On Fri, Jul 20, 2018 at 8:39 AM, Nguyễn Thái Ngọc Duy > wrote: >> This patch provides a better fallback that is >> >> - cheaper in terms of cpu and io because we won't have to read >> existing pack files as much >> >> - better in terms

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-20 Thread Elijah Newren
On Fri, Jul 20, 2018 at 8:39 AM, Nguyễn Thái Ngọc Duy wrote: > Let's start with some background about oe_delta_size() and > oe_set_delta_size(). If you already know, skip the next paragraph. > > These two are added in 0aca34e826 (pack-objects: shrink delta_size > field in struct object_entry - 201

Re: [PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-20 Thread Jeff King
On Fri, Jul 20, 2018 at 05:39:43PM +0200, Nguyễn Thái Ngọc Duy wrote: > Let's start with some background about oe_delta_size() and > oe_set_delta_size(). If you already know, skip the next paragraph. > > These two are added in 0aca34e826 (pack-objects: shrink delta_size > field in struct object_e

[PATCH] pack-objects: fix performance issues on packing large deltas

2018-07-20 Thread Nguyễn Thái Ngọc Duy
Let's start with some background about oe_delta_size() and oe_set_delta_size(). If you already know, skip the next paragraph. These two are added in 0aca34e826 (pack-objects: shrink delta_size field in struct object_entry - 2018-04-14) to help reduce 'struct object_entry' size. The delta size fiel