Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Fri, Mar 23, 2018 at 04:01:50PM +, Ramsay Jones wrote: > Not that it matters, but I assume this was something like: > > $ time (echo HEAD | git cat-file --batch-check="%(objectsize:disk)") > > ... and I suspect it was on the linux.git repo, yes? Yes to both. > If I do this on my

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On 23/03/18 05:50, Jeff King wrote: > On Thu, Mar 22, 2018 at 10:46:09PM -0400, Jeff King wrote: [snip] > I was curious whether my hand-waving there was true. It turns out that > it is: the radix sort has stayed about the same speed but the comparison > sort has gotten even slower. Here are

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On 23/03/18 02:46, Jeff King wrote: > On Fri, Mar 23, 2018 at 01:28:12AM +, Ramsay Jones wrote: > >>> Of the used heap after your patches: >>> >>> - ~40% of that is from packlist_alloc() >>> - ~17% goes to "struct object" >>> - ~10% for the object.c hash table to store all the "struct

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Fri, Mar 23, 2018 at 3:46 AM, Jeff King wrote: > On Fri, Mar 23, 2018 at 01:28:12AM +, Ramsay Jones wrote: > >> > Of the used heap after your patches: >> > >> > - ~40% of that is from packlist_alloc() >> > - ~17% goes to "struct object" >> > - ~10% for the object.c hash

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Thu, Mar 22, 2018 at 10:46:09PM -0400, Jeff King wrote: > > which begs the question, how much slower would it be if we > > replaced the radix-sort with an in-place sort (e.g. heapsort). > > > > I hacked up the patch below, just for fun. I don't have any > > large repos (or enough disk space)

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Fri, Mar 23, 2018 at 01:28:12AM +, Ramsay Jones wrote: > > Of the used heap after your patches: > > > > - ~40% of that is from packlist_alloc() > > - ~17% goes to "struct object" > > - ~10% for the object.c hash table to store all the "struct object" > > - ~7% goes to the delta cache

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On 22/03/18 09:32, Jeff King wrote: > On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote: > >>> I hate to be a wet blanket, but am I the only one who is wondering >>> whether the tradeoffs is worth it? 8% memory reduction doesn't seem >>> mind-bogglingly good, >> >> AEvar measured RSS.

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Thu, Mar 22, 2018 at 12:52 PM, Jeff King wrote: > On Thu, Mar 22, 2018 at 11:57:27AM +0100, Duy Nguyen wrote: > >> On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote: >> > That would still mean you could get into a broken state for serving >> > fetches, but you

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Thu, Mar 22, 2018 at 11:57:27AM +0100, Duy Nguyen wrote: > On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote: > > That would still mean you could get into a broken state for serving > > fetches, but you could at least get out of it by running "git repack". > > I was puzzled by

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote: > That would still mean you could get into a broken state for serving > fetches, but you could at least get out of it by running "git repack". I was puzzled by this "broken state" statement. But you were right! I focused on the

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Thu, Mar 22, 2018 at 09:23:42AM +0100, Duy Nguyen wrote: > > I wish you were right about the rarity, but it's unfortunately something > > I have seen multiple times in the wild (and why I spent time optimizing > > the many-packs case for pack-objects). Unfortunately I don't know how > > often

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Thu, Mar 22, 2018 at 05:32:12AM -0400, Jeff King wrote: > So 27% of the total heap goes away if you switch to a separate rev-list. > Though it's mostly just going to a different process, it does help peak > because that process would have exited by the time we get to the > revindex bits. > >

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote: > > I hate to be a wet blanket, but am I the only one who is wondering > > whether the tradeoffs is worth it? 8% memory reduction doesn't seem > > mind-bogglingly good, > > AEvar measured RSS. If we count objects[] array alone, the

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Thu, Mar 22, 2018 at 9:07 AM, Jeff King wrote: > On Wed, Mar 21, 2018 at 05:31:14PM +0100, Ævar Arnfjörð Bjarmason wrote: > >> > [...]Yes, having that many packs is insane, but that's going to be >> > small consolation to somebody whose automated maintenance program now >> >

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Wed, Mar 21, 2018 at 05:31:14PM +0100, Ævar Arnfjörð Bjarmason wrote: > > [...]Yes, having that many packs is insane, but that's going to be > > small consolation to somebody whose automated maintenance program now > > craps out at 16k packs, when it previously would have just worked to > >

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

Duy Nguyen writes: > And we could even do something like this to make custom builds > easier. Some more gluing is needed so you can set this from config.mak > but you get the idea. This removes all limits set by this > series. Yes, we _could_, but it would mean we would have

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Wed, Mar 21, 2018 at 5:53 PM, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason writes: > >> That's going to be super rare (and probably nonexisting) edge case, but >> (untested) I wonder if something like this on top would alleviate your >> concerns, i.e.

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

Ævar Arnfjörð Bjarmason writes: > That's going to be super rare (and probably nonexisting) edge case, but > (untested) I wonder if something like this on top would alleviate your > concerns, i.e. instead of dying we just take the first N packs up to our > limit: > > diff

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote: > About the 16k limit (and some other limits as well), I'm making these > patches with the assumption that large scale deployment probably will > go with custom builds anyway. Adjusting the limits back should be > quite easy while we can

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Wed, Mar 21 2018, Jeff King wrote: > On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote: > >> v6 fixes the one optimization that I just couldn't get right, fixes >> two off-by-one error messages and a couple commit message update >> (biggest change is in 11/11 to record some

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Wed, Mar 21, 2018 at 5:17 PM, Ævar Arnfjörð Bjarmason wrote: >>> I think ultimately to work on low-memory machines we'll need a >>> fundamentally different approach that scales with the objects since the >>> last pack, and not with the complete history. >> >> Absolutely.

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Wed, Mar 21 2018, Duy Nguyen wrote: > On Wed, Mar 21, 2018 at 9:24 AM, Jeff King wrote: >> On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote: >> >>> v6 fixes the one optimization that I just couldn't get right, fixes >>> two off-by-one error messages and a

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Wed, Mar 21, 2018 at 9:24 AM, Jeff King wrote: > On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote: > >> v6 fixes the one optimization that I just couldn't get right, fixes >> two off-by-one error messages and a couple commit message update >> (biggest change

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote: > v6 fixes the one optimization that I just couldn't get right, fixes > two off-by-one error messages and a couple commit message update > (biggest change is in 11/11 to record some numbers from AEvar) I was traveling during

Re: [PATCH v6 00/11] nd/pack-objects-pack-struct updates

On Sun, Mar 18 2018, Nguyễn Thái Ngọc Duy jotted: > v6 fixes the one optimization that I just couldn't get right, fixes > two off-by-one error messages and a couple commit message update > (biggest change is in 11/11 to record some numbers from AEvar) Thanks, aside from the minor typo I just

[PATCH v6 00/11] nd/pack-objects-pack-struct updates

v6 fixes the one optimization that I just couldn't get right, fixes two off-by-one error messages and a couple commit message update (biggest change is in 11/11 to record some numbers from AEvar) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index fb2aba80bf..4406af640f 100644 ---