On Fri, Mar 23, 2018 at 04:01:50PM +, Ramsay Jones wrote:
> Not that it matters, but I assume this was something like:
>
> $ time (echo HEAD | git cat-file --batch-check="%(objectsize:disk)")
>
> ... and I suspect it was on the linux.git repo, yes?
Yes to both.
> If I do this on my bigge
On 23/03/18 05:50, Jeff King wrote:
> On Thu, Mar 22, 2018 at 10:46:09PM -0400, Jeff King wrote:
[snip]
> I was curious whether my hand-waving there was true. It turns out that
> it is: the radix sort has stayed about the same speed but the comparison
> sort has gotten even slower. Here are best-
On 23/03/18 02:46, Jeff King wrote:
> On Fri, Mar 23, 2018 at 01:28:12AM +, Ramsay Jones wrote:
>
>>> Of the used heap after your patches:
>>>
>>> - ~40% of that is from packlist_alloc()
>>> - ~17% goes to "struct object"
>>> - ~10% for the object.c hash table to store all the "struct obj
On Fri, Mar 23, 2018 at 3:46 AM, Jeff King wrote:
> On Fri, Mar 23, 2018 at 01:28:12AM +, Ramsay Jones wrote:
>
>> > Of the used heap after your patches:
>> >
>> > - ~40% of that is from packlist_alloc()
>> > - ~17% goes to "struct object"
>> > - ~10% for the object.c hash table to store al
On Thu, Mar 22, 2018 at 10:46:09PM -0400, Jeff King wrote:
> > which begs the question, how much slower would it be if we
> > replaced the radix-sort with an in-place sort (e.g. heapsort).
> >
> > I hacked up the patch below, just for fun. I don't have any
> > large repos (or enough disk space) t
On Fri, Mar 23, 2018 at 01:28:12AM +, Ramsay Jones wrote:
> > Of the used heap after your patches:
> >
> > - ~40% of that is from packlist_alloc()
> > - ~17% goes to "struct object"
> > - ~10% for the object.c hash table to store all the "struct object"
> > - ~7% goes to the delta cache
>
On 22/03/18 09:32, Jeff King wrote:
> On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote:
>
>>> I hate to be a wet blanket, but am I the only one who is wondering
>>> whether the tradeoffs is worth it? 8% memory reduction doesn't seem
>>> mind-bogglingly good,
>>
>> AEvar measured RSS. I
On Thu, Mar 22, 2018 at 12:52 PM, Jeff King wrote:
> On Thu, Mar 22, 2018 at 11:57:27AM +0100, Duy Nguyen wrote:
>
>> On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote:
>> > That would still mean you could get into a broken state for serving
>> > fetches, but you could at least get out of it by r
On Thu, Mar 22, 2018 at 11:57:27AM +0100, Duy Nguyen wrote:
> On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote:
> > That would still mean you could get into a broken state for serving
> > fetches, but you could at least get out of it by running "git repack".
>
> I was puzzled by this "broken st
On Thu, Mar 22, 2018 at 10:32 AM, Jeff King wrote:
> That would still mean you could get into a broken state for serving
> fetches, but you could at least get out of it by running "git repack".
I was puzzled by this "broken state" statement. But you were right! I
focused on the repack case and fo
On Thu, Mar 22, 2018 at 09:23:42AM +0100, Duy Nguyen wrote:
> > I wish you were right about the rarity, but it's unfortunately something
> > I have seen multiple times in the wild (and why I spent time optimizing
> > the many-packs case for pack-objects). Unfortunately I don't know how
> > often i
On Thu, Mar 22, 2018 at 05:32:12AM -0400, Jeff King wrote:
> So 27% of the total heap goes away if you switch to a separate rev-list.
> Though it's mostly just going to a different process, it does help peak
> because that process would have exited by the time we get to the
> revindex bits.
>
> I
On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote:
> > I hate to be a wet blanket, but am I the only one who is wondering
> > whether the tradeoffs is worth it? 8% memory reduction doesn't seem
> > mind-bogglingly good,
>
> AEvar measured RSS. If we count objects[] array alone, the savin
On Thu, Mar 22, 2018 at 9:07 AM, Jeff King wrote:
> On Wed, Mar 21, 2018 at 05:31:14PM +0100, Ævar Arnfjörð Bjarmason wrote:
>
>> > [...]Yes, having that many packs is insane, but that's going to be
>> > small consolation to somebody whose automated maintenance program now
>> > craps out at 16k pa
On Wed, Mar 21, 2018 at 05:31:14PM +0100, Ævar Arnfjörð Bjarmason wrote:
> > [...]Yes, having that many packs is insane, but that's going to be
> > small consolation to somebody whose automated maintenance program now
> > craps out at 16k packs, when it previously would have just worked to
> > fix
Duy Nguyen writes:
> And we could even do something like this to make custom builds
> easier. Some more gluing is needed so you can set this from config.mak
> but you get the idea. This removes all limits set by this
> series.
Yes, we _could_, but it would mean we would have many variants of
the
On Wed, Mar 21, 2018 at 5:53 PM, Junio C Hamano wrote:
> Ævar Arnfjörð Bjarmason writes:
>
>> That's going to be super rare (and probably nonexisting) edge case, but
>> (untested) I wonder if something like this on top would alleviate your
>> concerns, i.e. instead of dying we just take the first
Ævar Arnfjörð Bjarmason writes:
> That's going to be super rare (and probably nonexisting) edge case, but
> (untested) I wonder if something like this on top would alleviate your
> concerns, i.e. instead of dying we just take the first N packs up to our
> limit:
>
> diff --git a/builtin/pack-
On Wed, Mar 21, 2018 at 04:59:19PM +0100, Duy Nguyen wrote:
> About the 16k limit (and some other limits as well), I'm making these
> patches with the assumption that large scale deployment probably will
> go with custom builds anyway. Adjusting the limits back should be
> quite easy while we can s
On Wed, Mar 21 2018, Jeff King wrote:
> On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote:
>
>> v6 fixes the one optimization that I just couldn't get right, fixes
>> two off-by-one error messages and a couple commit message update
>> (biggest change is in 11/11 to record some
On Wed, Mar 21, 2018 at 5:17 PM, Ævar Arnfjörð Bjarmason
wrote:
>>> I think ultimately to work on low-memory machines we'll need a
>>> fundamentally different approach that scales with the objects since the
>>> last pack, and not with the complete history.
>>
>> Absolutely. Which is covered in a s
On Wed, Mar 21 2018, Duy Nguyen wrote:
> On Wed, Mar 21, 2018 at 9:24 AM, Jeff King wrote:
>> On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote:
>>
>>> v6 fixes the one optimization that I just couldn't get right, fixes
>>> two off-by-one error messages and a couple commit mes
On Wed, Mar 21, 2018 at 9:24 AM, Jeff King wrote:
> On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote:
>
>> v6 fixes the one optimization that I just couldn't get right, fixes
>> two off-by-one error messages and a couple commit message update
>> (biggest change is in 11/11 to r
On Sun, Mar 18, 2018 at 03:25:15PM +0100, Nguyễn Thái Ngọc Duy wrote:
> v6 fixes the one optimization that I just couldn't get right, fixes
> two off-by-one error messages and a couple commit message update
> (biggest change is in 11/11 to record some numbers from AEvar)
I was traveling during so
On Sun, Mar 18 2018, Nguyễn Thái Ngọc Duy jotted:
> v6 fixes the one optimization that I just couldn't get right, fixes
> two off-by-one error messages and a couple commit message update
> (biggest change is in 11/11 to record some numbers from AEvar)
Thanks, aside from the minor typo I just not
v6 fixes the one optimization that I just couldn't get right, fixes
two off-by-one error messages and a couple commit message update
(biggest change is in 11/11 to record some numbers from AEvar)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index fb2aba80bf..4406af640f 100644
--- a
26 matches
Mail list logo
