[PATCH/RFC v3 00/12] Reduce pack-objects memory footprint

v3 cleans up a bit to avoid the horrible macros that v2 adds, and adds some more documentation for this struct since it's getting harder to just look at the struct and see what field is related to what. v3 also adds three more patches and reduces another 16 bytes (total struct reduction now is

Re: [PATCH/RFC v2 0/9] Reduce pack-objects memory footprint

On Sat, Mar 03, 2018 at 09:46:57AM +0700, Nguyễn Thái Ngọc Duy wrote: > The array of object_entry in pack-objects can take a lot of memory > when pack-objects is run in "pack everything" mode. On linux-2.6.git, > this array alone takes roughly 800MB. > > This series reorders some fields and

[PATCH/RFC v2 0/9] Reduce pack-objects memory footprint

The array of object_entry in pack-objects can take a lot of memory when pack-objects is run in "pack everything" mode. On linux-2.6.git, this array alone takes roughly 800MB. This series reorders some fields and reduces field size... to keep this struct smaller. Its size goes from 136 bytes to 96

Re: Reduce pack-objects memory footprint?

" on my poor laptop >> > consumes 1.7G out of 4G RAM, pushing lots of data to swap and making >> > all apps nearly unusuable (granted the problem is partly Linux I/O >> > scheduler too). So I wonder if we can reduce pack-objects memory >> > footprint a bit. &g

Re: [PATCH 00/11] Reduce pack-objects memory footprint

On Fri, Mar 02, 2018 at 07:14:01AM +0700, Duy Nguyen wrote: > > We have a big repo, and this gets repacked on 6-8GB of memory on dev > > KVMs, so we're under a fair bit of memory pressure. git-gc slows things > > down a lot. > > > > It would be really nice to have something that made it use

Re: Reduce pack-objects memory footprint?

" on my poor laptop >> > consumes 1.7G out of 4G RAM, pushing lots of data to swap and making >> > all apps nearly unusuable (granted the problem is partly Linux I/O >> > scheduler too). So I wonder if we can reduce pack-objects memory >> > footprint a bit. &g

Re: Reduce pack-objects memory footprint?

to swap and making > > all apps nearly unusuable (granted the problem is partly Linux I/O > > scheduler too). So I wonder if we can reduce pack-objects memory > > footprint a bit. > > Next low hanging fruit item: > > struct revindex_entry { > off_t offset; >

Re: Reduce pack-objects memory footprint?

Duy Nguyen wrote: > struct revindex_entry { > off_t offset; > unsigned int nr; > }; > > We need on entry per object, so 6.5M objects * 16 bytes = 104 MB. If > we break this struct apart and store two arrays of offset and nr in > struct packed_git, we save 4

Re: Reduce pack-objects memory footprint?

Linux I/O > scheduler too). So I wonder if we can reduce pack-objects memory > footprint a bit. Next low hanging fruit item: struct revindex_entry { off_t offset; unsigned int nr; }; We need on entry per object, so 6.5M objects * 16 bytes = 104 MB. If we break this st

Re: [PATCH 00/11] Reduce pack-objects memory footprint

On Thu, Mar 1, 2018 at 8:33 PM, Ævar Arnfjörð Bjarmason wrote: > > On Thu, Mar 01 2018, Nguyễn Thái Ngọc Duy jotted: > >> The array of object_entry in pack-objects can take a lot of memory >> when pack-objects is run in "pack everything" mode. On linux-2.6.git, >> this array

Re: [PATCH 00/11] Reduce pack-objects memory footprint

On Thu, Mar 01 2018, Nguyễn Thái Ngọc Duy jotted: > The array of object_entry in pack-objects can take a lot of memory > when pack-objects is run in "pack everything" mode. On linux-2.6.git, > this array alone takes roughly 800MB. > > This series reorders some fields and reduces field size... to

[PATCH 00/11] Reduce pack-objects memory footprint

The array of object_entry in pack-objects can take a lot of memory when pack-objects is run in "pack everything" mode. On linux-2.6.git, this array alone takes roughly 800MB. This series reorders some fields and reduces field size... to keep this struct smaller. Its size goes from 136 bytes to 96

Re: Reduce pack-objects memory footprint?

On Wed, Feb 28, 2018 at 06:22:33PM +, Eric Wong wrote: > Duy Nguyen wrote: > > which saves 12 bytes (or another 74 MB). 222 MB total is plenty of > > space to keep some file cache from being evicted. > > Nice! I can definitely benefit from lower memory usage when >

Re: Reduce pack-objects memory footprint?

Duy Nguyen wrote: > which saves 12 bytes (or another 74 MB). 222 MB total is plenty of > space to keep some file cache from being evicted. Nice! I can definitely benefit from lower memory usage when packing. Fwiw, I use pahole with other projects to help find packing

Re: Reduce pack-objects memory footprint?

On Wed, Feb 28, 2018 at 6:11 PM, Jeff King wrote: >> > The torvalds/linux fork network has ~23 million objects, >> > so it's probably 7-8 GB of book-keeping. Which is gross, but 64GB in a >> > server isn't uncommon these days. >> >> I wonder if we could just do book keeping for

Re: Reduce pack-objects memory footprint?

On Wed, Feb 28, 2018 at 05:58:50PM +0700, Duy Nguyen wrote: > > Yeah, the per object memory footprint is not great. Around 100 million > > objects it becomes pretty ridiculous. I started to dig into it a year or > > three ago when I saw such a case, but it turned out to be something that > > we

Re: Reduce pack-objects memory footprint?

swap and making >> all apps nearly unusuable (granted the problem is partly Linux I/O >> scheduler too). So I wonder if we can reduce pack-objects memory >> footprint a bit. > > Yeah, the per object memory footprint is not great. Around 100 million > objects it becomes pret

Re: Reduce pack-objects memory footprint?

; scheduler too). So I wonder if we can reduce pack-objects memory > footprint a bit. Yeah, the per object memory footprint is not great. Around 100 million objects it becomes pretty ridiculous. I started to dig into it a year or three ago when I saw such a case, but it turned out to be some

Reduce pack-objects memory footprint?

linux-2.6.git current has 6483999 objects. "git gc" on my poor laptop consumes 1.7G out of 4G RAM, pushing lots of data to swap and making all apps nearly unusuable (granted the problem is partly Linux I/O scheduler too). So I wonder if we can reduce pack-objects memory footp