On Wed, Feb 28 2018, Duy Nguyen jotted:
> linux-2.6.git current has 6483999 objects. "git gc" on my poor laptop
> consumes 1.7G out of 4G RAM, pushing lots of data to swap and making
> all apps nearly unusuable (granted the problem is partly Linux I/O
> scheduler too). So I wonder if we can reduce pack-objects memory
> footprint a bit.
>
> This demonstration patch (probably breaks some tests) would reduce the
> size of struct object_entry from from 136 down to 112 bytes on
> x86-64. There are 6483999 of these objects, so the saving is 17% or
> 148 MB.
Splitting this off into its own thread. Aside from the improvements in
your repack memory reduction (20180317141033.21545-1-pclo...@gmail.com)
and gc config (20180316192745.19557-1-pclo...@gmail.com) series's I'm
wondering why repack takes so much memory to incrementally repack new
stuff when you leave out the base pack.
Repacking git.git takes around 290MB of memory on my system, but I'd
think that this would make it take a mere few megabytes, since all I'm
asking it to do is pack up the few loose objects that got added and keep
the base pack:
(
rm -rf /tmp/git &&
git clone g...@github.com:git/git.git /tmp/git &&
cd /tmp/git &&
touch $(ls .git/objects/pack/*pack | sed 's/\.pack$/.keep/') &&
for i in {1..10}
do
touch $i &&
git add $i &&
git commit -m$i
done &&
/usr/bin/time -f %M git repack -A -d
)
But no, it takes around 230MB. But thinking about it a bit further:
* This builds on top of existing history, so that needs to be
read/consulted
* We might be reusing (if not directly, skipping re-comuting) deltas
from the existing pack.
But I get the same result if after cloning I make an orphan branch, and
pass all the "do this as cheaply as possible" branches I can find down
to git-repack:
(
rm -rf /tmp/git &&
git clone g...@github.com:git/git.git /tmp/git &&
cd /tmp/git &&
touch $(ls .git/objects/pack/*pack | sed 's/\.pack$/.keep/') &&
git checkout --orphan new &&
git reset --hard &&
for i in {1..10}
do
touch $i &&
git add $i &&
git commit -m$i
done &&
git tag -d $(git tag -l) &&
/usr/bin/time -f %M git repack -A -d -f -F --window=1 --depth=1
)
But the memory use barely changes, my first example used 227924 kb, but
this one uses 226788.
Of course nobody's going to clone a huge repo and then right away create
an --orphan branch, but is there an inherent reason for why this
couldn't be taking as much memory as if the repo was cloned with
--depth=1?
I.e. when I have a *.keep on an existing pack we would have some
low-memory mode to copy the trees/blobs needed for the current commit
over to the new pack, and use that as the basis for packing everything
going forward.
Jeff: Is this something ref islands[1] could be (ab)used to do, or have
I misunderstood that concept?
1. https://public-inbox.org/git/20130626051117.gb26...@sigill.intra.peff.net/
https://public-inbox.org/git/20160304153359.ga16...@sigill.intra.peff.net/
https://public-inbox.org/git/20160809174528.2ydgkhd7aycla...@sigill.intra.peff.net/
