On 1/10/2018 1:25 PM, Martin Fick wrote:
On Sunday, January 07, 2018 01:14:41 PM Derrick Stolee
wrote:
This RFC includes a new way to index the objects in
multiple packs using one file, called the multi-pack
index (MIDX).
...
The main goals of this RFC are:
* Determine interest in this feature.
* Find other use cases for the MIDX feature.
My interest in this feature would be to speed up fetches
when there is more than one large pack-file with many of the
same objects that are in other pack-files. What does your
MIDX design do when it encounters multiple copies of the
same object in different pack files? Does it index them all,
or does it keep a single copy?
The MIDX currently keeps only one reference to each object. Duplicates
are dropped during writing. (See the care taken in commit 04/18 to avoid
duplicates.) Since midx_sha1_compare() does not use anything other than
the OID to order the objects, there is no decision being made about
which pack is "better". The MIDX writes the first copy it finds and
discards the others.
It would not be difficult to include a check in midx_sha1_compare() to
favor one packfile over another based on some measurement (size?
mtime?). Since this would be a heuristic at best, I left it out of the
current patch.
In our Gerrit instance (Gerrit uses jgit), we have multiple
copies of the linux kernel repos linked together via the
alternatives file mechanism.
GVFS also uses alternates for sharing packfiles across multiple copies
of the repo. The MIDX is designed to cover all packfiles in the same
directory, but is not designed to cover packfiles in multiple
alternates; currently, each alternate would need its own MIDX file. Does
that cause issues with your setup?
These repos have many different
references (mostly Gerrit change references), but they share
most of the common objects from the mainline. I have found
that during a large fetch such as a clone, jgit spends a
significant amount of extra time by having the extra large
pack-files from the other repos visible to it, usually around
an extra minute per instance of these (without them, the
clone takes around 7mins). This adds up easily with a few
repos extra repos, it can almost double the time.
My investigations have shown that this is due to jgit
searching each of these pack files to decide which version of
each object to send. I don't fully understand its selection
criteria, however if I shortcut it to just pick the first
copy of an object that it finds, I regain my lost time. I
don't know if git suffers from a similar problem? If git
doesn't suffer from this then it likely just uses the first
copy of an object it finds (which may not be the best object
to send?)
It would be nice if this use case could be improved with
MIDX. To do so, it seems that it would either require that
MIDX either only put "the best" version of an object (i.e.
pre-select which one to use), or include the extra
information to help make the selection process of which copy
to use (perhaps based on the operation being performed)
fast.
I'm not sure if there is sufficient value in storing multiple references
to the same object stored in multiple packfiles. There could be value in
carefully deciding which copy is "best" during the MIDX write, but
during read is not a good time to make such a decision. It also
increases the size of the file to store multiple copies.
This also leads me to ask, what other additional information
(bitmaps?) for other operations, besides object location,
might suddenly be valuable in an index that potentially
points to multiple copies of objects? Would such
information be appropriate in MIDX, or would it be better in
another index?
For applications to bitmaps, it is probably best that we only include
one copy of each object. Otherwise, we need to include extra bits in the
bitmaps for those copies (when asking "is this object in the bitmap?").
Thanks for the context with Gerrit's duplicate object problem. I'll try
to incorporate it in to the design document (commit 01/18) for the v1 patch.
Thanks,
-Stolee
