Borrowing from "git pack-objects" manpage[1] we have: A packed archive is an efficient way to transfer a set of objects between two repositories as well as an access efficient archival format. In a packed archive, an object is either stored as a compressed whole or as a difference from some other object. The latter is often called a delta.
The packed archive format (.pack) is designed to be self-contained so that it can be unpacked without any further information. Therefore, each object that a delta depends upon must be present within the pack. A pack index file (.idx) is generated for fast, random access to the objects in the pack. Placing both the index file (.idx) and the packed archive (.pack) in the pack/ subdirectory [...] enables Git to read from the pack archive. As stated, they are self-contained and can encapsulate an entire chunk of development history starting at point A and ending at point B, which makes them ideal not just for transferring, but also for sharing that chunk of history between repositories with common ancestry. When you ask a server for something, it sends a single pack, because that is most efficient. So, using the kernel as an example, lets say we decide we want to use the 2005 git "epoch" as "A" and the v5.10 tag of a recent release as "B" to delineate an initial block of history. We can grab that chunk of history with a simple: git clone --bare --single-branch --branch v5.10 git://someserver/linux my-510-lib and we now have all the history up to and including v5.10 tag in a repo that we can use with further "clone --reference my-510-lib newstuff" as a static unchanging library of all those objects for "newstuff" to use. In making such a clone, Git stores the my-510-lib dir in the objects/info/alternates of "newstuff" and all the refs and tags carry forard into [packed]refs of "newstuff". Subsequent uses of the "newstuff" repo are only using the objects from "my-510-lib" and they are conveniently all stored in one pack, as there are no loose objects. In fact it is a waste of time even having git operations look for them. This means we can simply remove the alternates file and link the pack (and accompanying index file) into the pack dir of "newstuff". And by using symlinks with relative paths, we get a mirror friendly reference with no data download duplication. The "clone --reference" is simply used as a convenience tool to do 99% of the setup work. The linked in pack is "just a normal pack" to the consumer of the commit library. As such, we introduce a "packref" as a useful parallel addition to the more well known "altref". It is ideally suited to reference from static repositories containing a known block of commit history. Note that we don't limit ourselves to a single pack, so that a packref can itself be using another repo as its own packref and so on. Hence there is no arbitrary limit of a depth of 5 as there is with a reference using the more common alternates (i.e. "altref") method. This opens the door to insert an arbitrary number of split points between A and B to fragment large repos, if desired. An example that shows how the bulk of kernel history can be split in two, three or four chunks will be provided for recipes-kernel/linux subsequently. [1] https://git-scm.com/docs/git-pack-objects Signed-off-by: Paul Gortmaker <paul.gortma...@windriver.com> --- .../bitbake-user-manual-fetching.rst | 7 +++++ bitbake/lib/bb/fetch2/git.py | 31 ++++++++++++++++++- 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index 12a68accc2c5..db1d108f4736 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -453,6 +453,13 @@ This fetcher supports the following parameters: is responsible (via fetch dependency) for ensuring the reference is populated/cloned prior to being called on to act as a reference. +- *"packref":* Enables cloned ``git://`` URLs to use packs from the named + peer from the downloads dir as if they were their own packs of commit + history. Similar to ``altref`` above, but suited for bare static repos + for which no loose objects outside of packs will ever be present. + Recipe is responsible (via fetch dependency) for ensuring the reference + is populated/cloned prior to being called on to act as a reference. + Here are some example URLs: :: SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1" diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 3b0c068bc020..8aec12df2bf8 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -65,6 +65,14 @@ Supported SRC_URI options are: operations. Recipe is responsible (via fetch dependency) for ensuring the reference is populated/cloned prior to being called on to act as a reference. +- packref + Repository is to use packs from named peer from downloads dir by linking + them into objects/pack/ of the new repository as if they were their own + packs of commit history. Similar to altref above, but suited for bare + static repos for which no loose objects outside of packs will be present. + Recipe is responsible (via fetch dependency) for ensuring the reference + is populated/cloned prior to being called on to act as a reference. + - usehead For local git:// urls to use the current branch HEAD as the revision for use with AUTOREV. Implies nobranch. @@ -171,10 +179,15 @@ class Git(FetchMethod): ud.static = ud.parm.get("static","0") == "1" + ud.packref = ud.parm.get("packref","") + ud.altref = ud.parm.get("altref","") ud.dlname = ud.parm.get("dlname","") + if ud.packref and ud.altref: + raise bb.fetch2.ParameterError("Please pick only ONE reference style: alternates OR packs", ud.url) + # usehead implies nobranch ud.usehead = ud.parm.get("usehead","0") == "1" if ud.usehead: @@ -373,6 +386,14 @@ class Git(FetchMethod): static = self.get_git_config(ud, d, repo, "bitbake.static") return (static == "true") + def create_pack_links(self, refname, refdir, dstdir): + dstpkdir = os.path.join(dstdir, 'objects', 'pack') + refpkdir = os.path.join(refdir, 'objects', 'pack') + for item in os.listdir(refpkdir): + dst = os.path.join(dstpkdir, item) + src = os.path.join('..', '..', '..', refname, 'objects', 'pack', item) + os.symlink(src, dst) + def download(self, ud, d): """Fetch url""" @@ -388,7 +409,9 @@ class Git(FetchMethod): repourl = self._get_repo_url(ud) - refname = ud.altref + # Both style of reference are the same for initial clone operation. + # It is the post-clone absolute to relative path fixup that differs. + refname = ud.altref or ud.packref if refname: alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates') gitdir = d.getVar("GITDIR") or (d.getVar("DL_DIR") + "/git2") @@ -419,6 +442,12 @@ class Git(FetchMethod): if ud.static: runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir) + if ud.packref: + if not self.repo_is_static(ud, d, refdir): + raise bb.fetch2.FetchError("Pack reference '%s' is not marked as static content." % refname) + self.create_pack_links(refname, refdir, ud.clonedir) + os.remove(alts) + # Update the checkout if needed if self.clonedir_need_update(ud, d): if ud.altref: -- 2.25.1
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9643): https://lists.yoctoproject.org/g/linux-yocto/message/9643 Mute This Topic: https://lists.yoctoproject.org/mt/81808165/21656 Group Owner: linux-yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
