On Sun, Sep 27, 2020 at 10:09:03AM +0200, Johannes Schauer wrote: > Hi, > > Quoting Josh Triplett (2020-09-27 08:07:26) > > > Yes, one bit is missing. For the initial Essential:yes package set, > > > mmdebstrap deletes the *.deb files itself after installing them. So you > > > have to copy them out of the chroot before the deletion happens, so > > > between > > > downloading and extracting. Like so: > > > > > > $ sudo mmdebstrap --variant=apt --include='systemd-sysv udev' \ > > > > --setup-hook='mkdir -p cache "$1"/var/cache/apt/archives' \ > > > > --setup-hook='sync-in ./cache /var/cache/apt/archives' \ > > > > --extract-hook="sync-out /var/cache/apt/archives ./cache" \ > > > > --customize-hook='sync-out /var/cache/apt/archives ./cache' \ > > > > unstable debian-unstable > > > > Ah! That was indeed what I was missing, thank you. > > > > Is it documented somewhere that mmdebstrap deletes those files after > > installing them? > > no, that is not documented. I'm also unsure where to write that down and for > how many people this would be of interest.
I think it would make sense to have as part of the documentation of hooks, which goes hand-in-hand with an outline of mmdebstrap's operation and where those hooks fit. For instance, I didn't know until this conversation that mmdebstrap used a different mechanism to handle Essential than non-Essential. > Would it be useful in any scenario to keep those *.deb files? For caching, yes, to avoid downloading them more than once. > > Also, does mmdebstrap *use* the cache for the initial Essential package set? > > Short answer: yes. > > But as I was making sure what the right answer to your question is, I > discovered another problem with our current approach: mmdebstrap uses apt to > download the right packages but apt does not provide a machine readable > interface to figure out which packages it decided to fetch, so what mmdebstrap > does, is to just use all packages in /var/cache/apt/archives/ as the result of > the apt query for the Essential:yes packages. Ah, ouch. Would it be possible to use libapt for that, via one of its various bindings? > This is a problem if /var/cache/apt/archives/ also contains non-Essential > packages because many packages cannot be installed in the initial bootstrap > phase due to maintainer scripts. You can try out this effect by using the > above > command and putting more packages into --include and then during the second > run, mmdebstrap will fail. To make sure, that we don't run into this problem > without noticing in the future, I added a check to mmdebstrap whether > /var/cache/apt/archives/ is empty before acquiring any packages. To not have > this error prevent use-cases like yours, you can add the --skip=download/empty > option to your mmdebstrap invocation with the next mmdebstrap release. > > So we need *two* caches: One for the Essential packages and one for the rest. > I didn't test, but it would probably be something like this: > > $ sudo mmdebstrap --variant=apt --include='systemd-sysv udev' \ > > --setup-hook='mkdir -p ./cache.ess "$1"/var/cache/apt/archives' \ > > --setup-hook='sync-in ./cache.ess /var/cache/apt/archives' \ > > --extract-hook="sync-out /var/cache/apt/archives ./cache.ess" \ > > --essential-hook='mkdir -p ./cache.rest "$1"/var/cache/apt/archives' \ > > --essential-hook='sync-in ./cache.rest /var/cache/apt/archives' \ > > --customize-hook='sync-out /var/cache/apt/archives ./cache.rest' \ > > unstable debian-unstable This is definitely at the level of complexity where it'd be *really nice* to just be able to pass `--cache-debs cachedir`, without depending deeply on the specific nuance of mmdebstrap's essential/non-essential handling (and potential future changes to that). > > > > > This will not work with all modes. I guess you are using root or > > > > > fakechroot > > > > > mode? > > > > > > > > Yes, I'm using root mode, as I need a directory of files I can feed to > > > > `mkfs.ext4 -d`. In theory I might be able to use fakechroot mode and > > > > run mkfs.ext4 underneath mmdebstrap; I may try that once I've gotten the > > > > existing setup to do everything I need it to. > > > > > > Maybe the ability of mmdebstrap to produce ext2 filesystems directly can > > > be > > > useful for you. You will then not need root privileges. > > > > I did see that, but it looks like that uses genext2fs, and I need to use > > mkfs.ext4 so that I can generate an ext4 filesystem with specific > > options (as well as write files out with extents). It also doesn't > > support xattrs. > > You can upgrade ext2 to ext4 by using "tune2fs -O > extents,uninit_bg,dir_index,has_journal" but I see that this would be of no > use > to you. Right, that won't actually convert how existing files are handled. That would take an e2fsck run with specific options. And the initial filesystem would need more space than the final one, requiring an unecessarily large image.
