On Nov 4, 2023, at 13:56, Tatsuki Makino <[email protected]> wrote:
> Hello.
> (I am replying to freebsd-ports only.)
Seems reasonable.
> Mark Millard wrote on 2023/11/04 21:00:
>> # ls -a /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ | wc -l
>> 32767
>
> The number of packages will be inflated relative to the number of origins
> because FLAVORS uses PKGNAMEPREFIX and PKGNAMESUFFIX.
/usr/local/poudriere/data/logs/bulk/latest-per-pkg/ contains direcotries
that have the names of the packages with log files form bulk runs, not
origins.
> Therefore, something like BUILD_ALL_PYTHON_FLAVORS should not be used
> continuously for purposes other than development and debugging.
The from scratch bulk -a run is a kind of rare experiment that I do, not
a regular thing. It is deliberately a simple bulk -a run when I start
such an experimental sequence.
> Ports for which the origin directory no longer exists, and whose description
> no longer exists from MOVED (e.g. www/seamonkey), seem to remain in the
> package forever.
> It is better to run poudriere-pkgclean regularly.
The sequence used to start my experiments is as shown below:
#! /bin/sh
poudriere pkgclean -jmain-amd64-bulk_a -A
poudriere bulk -jmain-amd64-bulk_a -a
So pkgclean had been run. However, as I remember, this does
not manage log files. I do that separately.
For /usr/local/poudriere/data/logs/bulk/latest-per-pkg/
being empty at the start, the failure would still happen.
The 32767 is a UFS limitation via a 16 bit field associated with:
/usr/include/ufs/ufs/dinode.h:#define UFS_LINK_MAX 32767
Specifically di_nlink in sys/ufs/ufs/dinode.h :
struct ufs2_dinode {
uint16_t di_mode; /* 0: IFMT, permissions; see below. */
int16_t di_nlink; /* 2: File link count. */
. . .
Having more than 32767 packages attempted to be built
leads to:
/usr/local/poudriere/data/logs/bulk/latest-per-pkg/
trying to go beyond that UFS_LINK_MAX limit in what latest-per-pkg/
contains.
ZFS does not have the limitation and the prior from-scratch ZFS
based bulk -a experiment was able to finish its build. bulk -a
in the ZFS context historically used somewhat more RAM+SWAP space
than a UFS based bulk -a would.
> Would checking that area reduce that number?
For amd64, there are over 32767 packages that should get directories
in /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ . The usage
poudriere puts them to counts them in di_nlink in UFS's directory inode
and the UFS limitation is reached before all the packages are present.
/usr/local/poudriere/data/logs/bulk/latest-per-pkg/ ends up just not
fitting in UFS's limitations for how things are currently organized.
I used to be able to compare/contrast the ZFS vs. UFS RAM+SWAP usage
for bulk -a runs. But last I did such was before the builds going
beyond 32767 packages with logs.
(I've not checked if /usr/local/poudriere/data/logs/bulk/latest-per-pkg/
is the only issue. It is just the first context to stop the build.)
===
Mark Millard
marklmi at yahoo.com
