On 30/11/2018 19:31, Pete Wright wrote:
On 11/30/18 9:08 AM, Grzegorz Junka wrote:
On 29/11/2018 19:12, Grzegorz Junka wrote:
Hello,
Is it possible to run poudriere in an agent/server setup? Where one
central server maintains a distfiles folder for already compiled
packages and agents compile packages fetching and installing from
the central server dependencies?
Just to make it more clear, I am asking about the build process.
Poudriere starts the specified amount of jails to build packages in
parallel. In each jail it then install dependencies by fetching them
from the distfiles folder. After the build is done a new package is
created and uploaded to the distfiles folder to be used by builds
that require it.
This process is CPU-core-bound, i.e. with 32/64 threads cores it
can't really build more than 64 packages in parallel. And systems
with so many cores are really expensive. But the whole process is
inherently distributed (as long as the amount of packages is big
enough so that they can be build in parallel).
Being able to build on multiple cheaper systems with less CPU cores
per system over the network seems like a good idea, no?
haven't fully thought this idea through, but couldn't you achieve this
by mounting the Poudriere 'data' directories as an NFS mountpoint?
This way each jail would view currently available package metadata and
hopefully avoid doing duplicate work?
I'm sure there will be racing condition edge cases though which will
bite you since I don't think there is a build job dispatcher/scheduler
available in Poudriere. So for example you may find yourself in a
situation where multiple jails try building LLVM as it's first package
which wouldn't really be helpful.
I believe poudriere plans what needs to be build before it starts the
build in order to detect circular dependencies. I don't think a shared
NFS mountpoint would be of any use. There ought to be a single scheduler
which deletes the outdated packages and plans what needs to be build,
but then instead distributing the build across jails on the same system,
it would have to be able to distribute the workload across multiple systems.
Thanks for confirming it's not available yet. Would be a nice and useful
addition. I should start learning poudriere code perhaps to see if this
is something I could do.
Regards
GrzegorzJ
_______________________________________________
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
