Re: HEADS-UP: dpb changes
On Sun, May 10, 2015 at 08:16:05AM -0400, RD Thrush wrote: x6v64:build/packages 531cat dpb_hosts.amd64 DEFAULT timeout=10 build_user=dpb memory=200M stuck=1000 fetch_user=dpb_fetch log_user=dpb_log arch=amd64 STARTUP=/home/rd/OpenBSD/build/packages/dpb_start dpb@localhost stuck=2000 memory=1G sf=1 jobs=1 arch=amd64 x6v64:build/packages 532cat dirlist.amd64 databases/p5-ldap x6v64:build/packages 533cat pkg_info-qPa.all databases/p5-ldap This got changed late in the game. You have to specify FETCH_USER LOG_USER UNPRIV_USER as defines, not default stuff now. my current hosts file contains: STARTUP=/usr/ports/cleanup UNPRIV_USER=dpbcheck FETCH_USER=fetchuser LOG_USER=espie DIRMODE=0770 DEFAULT jobs=12 parallel=4 build_user=pbuild0 dirmode=0770 chroot=/build localhost openbsd-2 openbsd-3 COLOR=1 FETCH_JOBS=8 and I'm usually running dpb with 0 options.
Re: HEADS-UP: dpb changes
On 05/01/15 16:10, Marc Espie wrote: I've worked hard to allow for dpb to work in a new model, most specifically so that chroot always works, and also to have a slightly better security model. Thanks for this large improvement. I have struggled w/ the previous chroot model as I wanted to use stable boxes with idle cycles but did not want to compromise their working state. The corresponding code and documentation have been committed, but they probably need people to play with it a bit more to make sure all kinks are gone. There are some important implications with respect to ports bulk building security. In the new model, dpb no longer requires any kind of sudo operation, it's moved to a privilege separation model. - dpb should be started as root, it will drop privileges as needed. - the basic core of dpb runs as root, but any time it's actually looking at the ports tree, it will drop to a build_user (which has to be specified). This user does not require any root access. - dpb will stay root to run the STARTUP script, and also to run pkg_add/pkg_delete to handle dependencies. - fetch can be run as a separate user (thus, the build_user shouldn't even have internet access under normal rules). Commands in the ports tree will actually be run as chroot -u build_user /somedir cmd Thus, all builds are chrooted by default with / being used as a root when there's no chroot. Distant buils will ssh from root to root, and run the same command (chroot -u build_user /somedir cmd) on the distant host. This is somewhat necessary so that killing dpb will correctly propagate signals to all running jobs, which has been an issue with previous attempts at running chroot. That new user model is probably going to become the ONLY operating model of dpb in the near future. Having options suck and is a problem for maintenance and security. People running bulks should transition as soon as they can. The manpage mentions which files belong to whom. It is highly advisable to have a specific build user without sudo rights and (possibly) restricted net access. While I haven't been successful w/ a partial build, I was able to do a successful fetch for my partial build. With the latest changes today, I thought it was time to make a report. I've created accounts for dpb, dpb_fetch and dpb_log on the relevant boxes sharing the same gid. x6v64:build/packages 522grep ^dpb /etc/passwd dpb:*:1100:1100:dpb build user:/home/dpb:/bin/ksh dpb_fetch:*:1101:1100:dpb fetch user:/home/dpb_fetch:/bin/ksh dpb_log:*:1102:1100:dpb log user:/home/dpb_log:/bin/ksh x6v64:build/packages 523grep ^dpb /etc/group dpb:*:1100: I use a wrapper and script the session. Here's what I see targetting just the localhost: x6v64:build/packages 524$TIME sudo ./Do_dpb-without-a8v -l -v dirlist.$(arch -s);exit PACKAGE_REPOSITORY /nas3/work/OpenBSD/packages BULK_COOKIES_DIR /nas3/work/OpenBSD/x6v64/bulk/amd64 UPDATE_COOKIES_DIR /nas3/work/OpenBSD/x6v64/update/amd64 LOGGER_DIR /usr/obj/amd64/logs rm -f /nas3/work/OpenBSD/x6v64/bulk/amd64/* rm -f /nas3/work/OpenBSD/x6v64/update/amd64/* rm -f /usr/obj/amd64/logs/term-report.log /usr/obj/amd64/logs/debug.log OPTS=-L /usr/obj/amd64/logs -s -A amd64 -c -R -U -J 0 -X /home/rd/OpenBSD/build/packages/pkg_info-qPa.all -h /home/rd/OpenBSD/build/packages/dpb_hosts.amd64 -f 8 -D SYSLOG /usr/bin/time sudo /usr/ports/infrastructure/bin/dpb -L /usr/obj/amd64/logs -s -A amd64 -c -R -U -J 0 -X /home/rd/OpenBSD/build/packages/pkg_info-qPa.all -h /home/rd/OpenBSD/build/packages/dpb_hosts.amd64 -f 8 -D SYSLOG -P dirlist.amd64 Too early at /usr/ports/infrastructure/lib/DPB/Logger.pm line 33. DPB::Logger::new(DPB::Logger, DPB::State=HASH(0xc4e35c64670)) called at /usr/ports/infrastructure/lib/DPB/State.pm line 143 DPB::State::handle_options(DPB::State=HASH(0xc4e35c64670)) called at /usr/ports/infrastructure/bin/dpb line 145 0.84 real 0.45 user 0.13 sys 6.17 real 0.93 user 0.68 sys Here's more info related to the above invocation: x6v64:build/packages 531cat dpb_hosts.amd64 DEFAULT timeout=10 build_user=dpb memory=200M stuck=1000 fetch_user=dpb_fetch log_user=dpb_log arch=amd64 STARTUP=/home/rd/OpenBSD/build/packages/dpb_start dpb@localhost stuck=2000 memory=1G sf=1 jobs=1 arch=amd64 x6v64:build/packages 532cat dirlist.amd64 databases/p5-ldap x6v64:build/packages 533cat pkg_info-qPa.all databases/p5-ldap After the above, there are no file updates in the $LOGGER_DIR although ownership and perms seem ok: x6v64:build/packages 535ls -ld /usr/obj/amd64/logs drwxrwxrwx 6 rd dpb 1024 May 10 07:22 /usr/obj/amd64/logs I've collected the syslog debug level and don't see any clues (/usr /usr/local are separate partitions mounted read-only so the dpb_start script makes them read-only and results in the following sudo mount entries): x6v64:build/packages 536cat /var/log/debug May 10 07:41:19 x6v64 syslogd: start May
HEADS-UP: dpb changes
I've worked hard to allow for dpb to work in a new model, most specifically so that chroot always works, and also to have a slightly better security model. The corresponding code and documentation have been committed, but they probably need people to play with it a bit more to make sure all kinks are gone. There are some important implications with respect to ports bulk building security. In the new model, dpb no longer requires any kind of sudo operation, it's moved to a privilege separation model. - dpb should be started as root, it will drop privileges as needed. - the basic core of dpb runs as root, but any time it's actually looking at the ports tree, it will drop to a build_user (which has to be specified). This user does not require any root access. - dpb will stay root to run the STARTUP script, and also to run pkg_add/pkg_delete to handle dependencies. - fetch can be run as a separate user (thus, the build_user shouldn't even have internet access under normal rules). Commands in the ports tree will actually be run as chroot -u build_user /somedir cmd Thus, all builds are chrooted by default with / being used as a root when there's no chroot. Distant buils will ssh from root to root, and run the same command (chroot -u build_user /somedir cmd) on the distant host. This is somewhat necessary so that killing dpb will correctly propagate signals to all running jobs, which has been an issue with previous attempts at running chroot. That new user model is probably going to become the ONLY operating model of dpb in the near future. Having options suck and is a problem for maintenance and security. People running bulks should transition as soon as they can. The manpage mentions which files belong to whom. It is highly advisable to have a specific build user without sudo rights and (possibly) restricted net access.
dpb changes
A bit of stuff got committed. Now, build statistics get collected
under ${DISTDIR}/build-stats/${ARCH}
(as known as %f/build-stats/%a, since dpb can replace patterns inside
files now).
if you think the location is strange, I expect to, eventually, bootstrap
first run dpbs by asking one of the mirror sites to populate that file.
For now, this build-stats file is smart, using the same techniques as
per global distinfo, so that old stats will eventually vanish.
Expect things to change a bit more over the coming weeks...
One idea is that, now, you could simply zap the logs if you don't need them,
pass no option to dpb at all, and this file would ensure that eventually,
you'd get % for all builds, and also optimized builds from the previous builds.
One major point is that those stats don't have to be perfect, but just good
enough to direct dpb to build big stuff first.
dpb changes, summary
Since my commit messages are probably a bit obscure, here's a small progress report. - fixed a bug that would make dpb loop if it were not happy with some listing - fixed a few display bugs. dpb had a problem with display containing empty lines... not any more ! - distinguish between default and empty flavors. All error messages related to avahi, xscreensaver... should be gone. - ditched the P/I distinction. Some code gone, simpler stuff. - distinguish between our errors and other random locks so that... - in case of an error, rescan the affected directory when the lock gets removed. YES, this does pick up revision bumps and other meta-info changes. - create errors in case scanning doesn't grok directories. So, yes, you can fix cvs update errors, remove the lock and it will pick it up. - in case prepare fails, try depend anyways. Lightweight fix for the ImageMagick build. Not done yet: - stop/resume control over each host - devel/haddock,no_deps still shows up. Requires a full build to test that I know how to kill that...
Re: dpb changes, summary
On Thu, Oct 28, 2010 at 01:43:57PM +0200, Marc Espie wrote: Since my commit messages are probably a bit obscure, here's a small progress report. - fixed a bug that would make dpb loop if it were not happy with some listing - fixed a few display bugs. dpb had a problem with display containing empty lines... not any more ! - distinguish between default and empty flavors. All error messages related to avahi, xscreensaver... should be gone. - ditched the P/I distinction. Some code gone, simpler stuff. - distinguish between our errors and other random locks so that... - in case of an error, rescan the affected directory when the lock gets removed. YES, this does pick up revision bumps and other meta-info changes. For the people using dpb, it means you can now apply diffs containing fixes and a revision bump for a port and dpb will cope with it. So far you had to apply diffs and manually remove the revision bump.. Landry
