Re: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted!
On Mon, 28 Feb 2022 17:11:27 +0100
Michael Gmelin wrote:
[...]
schnipp
[...]
> >
> > poudriere jail -l:
> >
> > # poudriere jail -l
> > JAILNAME VERSION ARCH METHOD TIMESTAMP PATH
> > 123-amd64 12.3-RELEASE amd64
> > url=https://download.freebsd.org/releases/a ... 3-RELEASE/ 2022-02-24
> > 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE amd64
> > url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24
> > 14:11:32 /mnt/poudriere/jails/130-amd64
> >
> > The jail.conf for this specific jail is as follows:
> >
> > [...]
> > pulverfass-001 {
> > devfs_ruleset = 13;
> > enforce_statfs = 1;
> > exec.clean;
> > exec.consolelog =
> > /mnt/extensions/bastille/logs/pulverfass-001_console.log; exec.start
> > = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown';
> > host.hostname = X;
> > mount.devfs;
> > mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab;
> > path = /mnt/extensions/bastille/jails/pulverfass-001/root;
> > securelevel = 0;
> >
> > vnet;
> > vnet.interface = e0b_bastille4;
> > exec.prestart += "jib addm bastille4 igb0";
> > exec.prestart += "ifconfig e0a_bastille4 description \"vnet host
> > interface for Bastille jail pulverfass-001\""; exec.poststop += "jib
> > destroy bastille4";
> >
> > allow.mount;
> > allow.mount.fdescfs;
> > allow.mount.devfs;
> > allow.mount.tmpfs;
> > allow.mount.nullfs;
> > allow.mount.procfs;
> > allow.mount.linsysfs;
> > allow.mount.linprocfs;
> > allow.mount.zfs;
> >
> > allow.chflags;
> > allow.raw_sockets;
> > allow.socket_af;
> > allow.sysvipc;
> >
> > linux = new;
> >
> > exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere";
> > exec.start += "/sbin/zfs mount -a";
> > exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere";
> >
> > }
> > [...]
> >
> > Tracking the execution of the build process by issuing
> >
> > poudriere -x bulk ...
> >
> > and examin the resulting trace doesn' tgive me any hint, the error
> > reported above immediately occurs when the jail is about to be
> > started:
> >
> > + set -u +x
> > + jail -c persist 'name=123-amd64-head-default'
> > 'path=/mnt/poudriere/data/.m/ \ 123-amd64-head-default/ref'
> > 'host.hostname=basehost.local.domain' \ 'ip4.addr=127.0.0.1'
> > 'ip6.addr=::1' allow.chflags allow.sysvipc jail: jail_set: Operation
> > not permitted
> > + exit_handler
> > [...]
> >
> > Searching the net revealed some issues with setting IP4 and IP6 in
> > poudriere, but those findings are dated back to 2017 and 2014 and I
> > guess this is solved right now.
> >
> > The difference between our manually jail.conf driven setup and the
> > XigmaNAS/bastille based one is, bastille uses jib/netgraph based
> > seutups of the vnet and the ip4/ip6 is setup from rc.conf, while we
> > use epair in the other world and the ip is setup from withing the
> > jail definition in jail.conf.
> >
> > I'm out of ideas here and after two days of trial and error and
> > trying to understand what's going on lost ... Any hints or tipps?
> >
> > Thanks in advance,
> >
> > O. Hartmann
>
> Hi Oliver,
>
> I don't see `children.max` set in any of the configuration you shared
> above.
>
> Cheers
> Michael
>
Hello Michael,
bummer! I was so selfconfident because I copied the initial config from a
working test
and had this attribute already set that I never checked again its existence -
and started
reorganizing the jail.conf attributes ...
A fine observation and a full hit: after setting children.max= 128; the
poudriere jail
started working ... didn't wait for the finish so far.
I'm sorry for the noise - thanks for you eyes ...
Kind regards,
Oliver
Re: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted!
On Mon, 28 Feb 2022 16:15:45 +0100
FreeBSD User wrote:
> Hello folks,
>
> we run at least two poudriere build systems on recent CURRENT boxes
> and one of these poudriere build systems is working within a jail -
> setup via FreeBSD's /etc/jail.conf and by misusing the port ezjail
> for copying/deploying our self-compiled jail binary. The poudriere
> jail uses ZFS and is, to make it short, working like a charme.
>
> Now we try to setup another poudriere, but this time the base is
> XigmaNAS 12.3.0.4/9009, which is based upon 12.X-RELENG, utilizing
> "bastille". Bastille is up to date (in terms od the XigmaNAS plugin).
>
> Following the setup we used on the native CURRENT "jailed poudriere"
> builder and also following this reference (for those who want to
> check on this)
>
> https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere
>
> which seems quite recent and with the exception, that we use "vnet"
> on all of our systems for jails and so does XigmaNAS.
>
> Starting a building process via poudriere ends up with
>
>
> # poudriere bulk -p head -z default -j 123-amd64 -f
> /usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating
> the reference jail... done [00:00:01] Mounting system devices for
> 123-amd64-head-default [00:00:01] Warning: Using packages from
> previously failed, or uncommitted, build:
> /mnt/poudriere/data/packages/123-amd64-head-default/.building
> [00:00:01] Mounting ports from: /mnt/poudriere/ports/head [00:00:01]
> Mounting packages from:
> /mnt/poudriere/data/packages/123-amd64-head-default [00:00:01]
> Mounting distfiles from: /mnt/poudriere/ports/distfiles [00:00:01]
> Copying /var/db/ports from:
> /usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02]
> Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
> /etc/resolv.conf ->
> /mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf
> [00:00:02] Starting jail 123-amd64-head-default jail: jail_set:
> Operation not permitted [00:00:02] Cleaning up [00:00:02] Unmounting
> file systems
>
> poudriere jail -l:
>
> # poudriere jail -l
> JAILNAME VERSION ARCH METHOD TIMESTAMP PATH
> 123-amd64 12.3-RELEASE amd64
> url=https://download.freebsd.org/releases/a ... 3-RELEASE/ 2022-02-24
> 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE amd64
> url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24
> 14:11:32 /mnt/poudriere/jails/130-amd64
>
> The jail.conf for this specific jail is as follows:
>
> [...]
> pulverfass-001 {
> devfs_ruleset = 13;
> enforce_statfs = 1;
> exec.clean;
> exec.consolelog =
> /mnt/extensions/bastille/logs/pulverfass-001_console.log; exec.start
> = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown';
> host.hostname = X;
> mount.devfs;
> mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab;
> path = /mnt/extensions/bastille/jails/pulverfass-001/root;
> securelevel = 0;
>
> vnet;
> vnet.interface = e0b_bastille4;
> exec.prestart += "jib addm bastille4 igb0";
> exec.prestart += "ifconfig e0a_bastille4 description \"vnet host
> interface for Bastille jail pulverfass-001\""; exec.poststop += "jib
> destroy bastille4";
>
> allow.mount;
> allow.mount.fdescfs;
> allow.mount.devfs;
> allow.mount.tmpfs;
> allow.mount.nullfs;
> allow.mount.procfs;
> allow.mount.linsysfs;
> allow.mount.linprocfs;
> allow.mount.zfs;
>
> allow.chflags;
> allow.raw_sockets;
> allow.socket_af;
> allow.sysvipc;
>
> linux = new;
>
> exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere";
> exec.start += "/sbin/zfs mount -a";
> exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere";
>
> }
> [...]
>
> Tracking the execution of the build process by issuing
>
> poudriere -x bulk ...
>
> and examin the resulting trace doesn' tgive me any hint, the error
> reported above immediately occurs when the jail is about to be
> started:
>
> + set -u +x
> + jail -c persist 'name=123-amd64-head-default'
> 'path=/mnt/poudriere/data/.m/ \ 123-amd64-head-default/ref'
> 'host.hostname=basehost.local.domain' \ 'ip4.addr=127.0.0.1'
> 'ip6.addr=::1' allow.chflags allow.sysvipc jail: jail_set: Operation
> not permitted
> + exit_handler
> [...]
>
> Searching the net revealed some issues with setting IP4 and IP6 in
> poudriere, but those findings are dated back to 2017 and 2014 and I
> guess this is solved right now.
>
> The difference between our manually jail.conf driven setup and the
> XigmaNAS/bastille based one is, bastille uses jib/netgraph based
> seutups of the vnet and the ip4/ip6 is setup from rc.conf, while we
> use epair in the other world and the ip is setup from withing the
> jail definition in jail.conf.
>
> I'm out of ideas here and after two days of trial and error and
> trying to understand what's going on lost ... Any hints or tipps?
>
> Thanks in advance,
>
> O. Hartmann
Hi Oliver,
I don't see `children.max` set in any of the configuration you shared
above.
Cheers
Michael
--
Michael
Re: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted!
There is a number of places which can return EPERM, to paste an example:
if (gotslevel) {
if (slevel < ppr->pr_securelevel) {
error = EPERM;
goto done_deref_locked;
}
}
if (gotchildmax) {
if (childmax >= ppr->pr_childmax) {
error = EPERM;
goto done_deref_locked;
}
}
if (gotenforce) {
if (enforce < ppr->pr_enforce_statfs) {
error = EPERM;
goto done_deref_locked;
}
}
I see in your config you have enforce_statfs = 1; , perhaps that's
smaller than what the host has and in that case you would get the
error.
Ultimately, while cumbersome, you can add printf to the code to find
out which case is giving you the error.
The real thing to do here would be implement something like SET_ERROR
from illumos, which would allow to immediately pin point where the
problem is coming from.
On 2/28/22, FreeBSD User wrote:
> Hello folks,
>
> we run at least two poudriere build systems on recent CURRENT boxes and one
> of these
> poudriere build systems is working within a jail - setup via FreeBSD's
> /etc/jail.conf and
> by misusing the port ezjail for copying/deploying our self-compiled jail
> binary. The
> poudriere jail uses ZFS and is, to make it short, working like a charme.
>
> Now we try to setup another poudriere, but this time the base is XigmaNAS
> 12.3.0.4/9009,
> which is based upon 12.X-RELENG, utilizing "bastille". Bastille is up to
> date (in terms
> od the XigmaNAS plugin).
>
> Following the setup we used on the native CURRENT "jailed poudriere" builder
> and also
> following this reference (for those who want to check on this)
>
> https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere
>
> which seems quite recent and with the exception, that we use "vnet" on all
> of our systems
> for jails and so does XigmaNAS.
>
> Starting a building process via poudriere ends up with
>
>
> # poudriere bulk -p head -z default -j 123-amd64 -f
> /usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating the
> reference
> jail... done [00:00:01] Mounting system devices for 123-amd64-head-default
> [00:00:01] Warning: Using packages from previously failed, or uncommitted,
> build:
> /mnt/poudriere/data/packages/123-amd64-head-default/.building [00:00:01]
> Mounting ports
> from: /mnt/poudriere/ports/head [00:00:01] Mounting packages from:
> /mnt/poudriere/data/packages/123-amd64-head-default [00:00:01] Mounting
> distfiles from:
> /mnt/poudriere/ports/distfiles [00:00:01] Copying /var/db/ports from:
> /usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02]
> Appending to
> make.conf: /usr/local/etc/poudriere.d/make.conf /etc/resolv.conf ->
> /mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf [00:00:02]
> Starting
> jail 123-amd64-head-default jail: jail_set: Operation not permitted
> [00:00:02] Cleaning up
> [00:00:02] Unmounting file systems
>
> poudriere jail -l:
>
> # poudriere jail -l
> JAILNAME VERSION ARCH METHOD TIMESTAMP PATH
> 123-amd64 12.3-RELEASE amd64 url=https://download.freebsd.org/releases/a ...
> 3-RELEASE/
> 2022-02-24 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE
> amd64
> url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24
> 14:11:32
> /mnt/poudriere/jails/130-amd64
>
> The jail.conf for this specific jail is as follows:
>
> [...]
> pulverfass-001 {
> devfs_ruleset = 13;
> enforce_statfs = 1;
> exec.clean;
> exec.consolelog = /mnt/extensions/bastille/logs/pulverfass-001_console.log;
> exec.start = '/bin/sh /etc/rc';
> exec.stop = '/bin/sh /etc/rc.shutdown';
> host.hostname = X;
> mount.devfs;
> mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab;
> path = /mnt/extensions/bastille/jails/pulverfass-001/root;
> securelevel = 0;
>
> vnet;
> vnet.interface = e0b_bastille4;
> exec.prestart += "jib addm bastille4 igb0";
> exec.prestart += "ifconfig e0a_bastille4 description \"vnet host interface
> for Bastille
> jail pulverfass-001\""; exec.poststop += "jib destroy bastille4";
>
> allow.mount;
> allow.mount.fdescfs;
> allow.mount.devfs;
> allow.mount.tmpfs;
> allow.mount.nullfs;
> allow.mount.procfs;
> allow.mount.linsysfs;
> allow.mount.linprocfs;
> allow.mount.zfs;
>
> allow.chflags;
> allow.raw_sockets;
> allow.socket_af;
> allow.sysvipc;
>
> linux = new;
>
> exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere";
> exec.start += "/sbin/zfs mount -a";
> exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere";
>
> }
> [...]
>
> Tracking the execution of the build process by issuing
>
> poudriere -x bulk ...
>
> and examin the resulting trace doesn' tgive me any hint, the error reported
> above
> immediately occurs when the jail is about to be started:
>
> + set -u +x
> + jail -c persist
bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted!
Hello folks,
we run at least two poudriere build systems on recent CURRENT boxes and one of
these
poudriere build systems is working within a jail - setup via FreeBSD's
/etc/jail.conf and
by misusing the port ezjail for copying/deploying our self-compiled jail
binary. The
poudriere jail uses ZFS and is, to make it short, working like a charme.
Now we try to setup another poudriere, but this time the base is XigmaNAS
12.3.0.4/9009,
which is based upon 12.X-RELENG, utilizing "bastille". Bastille is up to date
(in terms
od the XigmaNAS plugin).
Following the setup we used on the native CURRENT "jailed poudriere" builder
and also
following this reference (for those who want to check on this)
https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere
which seems quite recent and with the exception, that we use "vnet" on all of
our systems
for jails and so does XigmaNAS.
Starting a building process via poudriere ends up with
# poudriere bulk -p head -z default -j 123-amd64 -f
/usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating the
reference
jail... done [00:00:01] Mounting system devices for 123-amd64-head-default
[00:00:01] Warning: Using packages from previously failed, or uncommitted,
build:
/mnt/poudriere/data/packages/123-amd64-head-default/.building [00:00:01]
Mounting ports
from: /mnt/poudriere/ports/head [00:00:01] Mounting packages from:
/mnt/poudriere/data/packages/123-amd64-head-default [00:00:01] Mounting
distfiles from:
/mnt/poudriere/ports/distfiles [00:00:01] Copying /var/db/ports from:
/usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02] Appending
to
make.conf: /usr/local/etc/poudriere.d/make.conf /etc/resolv.conf ->
/mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf [00:00:02]
Starting
jail 123-amd64-head-default jail: jail_set: Operation not permitted
[00:00:02] Cleaning up
[00:00:02] Unmounting file systems
poudriere jail -l:
# poudriere jail -l
JAILNAME VERSION ARCH METHOD TIMESTAMP PATH
123-amd64 12.3-RELEASE amd64 url=https://download.freebsd.org/releases/a ...
3-RELEASE/
2022-02-24 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE amd64
url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24 14:11:32
/mnt/poudriere/jails/130-amd64
The jail.conf for this specific jail is as follows:
[...]
pulverfass-001 {
devfs_ruleset = 13;
enforce_statfs = 1;
exec.clean;
exec.consolelog = /mnt/extensions/bastille/logs/pulverfass-001_console.log;
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
host.hostname = X;
mount.devfs;
mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab;
path = /mnt/extensions/bastille/jails/pulverfass-001/root;
securelevel = 0;
vnet;
vnet.interface = e0b_bastille4;
exec.prestart += "jib addm bastille4 igb0";
exec.prestart += "ifconfig e0a_bastille4 description \"vnet host interface for
Bastille
jail pulverfass-001\""; exec.poststop += "jib destroy bastille4";
allow.mount;
allow.mount.fdescfs;
allow.mount.devfs;
allow.mount.tmpfs;
allow.mount.nullfs;
allow.mount.procfs;
allow.mount.linsysfs;
allow.mount.linprocfs;
allow.mount.zfs;
allow.chflags;
allow.raw_sockets;
allow.socket_af;
allow.sysvipc;
linux = new;
exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere";
exec.start += "/sbin/zfs mount -a";
exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere";
}
[...]
Tracking the execution of the build process by issuing
poudriere -x bulk ...
and examin the resulting trace doesn' tgive me any hint, the error reported
above
immediately occurs when the jail is about to be started:
+ set -u +x
+ jail -c persist 'name=123-amd64-head-default' 'path=/mnt/poudriere/data/.m/ \
123-amd64-head-default/ref' 'host.hostname=basehost.local.domain' \
'ip4.addr=127.0.0.1' 'ip6.addr=::1' allow.chflags allow.sysvipc
jail: jail_set: Operation not permitted
+ exit_handler
[...]
Searching the net revealed some issues with setting IP4 and IP6 in poudriere,
but those
findings are dated back to 2017 and 2014 and I guess this is solved right now.
The difference between our manually jail.conf driven setup and the
XigmaNAS/bastille
based one is, bastille uses jib/netgraph based seutups of the vnet and the
ip4/ip6 is
setup from rc.conf, while we use epair in the other world and the ip is setup
from
withing the jail definition in jail.conf.
I'm out of ideas here and after two days of trial and error and trying to
understand
what's going on lost ... Any hints or tipps?
Thanks in advance,
O. Hartmann
