On 07/08/13 22:12, Teske, Devin wrote:
We also had to put one file into the etc directory on the / beneath the /etc
mount so that /sbin/init can read it before /etc is mounted. There were two or three ways
we could do that and each has a tradeoff.
I've been bitten by that.
Getting access to that file that's beneath once you've booted the system can
be ... less than easy.
yeah i prefer resorting to trickery or magic as little as possible
only as a last resort and i try to clutter up the standard tree of files
as little as possible. in this case we only needed the one file, just a
symlink actually. the under has only the following:
lrwxr-xr-x 1 root wheel 25 Jun 25 17:59 /login.conf.db@ -
../boot/etc/login.conf.db
and in the over /etc we still place an identical symlink so that the
real file is in /boot/etc/. cap_mkdb doesnt clobber the symlink, it
writes through to /boot/etc/login.conf.db for you. so in the normal
usual case, a user edits login.conf and runs cap_mkdb like they're
supposed to and everything is fine. its only if they rollback or restore
a backup to /etc that things potentially can end up out of sync.
i don't want anyone to get confused by me talking about jails in the
same email. The above snag we are working around involves /sbin/init
ONLY WHEN booting the host FreeBSD. Our jailed customers don't have to
worry about this because /etc is already in the right spot by the time
jail runs /etc/rc. /sbin/init isn't even involved in a jail, is it? Not
even in some hooked-in way? At any rate we dont have to do anything
special for a separate /etc dataset for jails.
We could just forgo the /etc dataset on the host but i am glad that we
can manage our bare metal customers using the same methods and tools.
Handling this symlink hack is less differentiation than giving up
separate /etc on the host i think.
I'm interested in your cost/benefit points of having /etc a separate filesystem.
On the face of it, I want to say that /etc is (or at least contains) the core
identity of the machine (and to a lesser extent -- because this is BSD after-all --
/usr/local/etc). In my mind, /etc and /usr/local/etc *are* the machine (metaphorically speaking),
so the merits of having it as a separate filesystem are weighed against your desired topology.
i agree. myself i like having such a lightweight identity and keeping
/, /usr and /usr/local (which are all just on sitting on / in my case)
mounted read-only. the prototype for a host is handled by a completely
different department than the people/customers who sysadmin their
deployments and instances. Early in the building/installing, before any
ports/packages, /usr/local/etc is made a symlink to /etc/local, so the
symlink is in the readonly / and every time you write or cd to
/usr/local/etc you end up in /etc/local. An /etc dataset ends up under a
MB zfs compressed and /var on a fresh instance is basically also
nothing. all-in-all a new jail costs you under a MB of zpool. we jail
stop/start and zfs send/receive instances in a blink of an eye and its
almost as good as having live migration.
We could get the same storage efficiency by simply cloning /, and having
no sub-datasets. some customers feel like they want to be able to write
anywhere and we give them those options but then they are on their own
and we don't manage the software updates for those guys and some like it
that way. we then bill each for all the storage they reference because a
year down the road they may be the only one still holding a reference to
the outdated prototype they're on even though they overwrote every file
twice with make world or freebsd-update. their memory usage is also way
higher than most because when executables are launched on the jails with
the read-only nullfs mounted /, those all access the same memory pages
but zfs isnt smart enough yet to let the virtual memory system maintain
those pointers through the indirection of zfs clones and snapshots.
so zfs separate /etc and /var give us great storage efficiency while
nullfs gives us great memory performance and efficiency.
If you want to bunch of machines to look and/or act differently, then a shared
/etc is precisely what you want. However, without allowing minor changes (ala
ZFS clone/snapshot or by way of UnionFS), you'll quickly find that the only way
to cope is with role-based scripting in /etc/rc.conf (it is after-all a shell
script) or complicated abstraction layers (for example, using netgraph eiface
devices with the jail-name inside them so that rc.conf have have jail-specific
ifconfig_* lines). But I digress.
I think the better solution to your loading of files beneath the eventual
/etc filesystem is to throw away the ZFS snapshot/clone method and instead move to a
UnionFS approach for /etc.
If you use UnionFS for your /etc, then what you do is for each of the machines
that you want *that* /etc to appear, you do something like:
(as
