login.conf.db, /sbin/init, separate /etc, and configs around thin provisioning WAS: Re: nuOS
On 07/08/13 22:12, Teske, Devin wrote: We also had to put one file into the etc directory on the / beneath the /etc mount so that /sbin/init can read it before /etc is mounted. There were two or three ways we could do that and each has a tradeoff. I've been bitten by that. Getting access to that file that's beneath once you've booted the system can be ... less than easy. yeah i prefer resorting to trickery or magic as little as possible only as a last resort and i try to clutter up the standard tree of files as little as possible. in this case we only needed the one file, just a symlink actually. the under has only the following: lrwxr-xr-x 1 root wheel 25 Jun 25 17:59 /login.conf.db@ - ../boot/etc/login.conf.db and in the over /etc we still place an identical symlink so that the real file is in /boot/etc/. cap_mkdb doesnt clobber the symlink, it writes through to /boot/etc/login.conf.db for you. so in the normal usual case, a user edits login.conf and runs cap_mkdb like they're supposed to and everything is fine. its only if they rollback or restore a backup to /etc that things potentially can end up out of sync. i don't want anyone to get confused by me talking about jails in the same email. The above snag we are working around involves /sbin/init ONLY WHEN booting the host FreeBSD. Our jailed customers don't have to worry about this because /etc is already in the right spot by the time jail runs /etc/rc. /sbin/init isn't even involved in a jail, is it? Not even in some hooked-in way? At any rate we dont have to do anything special for a separate /etc dataset for jails. We could just forgo the /etc dataset on the host but i am glad that we can manage our bare metal customers using the same methods and tools. Handling this symlink hack is less differentiation than giving up separate /etc on the host i think. I'm interested in your cost/benefit points of having /etc a separate filesystem. On the face of it, I want to say that /etc is (or at least contains) the core identity of the machine (and to a lesser extent -- because this is BSD after-all -- /usr/local/etc). In my mind, /etc and /usr/local/etc *are* the machine (metaphorically speaking), so the merits of having it as a separate filesystem are weighed against your desired topology. i agree. myself i like having such a lightweight identity and keeping /, /usr and /usr/local (which are all just on sitting on / in my case) mounted read-only. the prototype for a host is handled by a completely different department than the people/customers who sysadmin their deployments and instances. Early in the building/installing, before any ports/packages, /usr/local/etc is made a symlink to /etc/local, so the symlink is in the readonly / and every time you write or cd to /usr/local/etc you end up in /etc/local. An /etc dataset ends up under a MB zfs compressed and /var on a fresh instance is basically also nothing. all-in-all a new jail costs you under a MB of zpool. we jail stop/start and zfs send/receive instances in a blink of an eye and its almost as good as having live migration. We could get the same storage efficiency by simply cloning /, and having no sub-datasets. some customers feel like they want to be able to write anywhere and we give them those options but then they are on their own and we don't manage the software updates for those guys and some like it that way. we then bill each for all the storage they reference because a year down the road they may be the only one still holding a reference to the outdated prototype they're on even though they overwrote every file twice with make world or freebsd-update. their memory usage is also way higher than most because when executables are launched on the jails with the read-only nullfs mounted /, those all access the same memory pages but zfs isnt smart enough yet to let the virtual memory system maintain those pointers through the indirection of zfs clones and snapshots. so zfs separate /etc and /var give us great storage efficiency while nullfs gives us great memory performance and efficiency. If you want to bunch of machines to look and/or act differently, then a shared /etc is precisely what you want. However, without allowing minor changes (ala ZFS clone/snapshot or by way of UnionFS), you'll quickly find that the only way to cope is with role-based scripting in /etc/rc.conf (it is after-all a shell script) or complicated abstraction layers (for example, using netgraph eiface devices with the jail-name inside them so that rc.conf have have jail-specific ifconfig_* lines). But I digress. I think the better solution to your loading of files beneath the eventual /etc filesystem is to throw away the ZFS snapshot/clone method and instead move to a UnionFS approach for /etc. If you use UnionFS for your /etc, then what you do is for each of the machines that you want *that* /etc to appear, you do something like: (as
Re: Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On 07/08/13 00:12, Teske, Devin wrote: On Jul 7, 2013, at 2:58 PM, Chad J. Milios wrote: [snip] /etc is now a ZFS dataset of its own How did we do it? Decades of conventional wisdom says /etc must be on /. Check it out, discuss the whys and the trade-offs. Well, I see in nu_install on GitHub how you're doing it: You added: init_script=/boot/init.sh to /boot/loader.conf, wich -- among other things -- does these two interesting things (variable names changed to make things more clear): zfs rollback -r $zfs/swap/host@blank NOTE: $zfs is equal to $( /bin/kenv vfs.root.mountfrom ) minus the leading zfs: and zfs mount $zpool/etc NOTE: $zpool is equal to $zfs from above, leading up to (but not including) the first slash (/). Cute. Have to say I wasn't aware of the init_script feature of loader.conf(5). Not bad. We also had to put one file into the etc directory on the / beneath the /etc mount so that /sbin/init can read it before /etc is mounted. There were two or three ways we could do that and each has a tradeoff. What we did (mv /etc/login.conf.db /boot/etc; ln -s ../boot/etc/login.conf.db /etc/login.conf.db) has the undesirable effect that one must remember to (or be reminded/automated) run cap_mkdb anytime /etc is rolled to a different snapshot or a backup is restored to it (if that changes login.conf). With our customers at ccsys.com we have a proprietary management thing in userland (and you could lose out on that important event hook if you used anything other than our interface for zfs rollbacks and restoring backups, which we forbid). Since our goals at nuos.org are different, i'd like to implement that trigger somewhere better, ideally as-needed and immediate as possible. if anyone with more intimate knowledge of how and exactly when login.conf.db gets accessed has any thoughts... It could be a disaster for an admin to think their /etc is in a certain state and have that one file be out of sync. If better minds could chip in, I'm wondering if we're better off editing /sbin/init to run init_script _before_ loading the daemon class from login.conf.db (or explain why thats a bad idea) or if i should just add some sort of hook to run cap_mkdb right when needed using a DTrace script or auditd? Does anyone think this issue is moot? (Can't we just document this particular specific gotcha instance? I don't think so, I abhor any gotcha that deviates from behavior people expect from upstream fbsd.) Does anyone agree it's important we come as close to perfect a solution as we can? Is a separate /etc even worth it to people? Should i scrap that feature because of this issue? I think we can tighten this up so theres no twisted ankles and no one falling in this rare case but certainly potential manhole. (the manhole i'm talking about is login.conf and login.conf.db being out of sync because the later is a symlink to /boot/etc and someone might rollback to a more restrictive login.conf and think they're covered without running cap_mkdb again but their login.conf.db is actually out of sync and less restrictive in a way that burns them) Devin, thank you IMMENSELY for bsdinstall and especially bsdconfig. I use them both at work and they make life so much better. And thank you for the simplification using kenv. I was unaware of it ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
PLEASE reply to this only in freebsd-chat. I have posted this announcement to five freebsd mailing lists, I hope I am not overstepping. Hello everybody. My name's Chad J. Milios. Long-time lurker, sparse rare sporadic poster. TL;DR? -- Skip below to our summary of features in an outline format then grab it at http://nuos.org . I would like to enthusiastically announce the release of an open-source project of much pride and passion of my good friend Scott C. Ziegler and myself which we have brought forth thanks to the support and contributions of about 15 others. I believe it is solidly ready to be shared with the world in the hopes that others may help build out the software and community in a way that promotes quality, innovation and collaboration much like FreeBSD has led the open-source community at doing. The nuOS project ( http://nuos.org ) is about bringing back the power to the people! Currently, technical software, hardware and networking power. Ultimately, the power of personal communication and community self-organization. Currently made by geeks/nerds/hackers for geeks/nerds/hackers, our intent is to create an entirely new software ecosystem that promotes quality, easy to use software that is for any-and-every man woman and child yet without lassoing us all into one herd of sheeple. ;) Simple, common things should always be EASY. Complex, amazing or never-before imagined things should always be POSSIBLE. We have a live image for download from our site. (Fully functional at 189 MB, just cat or dd to your 4 GB or larger usb drive or select it as a flat-file virtual disk in your hypervisor of choice. It is not an ISO and nuOS does not work well from optical media.) Or grab our source (currently hosted by GitHub at https://github.com/CropCircleSys/nuOS ) and build the entire system from any FreeBSD 9.1 system with one simple yet deeply customizable command. (We only build/test on amd64 and would like that to change in the future.) It is my belief that our software is PRODUCTION READY with our new beta release. It might just be the answer to the management headaches you may be having. Take the plunge tonight and find yourself breezing through your day-job with nu-found ease tomorrow morning. If you're the comfortable yet cautious type, watch the discussion for a week or two first instead. Either way, we intend to cause a positive large and lasting motion in the FreeBSD community. I hope you will give nuOS a look and offer your assessments and ask any questions you have. Please tear it and us apart in discussion with the goal of a better FreeBSD for us all! Documentation is one area that is sorely lacking though it is mostly because Scott and I consider most of our code clear enough to have been pretty self-documenting [for our purposes we've had until now]. It is our hope that with the community's help we will bring more and more of this platform to the high standard of quality that FreeBSD is known for. We aren't trying to create our own new garden. We offer this code with hopes that it, in part or in whole, might be some day included in canonical FreeBSD releases. We have NO intention on forking FreeBSD and are instead developing a very lightweight suite of tools which hopefully capture and collect modern best practices while providing a testing and proving ground for advanced FreeBSD features. We want to bring computing to more people, bring more computer users to open source, bring more high-value and responsible open-source users to FreeBSD and bring more current FreeBSD users guidance and enlightenment regarding advanced features in the face of FreeBSD's typical adherence to maximal backward compatibility, legacy support and solid ground yet sometimes daunting array of intimately detailed configuration choices. We do not seek to limit those choices or to shift the ground beneath current FreeBSD users' feet. We seek to offer an alternative flavor of default system for those interested in taking a step back from their current perspective in order to take a giant flying leap forward. This doesn't mean giving up anything in terms of compatibility or configurabilty, quite the contrary. Throughout our evolution, we seek to always maintain the environment that FreeBSD users have come to know and love while reducing the issues that sometimes irk them. We simply seek to provide a better way to structure, provision and maintain production systems and development processes. Outline of features: Extends plain old FreeBSD 9.1 (RELEASE or STABLE) and maintains total compatibility We seek to remain nimble Expect a production-ready seal of approval to lag behind releases by no more than a week or two and prebuilt images and packages e.g. releases like 9.2 and 10.0, et al Someone should be able to build it and use all applicable features on 8.4 with ease we simply haven't the time
Re: Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On 07/07/13 22:05, C. Bergström wrote: trolling side comment omg you've created Solaris /trolling side comment If you're going to spam commercial stuff with absolutely no technically interesting details - please keep it brief at the least. Generally people will be curious about What are you actually adding to the ISO which FBSD-current can't do? If it's not upstream already - will it be contributed upstream? Please reply further on freebsd-chat, I'd like to consolidate any discussion this may garner. This doesn't provide anything to the core OS that can't already be done, albeit with many more keystrokes and the peril of possible confusion and misconfiguration. The main thing here is a collaboration of what we consider best practices and consolidating the more useful configurations into consistent recipes with useful simplification of parameters. We don't mean to add yet another layer in the name of simplicity that obscures or hides the real nuts and bolt beneath and limits your options. We want to make things more flexible and easier at the same time by using the sanctioned FreeBSD ways of doing things, simply allowing the ones with most merit to rise to the top, hopefully through community involvement. We've had a lot of success using this in our production deployments and hope that we don't have to be the only ones to maintain it forever. It is an open offer of contribution to The FreeBSD Project but it probably doesn't exactly belong there yet. It's a layer above, so to speak, and we think we have a place in the community working side by side. It's a distro around FreeBSD, think picoBSD or maybe FreeNAS. It's not going to be a fork like PC-BSD or Dragonfly. I'm hoping we can be a proving ground for the more advanced features of FreeBSD, by allowing more users to jump on board with them sooner, and then offer the applicable bits and pieces back upstream while continually pushing the innovation envelope in a way that more people and companies can participate in. The tool nu_install is basically sysinstall on steroids. It doesn't do all the things that sysinstall does and you may still use sysinstall to configure a system or a jail you've provisioned with nu_install or nu_jail. nu_install automates a process of building a ZFS only FreeBSD system and offers a default dataset layout featuring best practices we've deduced from using ZFS on FreeBSD since its infancy and reading and considering many various differing and conflicting ZFS on root how-tos. For instance, many ZFS on root tutorials use a UFS /boot partition and/or mountpoint=legacy and entries in /etc/fstab. We suffer neither of those holdovers. Another feature I've not yet found in any tutorial is /etc having its own dataset. nu_jail creates cloned datasets and jail.conf entries along the school of thought set out by our nu_install base system. Jails in FreeBSD allow many use cases that were never dreamed of on other platforms and we don't seek to force any particular cookie-cutter way of provisioning a jail, just simplifying the uses that we've found most common. We wanted ease and simplicity but refused to give up less-common possibilities or give up the simplicity just to tweak something a little differently to do something that's never been done. Thank you for reading and offering your thoughts. LOL @ the Solaris comment, as I am a long-time Solaris user and fan but always been a bigger fan of the BSDs and FreeBSD mostly in particular for the last decade. In short, we seek to do with FreeBSD something like what Joyent has done with illumos in their SmartOS but then continue further with that idea. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
