On Oct 10, 2013, at 10:18 AM, Allan Jude wrote: > On 2013-10-10 11:30, Teske, Devin wrote: >> On Oct 10, 2013, at 4:55 AM, Nathan Whitehorn wrote: >> >>> On 10/10/13 09:20, Allan Jude wrote: >>>> On 2013-10-10 03:00, Nathan Whitehorn wrote: >>>>> On 10/09/13 18:55, Teske, Devin wrote: >>>>>> On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote: >>>>>> >>>>>>> On 10/09/13 01:13, Allan Jude wrote: >>>>>>>> On 2013-10-08 16:17, Nathan Whitehorn wrote: >>>>>>>>> On 10/07/13 21:59, Allan Jude wrote: >>>>>>>>>> Devin Teske and I have been working on a big patch to bsdinstall to >>>>>>>>>> implement installing on a ZFS pool. It supports both GPT and MBR, >>>>>>>>>> the 4k >>>>>>>>>> sector gnop trick, and optional GELI encryption. We would like to >>>>>>>>>> commit >>>>>>>>>> this in time for 10.0-BETA1 so it needs some testing to work out any >>>>>>>>>> obvious bugs before we send it off to re@ to get it committed. >>>>>>>>>> >>>>>>>>>> It includes a single configuration menu that allows you to select >>>>>>>>>> all of >>>>>>>>>> the required details, including which drives to use (gets details >>>>>>>>>> from >>>>>>>>>> camcontrol, also includes an inspection utility that presents the >>>>>>>>>> detailed output of camcontrol inquiry/identify, and gpart show), what >>>>>>>>>> ZFS RAID level to use (taking in to consideration the selected >>>>>>>>>> number of >>>>>>>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Additional, it includes some other changes to bsdinstall: >>>>>>>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt >>>>>>>>>> to no >>>>>>>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with >>>>>>>>>> just 1 >>>>>>>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this >>>>>>>>>> feature has been combined into the regular 'services to enable' >>>>>>>>>> dialog >>>>>>>>>> and enabled by default >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> You can browse the patches here: >>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0A&s=6361ad5ccc04a6d8401158a3ee1b3d8df9fcd2ef47ff6ad6365a57b907cceaec >>>>>>>>>> >>>>>>>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, >>>>>>>>>> available compressed (48 MB) or uncompressed (211 MB): >>>>>>>>>> >>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0A&s=570daea25d9e629788c76c2b5a6eae19d32050363986a1004a6434c3466965c2 >>>>>>>>>> >>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0A&s=f52d604a2503a48b409f92f8376bb773de4dbc469d1e74b16d051a20ad894820 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We look forward to your feedback >>>>>>>>>> >>>>>>>>> Thanks for doing this! I had a few comments: >>>>>>>>> 1. ZFS is not bootable on all architectures. Could you adjust that >>>>>>>>> menu >>>>>>>>> item to only display for i386, amd64, and (I think?) sparc64. Use >>>>>>>>> uname >>>>>>>>> -m, not -p, for this. >>>>>>>> I had not considered that, I'll make that change >>>>>>>> >>>>>>>>> 1a. The script is broken on sparc64 in any case, which uses VTOC8 >>>>>>>>> instead of GPT. >>>>>>>> I'll disable sparc64 as well >>>>>>>> >>>>>>>>> 2. Why are you using camcontrol? That is guaranteed not to work on >>>>>>>>> non-CAM systems. You should use the GEOM ident string if you need an >>>>>>>>> ID. >>>>>>>> The GEOM ident string doesn't do enough to help the user identify which >>>>>>>> drive is which. >>>>>>>> More data is not exposed anywhere that I could find >>>>>>>> >>>>>>>> What we really need, is dev.ada.0.desc% like we have for network >>>>>>>> interfaces and a slew of other devices. GEOM data is great, but it is >>>>>>>> not exposed in a shell friendly way any place that I could find, other >>>>>>>> than the sysctl with DOT and XML data. >>>>>>> This is one of the reasons the partition editor is written in C. There >>>>>>> are a few other odd corner-cases where C is much more powerful than the >>>>>>> command-line for the GEOM operations that partedit needs to do. I'm not >>>>>>> sure how to usefully get it just from the shell. You can see how to do >>>>>>> it in C in the boot_disk() routine of partedit/auto_part.c. >>>>>>> >>>>>>>>> 3. Any plans to integrate this into the regular partition editor? ZFS >>>>>>>>> support is important enough that I will definitely not get in the way, >>>>>>>>> even as a bolt-on, but it would be a shame for it to stay that way. >>>>>>>>> The >>>>>>>>> editor is also designed for ZFS to be added. >>>>>>>> I am a sysadmin, not a programmer. I can't write C. Most people >>>>>>>> deploying servers can't write C. I agree with Devin Teske, if >>>>>>>> everything >>>>>>>> was in shell it would be a lot more usable for non-developers, who >>>>>>>> probably make up the majority of people who deploy FreeBSD. >>>>>>> There are some cases the other way too. Devin is probably the most >>>>>>> shell-proficient FreeBSD committer. >>>>>> Well, there's jilles ;D (he writes/maintains the sh(1) implementation >>>>>> itself). >>>>>> >>>>>> >>>>>>> I certainly can't write shell >>>>>>> scripts at that level, which means that for me bsdconfig, for example, >>>>>>> is effectively read-only (and quite hard to read as well). >>>>>> In the past few days of working on the bsdinstall_zfs patch-set with >>>>>> Allan >>>>>> Jude, I learned something new actually. >>>>>> >>>>>> It seems that sh(1) doesn't suffer so much from this "read only" concept. >>>>>> >>>>>> Imagine you're starting a new C project on an operating system that has >>>>>> all new syscalls and all new APIs that you've never seen. Would be pretty >>>>>> hard, no doubt. Now imagine that you're thrown a life-line called POSIX >>>>>> and hey, now you're slinging code in MinGW on Windows when you don't >>>>>> know a lick of M$ syscalls or APIs. >>>>>> >>>>>> In a similar manner, I've witnessed functionality be added that is truly >>>>>> functional *without* using any of the API calls. Then I come in and do >>>>>> a round of optimizations to leverage the existing API. >>>>>> >>>>>> Shell is kinda like that... >>>>>> >>>>>> I'm noticing Allan Jude doesn't know all the API calls yet (who could? >>>>>> other >>>>>> than me of course -- and even I have trouble remembering them _all_) yet >>>>>> this doesn't phase him (or others) from jumping in and lending a hand. >>>>>> >>>>>> No different than C, but the read-only aspect is lessened significantly I >>>>>> believe because there are so many people out there that know enough >>>>>> good shell syntax and are just a stone's throw away from *great* shell >>>>>> syntax (which I must admit jilles helped me cross that boundary). >>>>>> >>>>>> I think in C, the read-only aspect is greater because its harder to >>>>>> parcel-out >>>>>> the functions for a unit-test; harder to inject new code; and harder to >>>>>> get >>>>>> to a functioning end-state. >>>>> Look, I have no doubt that in the right hands shell can do amazing >>>>> things and C can be badly written. That isn't the issue though. My >>>>> statement was purely that most FreeBSD developers (me included) are more >>>>> comfortable with C than shell when used at the kind of level involved >>>>> here. This is not a value judgment but a statement of fact. Whether or >>>>> not, at some platonic level, shell or C or python or whatever are more >>>>> or less read-only is not the point here. The point is that I, and I >>>>> suspect many other developers, cannot write (or read) very advanced >>>>> levels of shell scripting but can read and write the equivalent programs >>>>> when written in C in many cases. It's what the rest of the system is >>>>> written in and what we spend most of our time using. Whether this should >>>>> be the case or not is immaterial; the fact remains that shell scripting >>>>> at any but a very basic level does introduce a very large barrier to >>>>> entry for probably a large majority of committers. >>>>> >>>>> This is not always a problem -- especially if using something more >>>>> obscure allows very active development by the set of people working on >>>>> it -- but does reduce the set of people who can make modifications >>>>> substantially. I have no ability to change, or understand, most of >>>>> bsdconfig, for example. This isn't a problem since you are doing all the >>>>> work and there is no reason I would need to or want to make changes to >>>>> it. But it could become a problem in a part of the system to which >>>>> multiple people needed to contribute. It's about other people's comfort >>>>> zones and knowledge in the end. >>>>> -Nathan >>>>> _______________________________________________ >>>>> freebsd-current@freebsd.org mailing list >>>>> https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/mailman/listinfo/freebsd-current&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0A&s=229c04e0b0ce43bcdc88819dc4a9cba8ffa47ef43ec9ad5352019bd7c850feff >>>>> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" >>>> I definately see your point, but the code going in to the zfsboot part >>>> in particular, is not that advanced. I started this project two weeks >>>> ago with a well below average knowledge of shell script, a minor bit of >>>> experience in php, and no real programming experience to speak of. >>>> >>>> I wrote most of the code that actually does things, Devin just cleaned >>>> it up (I didn't know anything about shstyle or the rules) and pointed me >>>> at some handy subroutines he had written to avoid writing out 20 lines >>>> of code every time I wanted to display a dialog box >>>> >>>> I wouldn't say any of what is in the proposed patches to bsdinstall is >>>> unmaintainable, and it definately makes it much more approachable for >>>> sysadmins who are the ones that actually need to be able to script >>>> bsdinstall, and generate various methods of automated deployment. This >>>> is a very important feature if we want FreeBSD to be taken seriously for >>>> deployment in mass virtualization environments (do not make my say >>>> cloud. we saw a great sign in Malta at some street festival "what is >>>> cloud?") >>> Sorry, I think you misunderstood me. I really appreciate your work, and >>> the ZFS stuff, as I said, is not that complex so there's no problem >>> there. Aside from a couple of issues I mentioned (with sparc, etc.), it >>> seems like a very good step forward. What I *was* trying to do was >>> discourage you from rewriting partedit as a shell script and to >>> encourage unifying the ZFS backend into partedit at some point in the >>> future significantly different from now. >>> >>> I also wanted to mention that bsdinstall is completely scriptable >>> (although not the new ZFS code, I guess...) >> Oh, but it is. >> >> The following variables have sensible defaults and are designed >> to be exported in a bsdinstall script to customize the actions >> performed by the zfsboot script: >> >> dte...@scribe9.vicor.com scripts $ grep '^: ' zfsboot >> : ${ZFSBOOT_POOL_NAME:=zroot} >> : ${ZFSBOOT_BEROOT_NAME:=bootenv} >> : ${ZFSBOOT_BOOTFS_NAME:=default} >> : ${ZFSBOOT_VDEV_TYPE:=stripe} >> : ${ZFSBOOT_GNOP_4K_FORCE_ALIGN:=1} >> : ${ZFSBOOT_GELI_ENCRYPTION:=} >> : ${ZFSBOOT_GELI_POOL_NAME:=bootpool} >> : ${ZFSBOOT_GELI_BOOT_SIZE:=2g} >> : ${ZFSBOOT_GELI_KEY_file:///=/boot/encryption.key} >> : ${ZFSBOOT_DISKS:=} >> : ${ZFSBOOT_PARTITION_SCHEME:=GPT} >> : ${ZFSBOOT_SWAP_SIZE:=2g} >> >> Notice the ZFSBOOT_ prefix matching the script name. >> These may not be the final names used as an overall >> effort is made (given time) to conflate the scripting variables >> and general namespace. >> >> >> >>> and has been for some time. >> The type of scripting set forth by bsdinstall is not the same kind of >> scripting set forth by bsdconfig. >> >> That being said, I've not yet made an effort to close the gap on >> bsdinstall scripting (to make it like bsdconfig scripting). >> >> In bsdconfig, we can literally load a sysinstall "install.cfg" file and >> there is no need to for a complicated embedding system such as >> bsdinstall has (wherein the "script" is actually a multi-part file). >> >> A lot of people have scoffed at backward compatibility because it >> would not be easy. However, I see a pretty big carrot dangling in >> front of the backward compatibility effort... >> >> It's not the act of implementing the individual reswords that's going >> to bring the biggest value-add, but instead the bringing back of the >> process-flow to deployment that will see the greatest gain in >> functionality. By this, I mean that currently you invoke bsdinstall >> and it runs through a UX given a set of values set in variables. >> This is sub-optimal if you don't like the UX choices -- better to >> instead allow the script to take the reigns and do things by way >> of having the script drive (in sysinstall, this amounts to the fact that >> if "install.cfg" doesn't call diskPartitionWrite or doesn't call >> distExtractAll, then those steps are not performed). In bsdinstall, a >> change in focus to allow the script to drive *could* be done by >> requiring the user to fork-exec to a bunch of scripts -- and in-turn >> then requiring each/every custom setting to be *export*ed... but >> ultimately the best bang-for-your-buck is going to come from >> (I'll say it again) namespace conflation (writing the functionality >> as a series of includable functions that are run in the same >> namespace as the script that is driving). Naturally, a bunch of >> functions can be lost if you don't know where to look for them, and >> that comes back full-circle to `script.subr' and `variable.subr' as the >> "points of discovery" (besides a man-page; which is where sys- >> install documented its own reswords). >> >> >> >>> Only the initial release that shipped with 9.0 was not. There's a >>> lengthy discussion of how this works in the man page. >> *nods* >> >> >> >>> The scripting >>> support is extremely flexible (you can run an arbitrary shell script in >>> addition to the usual installation steps) and lets installations from >>> PXE or media operate completely unattended. >> Yes, but I've been taking to simply creating /etc/installerconf and >> completely bypassing the scripting that's documented in the man-page >> *because* I abhor multi-part file which requires a split-architecture >> approach to installation (no like). >> >> Sysinstall often required me to do the same thing. Whereas in >> sysinstall the problem was that you couldn't speak native shell in the >> install.cfg file (requiring you to call out to a "system" resword to >> execute a script which generates a side "install.cfg" later brought >> back in with the "loadConfig" resword, bsdinstall instead requires me >> to have -- in the multi-part script it loads -- a preamble and a setup >> script and this causes a similar dichotomy in passing information >> between the stages). >> >> In bsdconfig, I worked extremely hard to solve that issue. The end- >> result is that both problems are solved. The problem of sysinstall >> is gone because the "install.cfg" file becomes a shell script so you >> can then start speaking native shell in it. The multi-part problem is >> gone when you provide all the reswords in the native language >> (conflated namespace). > Devin, we should make a note to talk to Rick Miller (Verisign, host of > vBSDCon) while we are there. They still use sysinstall, and having > bsdinstall understand their sysinstall configs may be of great value to > them.
That's the plan ;D Been working with ol' Rick for years now on that journey. We keep in- touch and I keep apprising him of how things are going. Also, I've been sharing with him recipes for /etc/installerconf that used the legacy reswords but branch out to the couple parts of bsdinstall that are needed to do the disk management and dist extraction. -- Devin _____________ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. _______________________________________________ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
