Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
On Wed, Jun 20, 2012 at 1:40 AM, Andres Perera wrote:
> sorry, but i never sold nm as the sole step granting immunity. i
> explicitly presented it as an example. nevertheless, the full list of
> things i do do not cover all of possible changes you pointed out. i
> constructed it in a way that also works with snapshots:
>
> diff include/sys/syscall{args,}.h with previous db (a la sysmerge);
> double check with with nm /bsd. syscallargs changing returns false
> whether or not nm shows the same set of calls.
>
> (i don't currently diff /sys/* in hopes of finding new or changed bitmap
flags)
i am talking about include/sys, not the kernel source repository
>
> diff include/sys/ioctl.h and header-includes with previous db. i don't
> attempt to detect new includes, this is fragile and is covered by
> acting on sys/*
>
> the rest of files are predictable sets of other kernel apis. i don't
> look at net/pfvar or anything outside sys even though i should
>
> as flaky as it can be, it works most of the time and it's better than
> "let the user decide"
>
> On Wed, Jun 20, 2012 at 12:59 AM, Matthew Dempsky
wrote:
>> On Tue, Jun 19, 2012 at 9:34 PM, Andres Perera wrote:
>>> all of the calls in syscalls.master map to a unique function, and all
>>> of them start with sys_. it's true that nm won't tell me about
>>> argument changes. i just risk it a little by assuming no one's that
>>> evil
>>
>> Okay, granted nm will tell you when new syscall entry points get
>> added... but you won't know about new syscall flags, new ioctls, new
>> device nodes, new sysctls, new behavior, etc.
>>
>> Not saying you can't use nm as a backup sanity check, but it's not
>> something I'd recommend relying on by default. Our userland is really
>> not designed to run on older kernels.
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
sorry, but i never sold nm as the sole step granting immunity. i
explicitly presented it as an example. nevertheless, the full list of
things i do do not cover all of possible changes you pointed out. i
constructed it in a way that also works with snapshots:
diff include/sys/syscall{args,}.h with previous db (a la sysmerge);
double check with with nm /bsd. syscallargs changing returns false
whether or not nm shows the same set of calls.
(i don't currently diff /sys/* in hopes of finding new or changed bitmap
flags)
diff include/sys/ioctl.h and header-includes with previous db. i don't
attempt to detect new includes, this is fragile and is covered by
acting on sys/*
the rest of files are predictable sets of other kernel apis. i don't
look at net/pfvar or anything outside sys even though i should
as flaky as it can be, it works most of the time and it's better than
"let the user decide"
On Wed, Jun 20, 2012 at 12:59 AM, Matthew Dempsky
wrote:
> On Tue, Jun 19, 2012 at 9:34 PM, Andres Perera wrote:
>> all of the calls in syscalls.master map to a unique function, and all
>> of them start with sys_. it's true that nm won't tell me about
>> argument changes. i just risk it a little by assuming no one's that
>> evil
>
> Okay, granted nm will tell you when new syscall entry points get
> added... but you won't know about new syscall flags, new ioctls, new
> device nodes, new sysctls, new behavior, etc.
>
> Not saying you can't use nm as a backup sanity check, but it's not
> something I'd recommend relying on by default. Our userland is really
> not designed to run on older kernels.
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
On Tue, Jun 19, 2012 at 9:34 PM, Andres Perera wrote: > all of the calls in syscalls.master map to a unique function, and all > of them start with sys_. it's true that nm won't tell me about > argument changes. i just risk it a little by assuming no one's that > evil Okay, granted nm will tell you when new syscall entry points get added... but you won't know about new syscall flags, new ioctls, new device nodes, new sysctls, new behavior, etc. Not saying you can't use nm as a backup sanity check, but it's not something I'd recommend relying on by default. Our userland is really not designed to run on older kernels.
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
and that will be an exception that i'll have to deal with, which is entirely reasonable given that they rarely do change another rare exception i could skirt around would be white space changes that would deter me from diffing syscalls.master instead of `nm /bsd` during automation, but the problem doesn't even come to that with snapshots, since i don't have a source referral; i only have the binary interface of the symbol list On Wed, Jun 20, 2012 at 12:18 AM, Philip Guenther wrote: > On Tue, Jun 19, 2012 at 9:34 PM, Andres Perera wrote: >> all of the calls in syscalls.master map to a unique function, and all >> of them start with sys_. it's true that nm won't tell me about >> argument changes. i just risk it a little by assuming no one's that >> evil > > Heh. *Yesterday* tedu asked me to add some backwards compat to a diff > I set around that did exactly that, changing the argument list for an > existing syscall. I guess I'm winning the evil contest with tedu! > > > Philip Guenhter
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
On Tue, Jun 19, 2012 at 9:34 PM, Andres Perera wrote: > all of the calls in syscalls.master map to a unique function, and all > of them start with sys_. it's true that nm won't tell me about > argument changes. i just risk it a little by assuming no one's that > evil Heh. *Yesterday* tedu asked me to add some backwards compat to a diff I set around that did exactly that, changing the argument list for an existing syscall. I guess I'm winning the evil contest with tedu! Philip Guenhter
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
since packages are done in synch with snapshots, i do not use the trees because i rather use packages it's not clear whether or not changes in snapshots are allowed to make the packages incompatible with what you find in the repositories. perhaps i would be able to retract what i said as silly (and benefit from knowing exactly what is it i'm running at the same time) On Tue, Jun 19, 2012 at 9:24 PM, Theo de Raadt wrote: >> never mind the premise that snapshots contain changes not found in the >> trees, you state things to the effect of "user chooses wether or not >> to reboot to new kernel". didn't even bother; e.g., comparing nm >> outputs > > well, hang on. quite often those diffs in snapshots are not yet > commited for a reason. > > those diffs are being tested by people brave enough to test snapshots. > of course, if people are brave enough to test snapshots, and any last > minute bugs are found in those diffs and fixed.. and everyone will be > able to run those juicy bits earlier. > > the diffs in snaps are chosen by me to try to advance so that i can > help that process ahead (but at the same time not drive myself > insane). after all, if i pick the wrong diffs at the wrong time, i > going break all of the build machines at the same time...
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
all of the calls in syscalls.master map to a unique function, and all of them start with sys_. it's true that nm won't tell me about argument changes. i just risk it a little by assuming no one's that evil On Tue, Jun 19, 2012 at 9:22 PM, Matthew Dempsky wrote: > On Tue, Jun 19, 2012 at 5:44 PM, Andres Perera wrote: >> didn't even bother; e.g., comparing nm >> outputs > > Er, what are you expecting to divine by comparing nm output?
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
On Tue, Jun 19, 2012 at 11:00 PM, Ted Unangst wrote: > On Tue, Jun 19, 2012 at 21:41, eagir...@cox.net wrote: > >> What may be a slightly faster method of tracking close to current: >> >> http://www.tedunangst.com/snapper.html >> >> I haven't used it in a while, because I used to build the kernel with NTFS >> support, and never got back to using it after that became part of GENERIC. > > I had a report that amd64 may not be working? i386 is, at least > assuming you catch a current snap. I've given up on trying to keep > pkgs up to date, though, it requires an insane amount of disk space. > I used bluesnapper for amd64 twice today and worked just fine. Luis.
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
On Tue, Jun 19, 2012 at 21:41, eagir...@cox.net wrote: > What may be a slightly faster method of tracking close to current: > > http://www.tedunangst.com/snapper.html > > I haven't used it in a while, because I used to build the kernel with NTFS > support, and never got back to using it after that became part of GENERIC. I had a report that amd64 may not be working? i386 is, at least assuming you catch a current snap. I've given up on trying to keep pkgs up to date, though, it requires an insane amount of disk space.
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
At 2012-06-20 0:00:21, Tony Sidaway wrote: Summary: I want to turn my main system into a semi-automatic follower of "-current" and I think this strategy may useful to the project. Is this something that is already being done? My rationale here is that it's a good thing for OpenBSD users who have the technical skills to follow development as closely as possible, Running from a tightly synchronized copy of "-current" enables the user to produce the most useful bug reports in a timely manner. Seeing a list of CVS updates also helps the user to understand how the project is ticking. While not a system expert, I've got a lot of application development skills to offer but I'm also lazy enough to want to script the laborious process of following "-current". I've searched for automated update tools for "-current" but I don't see what I think should be there. What I have in mind is a layered set of tools that keeps the /usr/src, /usr/xenocara and /usr/ports trees up to date by regular synchronization, then builds a kernel if a successful sync occurs. I have enough slack time to make this easy on my main system. The idea is that the system regularly (nightly) synchronizes all three main source directories, then rebuilds and installs the latest GENERIC kernel if synchronization is successful. As owner I can decide whether or not to reboot into the new kernel. I then have the option of starting a rebuild of the userland to synchronize it with the kernel. The same procedure can perhaps try to sync the installed packages (# pkg-add -u). Perhaps also an automated script to rebuild installed packages from the synchronized ports tree. This would enable users like me to quickly check our bugs against the latest build with kernel, userland and ports all synchronized thus encouraging us to make a bug report in the knowledge that it will be useful. If there is an RSS feed for the "Following current" page that can be folded in. I've got a prototype that tries to do most of these steps. Am I reinventing the wheel? Does this kind of thinking fit in with OpenBSD project requirements? Please let me know. I'm interested in helping OpenBSD in any way I can. Up to now I've followed Snapshots, but I find that less than satisfactory because from that point of view the development process is removed and rather opaque. My scripts enable me to watch the workflow across the project, and give me a feeling of involvement that I could not get from upgrading from binary image every few days. >> What may be a slightly faster method of tracking close to current: http://www.tedunangst.com/snapper.html I haven't used it in a while, because I used to build the kernel with NTFS support, and never got back to using it after that became part of GENERIC. -- Ed Ahlsen-Girard Ft. Walton Beach FL
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
On Tue, Jun 19, 2012 at 5:00 PM, Tony Sidaway wrote: > Summary: I want to turn my main system into a semi-automatic follower > of "-current" and I think this strategy may useful to the project. Is > this something that is already being done? That's more or less what the snapshot process is, except that Theo and the others who manage it know the extra steps that needs to be done to make sure the builds are done correctly. That said, I have a somewhat similar semi-automatic build process that I use for sanity testing diffs in a chroot before installing them to my system (it's only fun repairing a broken ld.so or libc.so so many times), and I know at least a few other devs do so too. I'd love to see some more automation here especially if it could integrate with the regress tree. If someone could write some scripts to boot qemu for various supported arches, automatically install a snapshot from sets, and then let you run some commands on it after it reboots... that would be super handy IMO.
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
> never mind the premise that snapshots contain changes not found in the > trees, you state things to the effect of "user chooses wether or not > to reboot to new kernel". didn't even bother; e.g., comparing nm > outputs well, hang on. quite often those diffs in snapshots are not yet commited for a reason. those diffs are being tested by people brave enough to test snapshots. of course, if people are brave enough to test snapshots, and any last minute bugs are found in those diffs and fixed.. and everyone will be able to run those juicy bits earlier. the diffs in snaps are chosen by me to try to advance so that i can help that process ahead (but at the same time not drive myself insane). after all, if i pick the wrong diffs at the wrong time, i going break all of the build machines at the same time...
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
On Tue, Jun 19, 2012 at 5:44 PM, Andres Perera wrote: > didn't even bother; e.g., comparing nm > outputs Er, what are you expecting to divine by comparing nm output?
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
ultimately naive/incomplete approach never mind the premise that snapshots contain changes not found in the trees, you state things to the effect of "user chooses wether or not to reboot to new kernel". didn't even bother; e.g., comparing nm outputs
Re: Following -current through a semi-automatic process: a strategy for encouraging user involvement?
Hi Tony, Tony Sidaway wrote on Wed, Jun 20, 2012 at 01:00:21AM +0100: > Summary: I want to turn my main system into a semi-automatic follower > of "-current" and I think this strategy may useful to the project. Is > this something that is already being done? No. The main reason being that following -current just isn't semi-automatic. There are times when updating is quite safe (like, usually). There are times when updating is more risky (say, near the end of a large or invasive hackathon, or when art@ just committed some three-line diff). There are times when updating requires special changes to be applied manually, see faq/current.html. Nobody can tell you whether, at any given time, it's a good time to update *your* particular machine or whether you should better wait a few days for some dust to settle. Running -current means that you should be able to judge that for yourself, depending on what you are using that particular machine for. On a workstation, it probably doesn't matter. If you misjudge and it breaks, well, you fix it. On a critical production server that makes your boss panic when you break it, you shouldn't run -current unless you feel confident that you can properly judge when to update (and when not to). > The idea is that the system regularly (nightly) synchronizes all > three main source directories, then rebuilds and installs Gah. No! When updating a -current system for a user, use snapshots, do not build from source. When implemented as perfectly as possible, the process you are proposing will work in most cases, but occasionally, it will leave you with a badly broken system. Updating from source cannot be fully automated. Feel free to fool around with compiling from source for experimentation and learning, but don't rely on it for production. > I'm interested in helping OpenBSD in any way I can. Look out for anything that gets in your way (bugs, missing features). Try to fix those and send patches. Start with small and simple things. Do not try to change the whole system all at once. Do not start by changing the build system. Yours, Ingo
Following -current through a semi-automatic process: a strategy for encouraging user involvement?
Summary: I want to turn my main system into a semi-automatic follower of "-current" and I think this strategy may useful to the project. Is this something that is already being done? My rationale here is that it's a good thing for OpenBSD users who have the technical skills to follow development as closely as possible, Running from a tightly synchronized copy of "-current" enables the user to produce the most useful bug reports in a timely manner. Seeing a list of CVS updates also helps the user to understand how the project is ticking. While not a system expert, I've got a lot of application development skills to offer but I'm also lazy enough to want to script the laborious process of following "-current". I've searched for automated update tools for "-current" but I don't see what I think should be there. What I have in mind is a layered set of tools that keeps the /usr/src, /usr/xenocara and /usr/ports trees up to date by regular synchronization, then builds a kernel if a successful sync occurs. I have enough slack time to make this easy on my main system. The idea is that the system regularly (nightly) synchronizes all three main source directories, then rebuilds and installs the latest GENERIC kernel if synchronization is successful. As owner I can decide whether or not to reboot into the new kernel. I then have the option of starting a rebuild of the userland to synchronize it with the kernel. The same procedure can perhaps try to sync the installed packages (# pkg-add -u). Perhaps also an automated script to rebuild installed packages from the synchronized ports tree. This would enable users like me to quickly check our bugs against the latest build with kernel, userland and ports all synchronized thus encouraging us to make a bug report in the knowledge that it will be useful. If there is an RSS feed for the "Following current" page that can be folded in. I've got a prototype that tries to do most of these steps. Am I reinventing the wheel? Does this kind of thinking fit in with OpenBSD project requirements? Please let me know. I'm interested in helping OpenBSD in any way I can. Up to now I've followed Snapshots, but I find that less than satisfactory because from that point of view the development process is removed and rather opaque. My scripts enable me to watch the workflow across the project, and give me a feeling of involvement that I could not get from upgrading from binary image every few days.
