Re: XO in-field upgrades
On Tue, 2007-06-26 at 17:47 -0400, Mike C. Fletcher wrote: > I may be missing an operation or two in there, but I *think* that's the > intention. That is, there's an ultimate loading level that sits outside > the standard root to make it all work. That loading level could be in > the BIOS or on an initrd, but there does need to be a level to manage > the ultimate overlay-based structure AFAIK. And at normal boot, > something has to set up the overlays, regardless of what overlay set is > loaded. That level of access is where the update manager has to sit to > be able to do the updates AFAICS. That is to accomplish a merge or > remount of the core fs we need a way to ask the overlay manager to do > some work for us at the update-manager level. Oh I see. So, you update the root, but you can't update all the system software (like the part that does the switching between roots). In such a setup, how do you update e.g. the kernel? > (how does that > work for soft links, incidentally, I gather you recreate them rather > than hard-linking?) They are recreated, same with fifos, directories, and device nodes. At least this is how update-manifest in updatinator does it, but I don't think there is any other way really. With update transaction on the filesystem level (like my jffs2 proposal) this kind of "outer" manager is not needed. However, with an outer controller I can see this working. One could even use symlinks to make this pretty simple: /rootfs/olpc.5 [contains image version 5] /rootfs/olpc.6 [contains image version 6, shared hardlinks with olpc.5] /rootfs/current -> olpc.6 [a symlink] /rootfs/previous -> olpc.5 /usr -> /rootfs/current/usr /var -> /rootfs/current/var /etc -> /rootfs/current/etc Then, to upgrade almost atomically, one just does: clone_w_hardlinks (/rootfs/olpc.6, /rootfs/olpc.7) apply_changes (/rootfs/olpc.7) ln -sf /rootfs/olpc.6 /rootfs/previous [1] ln -sf /rootfs/olpc.7 /rootfs/current [2] rm -rf /rootfs/olpc.5 A power failure during [1], or between 1 and 2 can mess up the "previous" link, and since symlink isn't an atomic operation if the file exists a failure during [2] can cause "current" to disappear. However, we will never have both "previous" and "current" missing. Using symlinks like this means the jffs2 parser in the firmware (if it supports symlinks) will even be able to pick up the right kernel. (Although it will always pick the "current" kernel.) There is one tricky area with hardlinks though. All hard links of the same inode share the permission and other metadata with each other. What if the trees have different metadata for the same file? Or what if running the "current" image changes metadata for a file shared between the images? Hmm, i guess any kind of in-place change to the hardlinked files while running the image is also mirrored in the other image. I guess this is where the COW stuff is needed... I guess this means the /usr -> /rootfs/current/usr symlinks don't cut it, but one needs something heavier, like overlays or vserver+COW. Sad, it was an interesting idea. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
Alexander Larsson wrote: > On Mon, 2007-06-25 at 15:38 -0400, Noah Kantrowitz wrote: > ... > I still don't understand at all how vserver can be used to upgrade the > main image. Yes, one can clone the main image into a chroot, and use > vserver to boot into that. One can also enable some COW stuff so that we > can change that into the new version. > > But, how do we apply this new image (which is a copy of the root + > applied updates) to be the new master image the next time we boot? i.e. > when the kernel boots it will launch /bin/init, > not /vserver/updated_image/bin/init. > Any COW/overlay system used for the root file system requires a boot-time operation that sets up the root file system image, similar to an initrd based system. This operation can be extremely simple with Ivan's two-version-only operation: if key_down( 'esc' ): load_readonly_root /rootfs/previous/bin/init else: load_readonly_root /rootfs/latest/bin/init + /rootfs/previous/bin/init It's the same basic idea as a bootable CD where the initrd constructs a virtual root of a tmpfs and the CD image on disk, save that it allows for an interaction in which a keyboard operation changes the set of overlays used to construct the root fs. Eventually there might be a UI for selecting overlays to strip them back one-by-one until you get a working system, but that's been tabled as an idea. In the two-level approach, the upgrade system would have to first compress down the two levels of the current image before it began the update (since there's no provision for a third or higher level of overlay): * def update(): o download_update into /roofts/unverified-version o if verify( /rootfs/unverified-version ): + assert no-activity-running-on-old-version + # merging moves the files from the one overlay directory to the other + # applying deletions and the like. Can be atomic for each file, hard-link + # the files rather than copying so that open files are not disturbed + merge /rootfs/latest into /rootfs/previous + # make sure running activities still see the previous version... + for each running core service and activity, remount remount fs tree to remove /rootfs/latest # (in aufs, for instance use the "del" option on /rootfs/latest branch) + assert /rootfs/latest empty + move /rootfs/unverified-version to /rootfs/latest o else: + delete /rootfs/unverified-version I may be missing an operation or two in there, but I *think* that's the intention. That is, there's an ultimate loading level that sits outside the standard root to make it all work. That loading level could be in the BIOS or on an initrd, but there does need to be a level to manage the ultimate overlay-based structure AFAIK. And at normal boot, something has to set up the overlays, regardless of what overlay set is loaded. That level of access is where the update manager has to sit to be able to do the updates AFAICS. That is to accomplish a merge or remount of the core fs we need a way to ask the overlay manager to do some work for us at the update-manager level. > Btw. I don't see why we would have to use vserver and COW for doing > this, we just have to do the hardlink copy, and then apply the update on > that, carefully always changing files so that hardlinks are broken (i.e. > no in-place updates, just replace the files). However, that still gets > us into the same situation, with a copy of the new image in a > subdirectory that is not the main image. > Yes, same basic effect, something outside the root needs to be able to switch to the root you choose, you could have a UI element that allows you to select the root, or you could have the service switch the name of the directories so that the new version is called "latest" and the last one "previous". Basically the same problem, you need a way to ask something at a higher level of access to do the promotion. The promotion could be something as simple as writing a file at the root that says "root=myversion" for the next boot, letting the boot-time process read that file in order to choose which root to boot. BTW, reading the two descriptions, the hardlink-copy version is easier than the overlay AFAICS. You don't have to remount the running services/activities (as you do if you require that there be a maximum of 1 overlay). Other than that the operations are essentially identical, you replace the merge with a recursive hard-linking copy (how does that work for soft links, incidentally, I gather you recreate them rather than hard-linking?) Something at boot time then decides what to load, with some operation setting the default t
Re: XO in-field upgrades
On Mon, 2007-06-25 at 15:38 -0400, Noah Kantrowitz wrote:
> > Caveat: I'm not an expert here. I haven't read the code, just the
> > documentation. So we can all follow along, start here:
> >http://linux-vserver.org/Paper#Unification
> >http://linux-vserver.org/Frequently_Asked_Questions#What_is_vhashify.3F
> >
> > Basically, copy-on-write works by running a tool ('vhashify') which
> > looks for identical files in the different containers and hard links
> > them together, then marks them immutable. The copy-on-write mechanism
> > works by intercepting writes to immutable files and cloning the file
> > before making it writable by the container.
> >
> It is worth noting we are not using vhashify or any of the other util
> scripts. The rainbow daemon sets up the chroot for each activity itself.
> We are a bit non-standard in that we are doing process-level
> containerization, instead of a more guest-OS system like many vserver
> users (most?).
I still don't understand at all how vserver can be used to upgrade the
main image. Yes, one can clone the main image into a chroot, and use
vserver to boot into that. One can also enable some COW stuff so that we
can change that into the new version.
But, how do we apply this new image (which is a copy of the root +
applied updates) to be the new master image the next time we boot? i.e.
when the kernel boots it will launch /bin/init,
not /vserver/updated_image/bin/init.
Nobody has explained this, yet everyone keeps talking about using
vserver and COW for the upgrades.
Btw. I don't see why we would have to use vserver and COW for doing
this, we just have to do the hardlink copy, and then apply the update on
that, carefully always changing files so that hardlinks are broken (i.e.
no in-place updates, just replace the files). However, that still gets
us into the same situation, with a copy of the new image in a
subdirectory that is not the main image.
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
Re: Fwd: XO in-field upgrades
On Mon, 2007-06-25 at 16:28 -0400, C. Scott Ananian wrote: > > > It also might be interesting to layer a diff system on top of what > Alex > > has proposed so you could do it either way - diffs if ya got 'em, > raw if > > ya don't. > > That sounds suspiciously like the complexity you're afraid of. Let's > just do diffs and call it done. Ahem. There's complexity and then there's complexity. :) Worth looking into. I'll look for Alex's responses on this one. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Fwd: XO in-field upgrades
On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote: > On Mon, 2007-06-25 at 15:38 -0400, C. Scott Ananian wrote: > > To my mind, this makes it clear that the on-wire format should be a > > binary diff. We could uncompress these diffs on receipt and maintain > > a blob store as in the current proposal, but it seems much more > > reasonable to sign, authenticate, and store the *diffs* rather than > > the *blobs*, so that we don't need to recompute diffs when we want to > > pass on our upgrade to a neighboring XO. > > It's not clear what that data means, exactly. Do those files like this: > > 158 usr-bin-clamconf > 158 usr-bin-clamdscan > 161 usr-bin-clamscan > 159 usr-bin-freshclam > 162 usr-bin-sigtool > > Actually include binary files changes or are they just permissions > changes? If so you're >75x larger charge seems a little over the top > since they wouldn't be transferred at all in Alex's case outside of the > manifest. :) No, they are actually 4-byte changes to each binary, buried in the linker-generated section, presumably caused by an offset change in the library. I stared at this and the patches for a while. You're correct that most of the size of this diff is spent coding the filename, but there is a real diff there. There's a 60x change if you just look at the libraries. So this savings is definitely real. > That being said, this is pretty compelling stuff. I would love to see > this done against various versions of libgklayout.so, which will > probably be our largest offender. If I were bored, I might indulge you. ;-) > It also might be interesting to layer a diff system on top of what Alex > has proposed so you could do it either way - diffs if ya got 'em, raw if > ya don't. That sounds suspiciously like the complexity you're afraid of. Let's just do diffs and call it done. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 16:08 -0400, C. Scott Ananian wrote: > On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote: > > On Mon, 2007-06-25 at 15:56 -0400, Noah Kantrowitz wrote: > > > minimal. Are we expecting to do online updating or will it be more of a > > > windows-style "shut it all down then patch"? > > I think that for phase one we can do it during a shutdown/restart cycle. > > Will this be fast enough if we find a big ol' root exploit in our > system? I recall people arguing before that making boot pretty wasn't > a priority "because the kids will never reboot". I assume that > updating during shutdown/restart means we have a big "you must reboot > now" message which comes up and annoys the user? Will kids ignore > this? That's largely a policy + design decision. I don't think that anything that we're talking about here adds enough latency to updates that we'll be exposing anyone to any risk any more than we are today. (Maybe less given that people tend to upgrade machines and leave them running linked to old versions of binaries all the time - a reboot certainly fixes that!) We could also say that certain updates required an immediate reboot and some don't. i.e. critical severity vs. minor bug fixes. Not all fixes are created equal. > > In any case, we need to do some sort of COW system even if we do > shutdown/restart, because the system will continue running > indefinitely with the old code until the kids get bored enough to > reboot (if ever). > --scott > Not sure how that's related to CoW? A running binary will keep using an old library until you shut it down which is where a lot of our risk comes from. An online update would still work. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Fwd: XO in-field upgrades
On Mon, 2007-06-25 at 15:38 -0400, C. Scott Ananian wrote: > > > To my mind, this makes it clear that the on-wire format should be a > binary diff. We could uncompress these diffs on receipt and maintain > a blob store as in the current proposal, but it seems much more > reasonable to sign, authenticate, and store the *diffs* rather than > the *blobs*, so that we don't need to recompute diffs when we want to > pass on our upgrade to a neighboring XO. It's not clear what that data means, exactly. Do those files like this: 158 usr-bin-clamconf 158 usr-bin-clamdscan 161 usr-bin-clamscan 159 usr-bin-freshclam 162 usr-bin-sigtool Actually include binary files changes or are they just permissions changes? If so you're >75x larger charge seems a little over the top since they wouldn't be transferred at all in Alex's case outside of the manifest. :) That being said, this is pretty compelling stuff. I would love to see this done against various versions of libgklayout.so, which will probably be our largest offender. It also might be interesting to layer a diff system on top of what Alex has proposed so you could do it either way - diffs if ya got 'em, raw if ya don't. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
There are multicast MAC addresses at Layer 2, and if I'm not mistaken we can bind to them and thus get woken up on multicast. Michail, is this right? (You seemed to say something slightly different.) On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote: > On Mon, 2007-06-25 at 15:56 -0400, Noah Kantrowitz wrote: > > minimal. Are we expecting to do online updating or will it be more of a > > windows-style "shut it all down then patch"? > I think that for phase one we can do it during a shutdown/restart cycle. Will this be fast enough if we find a big ol' root exploit in our system? I recall people arguing before that making boot pretty wasn't a priority "because the kids will never reboot". I assume that updating during shutdown/restart means we have a big "you must reboot now" message which comes up and annoys the user? Will kids ignore this? In any case, we need to do some sort of COW system even if we do shutdown/restart, because the system will continue running indefinitely with the old code until the kids get bored enough to reboot (if ever). --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 15:56 -0400, Noah Kantrowitz wrote: > > Design docs? We're still at the proof-of-concept phase really ;-). But > yes, each chroot needs to be generated on the fly when a new activity > starts (unless we do some funky magic with unionfs, which is probably > not a great idea). The load of a few directory hardlinks should be > minimal. Are we expecting to do online updating or will it be more of > a > windows-style "shut it all down then patch"? > I think that for phase one we can do it during a shutdown/restart cycle. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
> > > > You're misunderstanding me. My concern is with waking machines up by > > broadcasting this information. We don't wake up on presence, but we > > do want to wake up on (some, urgent) upgrades. > > That's going to be interesting, yeah. You would need to teach the > wireless firmware about it? How about just checking on wakeup? Some > kind of wake-on-lan signal? > In general, we won't be waking up the host on broadcast/multicast frames. We can make an exception for anycast addresses (that the firmware listens for), however I can't see a scenario where we wake up on broadcast. M. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 15:38 -0400, Noah Kantrowitz wrote: > > It is worth noting we are not using vhashify or any of the other util > scripts. The rainbow daemon sets up the chroot for each activity > itself. > We are a bit non-standard in that we are doing process-level > containerization, instead of a more guest-OS system like many vserver > users (most?). > Are there any docs up on this anywhere? --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 15:35 -0400, C. Scott Ananian wrote:
> On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote:
> > That's going to be interesting, yeah. You would need to teach the
> > wireless firmware about it? How about just checking on wakeup? Some
> > kind of wake-on-lan signal?
>
> Binding upgrade notifications to a multicast address as I previously
> proposed fixes this problem without any kind of firmware hacking.
Ahh, sorry, I thought you meant _really_ asleep - like not waking up on
network events. Although does our independent firmware know enough to
wake us up on multicast traffic? I thought that it only worked on the
lower level protocols and that a packet had to be specifically destined
for our MAC address to get a wake up event. I'll show my ignorance of
multicast here: does it include specific MAC addresses or is it a wide
broadcast at that layer? I always assumed that it was the latter.
>
> > Can you explain how they are odd? It sure would help everyone.
>
> Caveat: I'm not an expert here. I haven't read the code, just the
> documentation. So we can all follow along, start here:
>http://linux-vserver.org/Paper#Unification
>http://linux-vserver.org/Frequently_Asked_Questions#What_is_vhashify.3F
>
> Basically, copy-on-write works by running a tool ('vhashify') which
> looks for identical files in the different containers and hard links
> them together, then marks them immutable. The copy-on-write mechanism
> works by intercepting writes to immutable files and cloning the file
> before making it writable by the container.
>
> Quoting from their FAQ:
> (when running vhashify:) "The guest needs to be running because
> vhashify tries to figure out what files not to hashify by calling the
> package manager of the guest via vserver enter.
>
> In order for the OS cache to benefit from the hardlinking, you'll have
> to restart the vservers."
Holy crap, this sounds like a steaming _pile_ of complexity. Are we
seriously going to try to deploy on this?
>
> Since vserver is doing file hashification anyway, it seems like it
> would be a much better idea to use this rather than reinvent the
> wheel. Some other issues:
> a) we may need to be aware of which files are hardlinked where in
> order to do proper updates.
> b) not clear how/if we can make updates to an entire tree atomically
> c) how/can we restart the vservers? how important is this?
>
> I think we need to bring in a vserver expert (ie, not me) to get these
> details right at the start.
Am happy to get more advice on this, for sure. I suspect that all of
the vserver people we can call on are on this list.
Our current thinking basically is that we can do an update as part of an
update/shutdown procedure. So you can apply the updates on the way
down, get a new env on restart. That would handle the vserver restarts
and also "how to get a new kernel" issue that no one else has mentioned.
I'm not sure if hashing for updates and hashing for vserver are the
kinds of things we want to share or not. I would love to hear more
about how vserver does its hashing and see if we can share. I still
feel that keeping the update system as simple and uncomplex as possible
is a very good way to go - it lets us advance in parallel and come up
with something that works well.
It also sounds like it's pretty easy to do something like:
o Start root container
o Start guest container
o Apply update
o Start activities in that guest container
And the hardlink/CoW stuff will then give us an updated container.
Still doesn't help with updates on the base system nor kernel bits, but
it's a start.
>
> > Yeah, but you always need both sets of information to be able to
> > generate them. So you have to host full file + diff data if you want to
> > host an update.
>
> My proposal would be that XOs pass around binary diffs *only* among
> themselves, and that if someone needs an older version or to leapfrog
> versions, they ask the school server. This allows XOs to cheaply host
> updates in the common case.
You could do that with Alex's system as well. But in Alex's case the XO
doesn't have to carry both the system it's using + diff. Because the
system you're using is the update.
>
> > The nice thing about Alex's system is that you only
> > have to host the file data that you're using on your system instead of
> > file + diff data. You end up using less space that way.
>
> If you look at the numbers I just posted, file+diff is 1.3% larger
> than just files.
>
> > If you want to
> > downgrade, you have to get the files or use the vserver versions (maybe
> > you could just use the old files handled by the CoW stuff to drive the
> > update system - that might work pretty well!)
>
> Now we're talking! ;-)
>
> > Keep in mind that those "blobs" he's talking about are just files. The
> > only place where we would add binary diffs would be for individual
> > files, not entire trees. So what we're downloading toda
Re: XO in-field upgrades
C. Scott Ananian wrote: > On 6/25/07, Noah Kantrowitz <[EMAIL PROTECTED]> wrote: >> It is worth noting we are not using vhashify or any of the other util >> scripts. The rainbow daemon sets up the chroot for each activity itself. >> We are a bit non-standard in that we are doing process-level >> containerization, instead of a more guest-OS system like many vserver >> users (most?). > > So we're manually copying (or hard linking?) files into the chroot on > every process activation? Won't this wear down our flash? (Or are > all our chroots in RAM?) How do/can we update the libraries/etc used > by the process in the container? Are services (like the proposed > upgrade daemon) chrooted as well? If so, how do upgraded files get > out of the chroot? > > Could you give me a pointer to the design docs on rainbow/etc? > --scott Design docs? We're still at the proof-of-concept phase really ;-). But yes, each chroot needs to be generated on the fly when a new activity starts (unless we do some funky magic with unionfs, which is probably not a great idea). The load of a few directory hardlinks should be minimal. Are we expecting to do online updating or will it be more of a windows-style "shut it all down then patch"? --Noah signature.asc Description: OpenPGP digital signature ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
Following up, after re-reading the BitFrost docs: If vserver COW isn't being used, what is the planned implementation strategy for the BitFrost P_SF_RUN functionality? --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/25/07, Noah Kantrowitz <[EMAIL PROTECTED]> wrote: > It is worth noting we are not using vhashify or any of the other util > scripts. The rainbow daemon sets up the chroot for each activity itself. > We are a bit non-standard in that we are doing process-level > containerization, instead of a more guest-OS system like many vserver > users (most?). So we're manually copying (or hard linking?) files into the chroot on every process activation? Won't this wear down our flash? (Or are all our chroots in RAM?) How do/can we update the libraries/etc used by the process in the container? Are services (like the proposed upgrade daemon) chrooted as well? If so, how do upgraded files get out of the chroot? Could you give me a pointer to the design docs on rainbow/etc? --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
C. Scott Ananian wrote:
> On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote:
>
>> That's going to be interesting, yeah. You would need to teach the
>> wireless firmware about it? How about just checking on wakeup? Some
>> kind of wake-on-lan signal?
>>
>
> Binding upgrade notifications to a multicast address as I previously
> proposed fixes this problem without any kind of firmware hacking.
>
>
>> Can you explain how they are odd? It sure would help everyone.
>>
>
> Caveat: I'm not an expert here. I haven't read the code, just the
> documentation. So we can all follow along, start here:
>http://linux-vserver.org/Paper#Unification
>http://linux-vserver.org/Frequently_Asked_Questions#What_is_vhashify.3F
>
> Basically, copy-on-write works by running a tool ('vhashify') which
> looks for identical files in the different containers and hard links
> them together, then marks them immutable. The copy-on-write mechanism
> works by intercepting writes to immutable files and cloning the file
> before making it writable by the container.
>
It is worth noting we are not using vhashify or any of the other util
scripts. The rainbow daemon sets up the chroot for each activity itself.
We are a bit non-standard in that we are doing process-level
containerization, instead of a more guest-OS system like many vserver
users (most?).
--Noah
signature.asc
Description: OpenPGP digital signature
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
Fwd: XO in-field upgrades
[grumble, sent this from the wrong email address.] -- Forwarded message -- Here are some quick numbers justifying the use of binary diffs as an upgrade delivery format. I just grabbed the latest debian security advisory for Etch, which happened to be: http://www.us.debian.org/security/2007/dsa-1320 on clamav. This upgrade appeared not atypical, but feel free to point out if it is widely uncharacteristic for a lightweight package (ie, not mozilla). I downloaded and unpacked the packages corresponding to the 'before' and 'after' versions of all the packages affected. For reference, those versions are 0.90.1-3etch2 and 0.90.1-3etch3. I then deleted the files in /usr/share/doc and /usr/share/man (neither of which 'should' have changed, but apparently they did) and /usr/lib/debug. I wrote a script to generate binary diffs using both xdiff and bsdiff for the files which changed. (I also munged slashes to dashes.) These files were: usr-bin-clamconf usr-bin-freshclamusr-lib-libclamav.so.2.0.1 usr-bin-clamdscan usr-bin-sigtool usr-sbin-clamav-milter usr-bin-clamscan usr-lib-libclamav.a usr-sbin-clamd For all files other than libclamav.a and libclamav.so.2.0.1, the binary diff was less than 164/225 bytes (xdiff/bsdiff). libclamav.a had a 3529/6571 byte diff, and libclamav.so.2.0.1 had a 13398/74461 byte diff. (Note larger diff because all the relocation table entries moved. Maybe a smarter diff is possible.) In comparison, the full libclamav.a blob is 567,140 bytes and libclamav.so.2.0.1 is 432,668 bytes. Details: blob: 17168 usr/bin/clamconf 33120 usr/bin/clamdscan 49600 usr/bin/clamscan 65584 usr/bin/freshclam 77868 usr/bin/sigtool 567140 usr/lib/libclamav.a 432668 usr/lib/libclamav.so.2.0.1 76260 usr/sbin/clamav-milter 49948 usr/sbin/clamd bdiff: 158 usr-bin-clamconf 158 usr-bin-clamdscan 161 usr-bin-clamscan 159 usr-bin-freshclam 162 usr-bin-sigtool 3529 usr-lib-libclamav.a 13398 usr-lib-libclamav.so.2.0.1 164 usr-sbin-clamav-milter 162 usr-sbin-clamd Total bsdiff size: 18,051 Total raw size: 1,369,356 bytes. (>75x larger) Further, although my goal of single packet upgrades seems overoptimistic, all but two of the file diffs can fit in a single packet, helping assure atomicity. The total upgrade information fits in 13 packets. To my mind, this makes it clear that the on-wire format should be a binary diff. We could uncompress these diffs on receipt and maintain a blob store as in the current proposal, but it seems much more reasonable to sign, authenticate, and store the *diffs* rather than the *blobs*, so that we don't need to recompute diffs when we want to pass on our upgrade to a neighboring XO. --scott -- ( http://cscott.net/ ) -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote:
> That's going to be interesting, yeah. You would need to teach the
> wireless firmware about it? How about just checking on wakeup? Some
> kind of wake-on-lan signal?
Binding upgrade notifications to a multicast address as I previously
proposed fixes this problem without any kind of firmware hacking.
> Can you explain how they are odd? It sure would help everyone.
Caveat: I'm not an expert here. I haven't read the code, just the
documentation. So we can all follow along, start here:
http://linux-vserver.org/Paper#Unification
http://linux-vserver.org/Frequently_Asked_Questions#What_is_vhashify.3F
Basically, copy-on-write works by running a tool ('vhashify') which
looks for identical files in the different containers and hard links
them together, then marks them immutable. The copy-on-write mechanism
works by intercepting writes to immutable files and cloning the file
before making it writable by the container.
Quoting from their FAQ:
(when running vhashify:) "The guest needs to be running because
vhashify tries to figure out what files not to hashify by calling the
package manager of the guest via vserver enter.
In order for the OS cache to benefit from the hardlinking, you'll have
to restart the vservers."
Since vserver is doing file hashification anyway, it seems like it
would be a much better idea to use this rather than reinvent the
wheel. Some other issues:
a) we may need to be aware of which files are hardlinked where in
order to do proper updates.
b) not clear how/if we can make updates to an entire tree atomically
c) how/can we restart the vservers? how important is this?
I think we need to bring in a vserver expert (ie, not me) to get these
details right at the start.
> Yeah, but you always need both sets of information to be able to
> generate them. So you have to host full file + diff data if you want to
> host an update.
My proposal would be that XOs pass around binary diffs *only* among
themselves, and that if someone needs an older version or to leapfrog
versions, they ask the school server. This allows XOs to cheaply host
updates in the common case.
> The nice thing about Alex's system is that you only
> have to host the file data that you're using on your system instead of
> file + diff data. You end up using less space that way.
If you look at the numbers I just posted, file+diff is 1.3% larger
than just files.
> If you want to
> downgrade, you have to get the files or use the vserver versions (maybe
> you could just use the old files handled by the CoW stuff to drive the
> update system - that might work pretty well!)
Now we're talking! ;-)
> Keep in mind that those "blobs" he's talking about are just files. The
> only place where we would add binary diffs would be for individual
> files, not entire trees. So what we're downloading today is only the
> changed files, largely for the sake of expediency and what I describe
> above for the space savings.
I have some issues with the fact that the manifest includes the entire
tree as well. Upgrades should only include entries *changed* in the
manifest.
> But if I have to choose
> between having something that works today with full files and saves some
> space[...]
We're only talking about a week's worth of work here so far. I agree
that our schedule is aggressive, so let's get this right at design
time, rather than trying to fix things after implementation.
--scott
--
( http://cscott.net/ )
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
[ Fixing Alex's address. ] On Mon, 2007-06-25 at 14:58 -0400, C. Scott Ananian wrote: > On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote: > > The broadcast just contains a product/version ID - doesn't have to > > include the entire update. No more expensive than the presence stuff we > > have today. > > You're misunderstanding me. My concern is with waking machines up by > broadcasting this information. We don't wake up on presence, but we > do want to wake up on (some, urgent) upgrades. That's going to be interesting, yeah. You would need to teach the wireless firmware about it? How about just checking on wakeup? Some kind of wake-on-lan signal? > > > I think that what Alex here can do it either way. You could use the > > vserver copy on write stuff if you want to use the hammer of a > > Again, you're misunderstanding me slightly. Vserver has *very odd* > semantics for its copy-on-write and for writing inside containers. We > need to sync up on that & come up with a plan, or we run the risk of > creating a useless tool. Can you explain how they are odd? It sure would help everyone. > > Binary diffs are also bidirectional, FWIW. Yeah, but you always need both sets of information to be able to generate them. So you have to host full file + diff data if you want to host an update. The nice thing about Alex's system is that you only have to host the file data that you're using on your system instead of file + diff data. You end up using less space that way. If you want to downgrade, you have to get the files or use the vserver versions (maybe you could just use the old files handled by the CoW stuff to drive the update system - that might work pretty well!) > > > When you hear "complete blobs" can you describe what you mean? I > > suspect that you're thinking of something different than what Alex has > > actually implemented. > > Quoting from https://hosted.fedoraproject.org/projects/updatinator/ : > --- > Each computer running Updatinator tracks a specific image id, and runs > a specific version of that image. Whenever it finds a manifest for a > later version of that image id (and that manifest is signed by the > right gpg key) that manifest and the required blobs for updating to it > is automatically downloaded. When the manifest and all required blobs > are downloaded Updatinator sends a dbus signal so that the system may > let the user apply the update (e.g. automatically, or by showing some > ui asking the user if he wants to update). > --- > > My next post will give some concrete #s justifing the use of binary diffs. > --scott > Keep in mind that those "blobs" he's talking about are just files. The only place where we would add binary diffs would be for individual files, not entire trees. So what we're downloading today is only the changed files, largely for the sake of expediency and what I describe above for the space savings. Speaking for myself I've never been opposed to use binary diffs. To be sure, all of my original ideas included them. But if I have to choose between having something that works today with full files and saves some space and adding the complexity of binary diffs later, I will use the former. I would love to hear what you have to say about numbers for binary diffs. (I believe that in earlier discussions with people who tried these systems before that any diff system that didn't understand elf was doomed to failure, but I am ready to be shown the money.) I think that both Alex and I are happy to listen to ideas here (esp about the vserver stuff that you mention!) but it's June 25th and we have to be pretty practical about what we can do between now and when we have to ship. The update system needs to be very simple, easy to test and easy to validate + understand. The key to that is using a simple design and simple implementation. Added complexity has a high bar for inclusion here. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote: > The broadcast just contains a product/version ID - doesn't have to > include the entire update. No more expensive than the presence stuff we > have today. You're misunderstanding me. My concern is with waking machines up by broadcasting this information. We don't wake up on presence, but we do want to wake up on (some, urgent) upgrades. > I think that what Alex here can do it either way. You could use the > vserver copy on write stuff if you want to use the hammer of a Again, you're misunderstanding me slightly. Vserver has *very odd* semantics for its copy-on-write and for writing inside containers. We need to sync up on that & come up with a plan, or we run the risk of creating a useless tool. Binary diffs are also bidirectional, FWIW. > When you hear "complete blobs" can you describe what you mean? I > suspect that you're thinking of something different than what Alex has > actually implemented. Quoting from https://hosted.fedoraproject.org/projects/updatinator/ : --- Each computer running Updatinator tracks a specific image id, and runs a specific version of that image. Whenever it finds a manifest for a later version of that image id (and that manifest is signed by the right gpg key) that manifest and the required blobs for updating to it is automatically downloaded. When the manifest and all required blobs are downloaded Updatinator sends a dbus signal so that the system may let the user apply the update (e.g. automatically, or by showing some ui asking the user if he wants to update). --- My next post will give some concrete #s justifing the use of binary diffs. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 12:25 -0400, C. Scott Ananian wrote: > > > mDNS *is* multicast. But the blobs won't be exposed over mDNS, that > is > > far to much data for a protocol like that. > > Really? Do we know that? What's a typical 0-day patch look like? > Have we tried to see how few bits it could be squashed into? The broadcast just contains a product/version ID - doesn't have to include the entire update. No more expensive than the presence stuff we have today. > > > Binary diffs seem much less useful. They enforce a specific base > version > > that you have to have around, and they enforce the direction of > upgrade. > > This is *exactly* why we need to have the big picture view. My > understanding is that we *are* expecting all laptops to have identical > bases, that the upgrade propagation rate and upgrade (in)frequency > will be sufficient that all laptops will be running the same version > of the software (save for a few stragglers who go on a long trip for a > few weeks), and that the vserver copy-on-write mechanism will be used > to perform rollback (so that we only need to worry about the forward > direction). I think that what Alex here can do it either way. You could use the vserver copy on write stuff if you want to use the hammer of a filesystem or you could use the image code he has to move it back whether or not the copy on write stuff is there. Or even after you have "committed" via vserver. I strongly suggest that we make sure that we keep the ability to go both directions. It gives us a lot more flexibility down the road, and for the countries and our users as well. > > I'm not saying that my assumptions are correct. But I feel that > deciding file formats before we've come to a big picture consensus may > be premature. > > > If you can cheaply generate at runtime (on the client) a minimal > diff > > between any two versions you can avoid storing unnecessary > information, > > Not sure exactly what you're getting at here: that we transfer > complete blobs over the network but store them on the XO as binary > diffs? My working assumption is that network and storage costs on the > XO should be minimized as much as possible. Transferring complete > blobs fails on these grounds. When you hear "complete blobs" can you describe what you mean? I suspect that you're thinking of something different than what Alex has actually implemented. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Mon, 2007-06-25 at 12:25 -0400, C. Scott Ananian wrote: > On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote: > > Most of the stuff that Alex has done is (carefully) independent of any > > vserver or container discussion. Specifically, the update system in > > question could be applied inside of a container as easy as it would be > > outside of the container. > > I not sure I agree that the filesystem portion of the implementation > ought to be completely independent of the network portion. I think > that networking desiderata are going to impact our choices among the > various upgrade schemes, and I'd prefer to have a high level design > for the whole thing before we get too attached to particular bits of > code. As one example, since network reliability goes down quickly as > message size increases, it seems (to me, at least) that our upgrade > data messages should be made as small as possible. It's not clear to > me that our current design minimizes upgrade transfer size. > > > Re: broadcast, that's basically the same as any laptop exposing presence > > information. For _transmission_ of an update, it's an interesting > > question as to whether or not to use a multicast update kind of thing. > > Do laptops usually wake each other up to process presence information? > [Hopefully not.] Should they do so for urgent security upgrades? > [Hopefully.] > > Here's a draft proposal: > We listen to multicast address foo:bar:xxx:xxx: if we > currently have version # n installed on the machine. Then we won't be > woken up by our friends announcing that they have version n like we > do, but we will be woken up as soon as someone gets version n+1. > > On 6/25 Alex L wrote: > > mDNS *is* multicast. But the blobs won't be exposed over mDNS, that is > > far to much data for a protocol like that. > > Really? Do we know that? What's a typical 0-day patch look like? > Have we tried to see how few bits it could be squashed into? No, stuffing actual update data into the mDNS TXT records is completely impractical. First off, you're limited by UDP packet size, and I'm sure our updates are going to be larger than 512 bytes (even with UDP packet sizes up to the MTU). mDNS never carries actual _data_, it carries a service description about the data. The data itself is always out-of-band WRT mDNS. Dan > > Binary diffs seem much less useful. They enforce a specific base version > > that you have to have around, and they enforce the direction of upgrade. > > This is *exactly* why we need to have the big picture view. My > understanding is that we *are* expecting all laptops to have identical > bases, that the upgrade propagation rate and upgrade (in)frequency > will be sufficient that all laptops will be running the same version > of the software (save for a few stragglers who go on a long trip for a > few weeks), and that the vserver copy-on-write mechanism will be used > to perform rollback (so that we only need to worry about the forward > direction). > > I'm not saying that my assumptions are correct. But I feel that > deciding file formats before we've come to a big picture consensus may > be premature. > > > If you can cheaply generate at runtime (on the client) a minimal diff > > between any two versions you can avoid storing unnecessary information, > > Not sure exactly what you're getting at here: that we transfer > complete blobs over the network but store them on the XO as binary > diffs? My working assumption is that network and storage costs on the > XO should be minimized as much as possible. Transferring complete > blobs fails on these grounds. > --scott > ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/25/07, Christopher Blizzard <[EMAIL PROTECTED]> wrote: > Most of the stuff that Alex has done is (carefully) independent of any > vserver or container discussion. Specifically, the update system in > question could be applied inside of a container as easy as it would be > outside of the container. I not sure I agree that the filesystem portion of the implementation ought to be completely independent of the network portion. I think that networking desiderata are going to impact our choices among the various upgrade schemes, and I'd prefer to have a high level design for the whole thing before we get too attached to particular bits of code. As one example, since network reliability goes down quickly as message size increases, it seems (to me, at least) that our upgrade data messages should be made as small as possible. It's not clear to me that our current design minimizes upgrade transfer size. > Re: broadcast, that's basically the same as any laptop exposing presence > information. For _transmission_ of an update, it's an interesting > question as to whether or not to use a multicast update kind of thing. Do laptops usually wake each other up to process presence information? [Hopefully not.] Should they do so for urgent security upgrades? [Hopefully.] Here's a draft proposal: We listen to multicast address foo:bar:xxx:xxx: if we currently have version # n installed on the machine. Then we won't be woken up by our friends announcing that they have version n like we do, but we will be woken up as soon as someone gets version n+1. On 6/25 Alex L wrote: > mDNS *is* multicast. But the blobs won't be exposed over mDNS, that is > far to much data for a protocol like that. Really? Do we know that? What's a typical 0-day patch look like? Have we tried to see how few bits it could be squashed into? > Binary diffs seem much less useful. They enforce a specific base version > that you have to have around, and they enforce the direction of upgrade. This is *exactly* why we need to have the big picture view. My understanding is that we *are* expecting all laptops to have identical bases, that the upgrade propagation rate and upgrade (in)frequency will be sufficient that all laptops will be running the same version of the software (save for a few stragglers who go on a long trip for a few weeks), and that the vserver copy-on-write mechanism will be used to perform rollback (so that we only need to worry about the forward direction). I'm not saying that my assumptions are correct. But I feel that deciding file formats before we've come to a big picture consensus may be premature. > If you can cheaply generate at runtime (on the client) a minimal diff > between any two versions you can avoid storing unnecessary information, Not sure exactly what you're getting at here: that we transfer complete blobs over the network but store them on the XO as binary diffs? My working assumption is that network and storage costs on the XO should be minimized as much as possible. Transferring complete blobs fails on these grounds. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Sun, 2007-06-24 at 12:44 -0400, C. Scott Ananian wrote: > > Has this already been discussed in more depth off-list? I'd prefer to > get the relevant folks together for a quick f2f, since it seems like a > number of different groups are involved (networking, security, XO > build), but I can post a more detailed critique if we want to continue > the discussion on list instead. It's been discussed a little bit off-list but not to extreme depths. Most of the stuff that Alex has done is (carefully) independent of any vserver or container discussion. Specifically, the update system in question could be applied inside of a container as easy as it would be outside of the container. Re: broadcast, that's basically the same as any laptop exposing presence information. For _transmission_ of an update, it's an interesting question as to whether or not to use a multicast update kind of thing. But one step at a time - that's the kind of thing that we can add later once we show that we can get actual updates working. --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On Sun, 2007-06-24 at 13:24 -0400, C. Scott Ananian wrote: > On 6/24/07, Ivan Krstić <[EMAIL PROTECTED]> wrote: > > I should have a concrete spec ready for discussion later today. > > I will wait with bated breath. =) > > Some concrete concerns -- I've got some answers to these, but I'll try > to just present the questions at this point: > a) Robustness -- what can break our upgrade mechanism? Do we have a > fallback? What dependencies does the upgrade mechanism have? > b) Current vserver code requires restarting the containers when > files inside the container are modified by the root context. There is > also a relinking process necessary. Have we thought through these > interactions? I haven't really seen a viable way to use vserver for updating the root filesystem. Is there a proposal for how this would work? I had a different idea to do a very robust root filesystem update. Given the layout and format of jffs2 (the filesystem we're using) it is very easy to add filessystem transaction support. That is, we'd have a way to trigger a transaction at the start of an update, then we just do all the changes we want (including just updating the kernel image file). Then when the update is done, we commit the transaction. If at any time we lose power or anything like that, none of the changes after the transaction start will be visible on the next boot. And also, if we have any problems during update (e.g. out of flash) we an rollback and abort. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Fwd: XO in-field upgrades
On Sun, 2007-06-24 at 12:46 -0400, C. Scott Ananian wrote: > Sorry, wrong email... > --scott > -- Forwarded message -- > From: C. Scott Ananian <[EMAIL PROTECTED]> > Date: Jun 24, 2007 12:44 PM > Subject: XO in-field upgrades > To: [EMAIL PROTECTED] > Cc: Christopher Blizzard <[EMAIL PROTECTED]>, Michail Bletsas > <[EMAIL PROTECTED]>, John Watlington <[EMAIL PROTECTED]>, > devel@lists.laptop.org > > > In the weekend I just read about Alex' plan for XO in-field upgrades. > The mailing list message is here: >http://lists.laptop.org/pipermail/devel/2007-June/005381.html > I think the proposal deserves more discussion; in particular there are > some interactions with vserver and copy-on-write, as well as > networking (I really don't want the laptops broadcasting information > about what upgrade blobs they have on mDNS! We should use a multicast > address at least...) Also, I think we've discussed shipping binary > diffs rather than git's integral blobs, which may (or may not) be > incompatible with the proposal so far. Fundamentally, I don't think > that re-using git's design philosophy wholesale is actually the Right > Thing for this application. For the stuff in that mail I don't see how vserver and copy-on-write has any bearing. That mail is strictly about how to get the bits of a specific update to the laptop. How we then later apply the bits to the machine is a different thing. mDNS *is* multicast. But the blobs won't be exposed over mDNS, that is far to much data for a protocol like that. mDNS is just used to announce "here is a laptop that has this version of the image installed", and if the version is later we can upgrade another laptop from that one. Binary diffs seem much less useful. They enforce a specific base version that you have to have around, and they enforce the direction of upgrade. If you can cheaply generate at runtime (on the client) a minimal diff between any two versions you can avoid storing unnecessary information, and you can upgrade between any two versions (including going backwards). I find that quite superiour to having to e.g. upgrade multiple steps by using multiple diffs. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/25/07, C. Scott Ananian <[EMAIL PROTECTED]> wrote: > Also, I think we've discussed shipping binary > diffs rather than git's integral blobs, Git can also use "thin" packs which are basically binary diffs. So you _can_ use the git machinery for distributing upgrades with truly minimal bw usage -- if you can guarantee taht the receiving end will have the thing that it's meant to diff against ;-) cheers m ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
On 6/24/07, Ivan Krstić <[EMAIL PROTECTED]> wrote: > I should have a concrete spec ready for discussion later today. I will wait with bated breath. =) Some concrete concerns -- I've got some answers to these, but I'll try to just present the questions at this point: a) Robustness -- what can break our upgrade mechanism? Do we have a fallback? What dependencies does the upgrade mechanism have? b) Current vserver code requires restarting the containers when files inside the container are modified by the root context. There is also a relinking process necessary. Have we thought through these interactions? c) How do XOs know there is an upgrade available? Does this query/notification work both with and without a school server? d) We can't afford to have all the XOs contact Cambridge directly to get updates. Likewise, if the XOs get all their updates from the school server, the machines closest to the mesh portal will be passing the same bits around redundantly. Can we use mesh topography to improve distribution? e) How small/fast can we make an upgrade, to patch (say) a specific 0-day in one of our applications? If we can fit a patch into a single network packet, our distribution success probability should increase dramatically. f) Does our mechanism scale gracefully for both small (0-day security) and large (FC7-to-FC8) upgrades? g) Related to the above three: I've been told that the XOs can't possibly afford the space to store upgrades in order to be able to provide them to neighbors. I'm not sure I agree with this, but this concern should be addressed. h) What happens if we need to patch the kernel to fix a security problem? What if we need to patch firmware? i) What's the development process like? Can we easily create and test upgrades? j) Is the "base system" upgrade mechanism related in any way to the activity upgrade mechanism? If so, activity upgrades have a whole 'nuther set of concerns. If not, how can we avoid reinventing the same wheels (esp. with regard to distributing an activity upgrade efficiently to a classroom full of machines). Since I can't resist a few stabs at answers: a) I'd like to see two independent upgrade mechanisms available, one of them as simple as practical and hard-coded into firmware. d) If upgrades are broadcast to the mesh from the portal, the flood fill will ensure the bits efficiently reach most of the machines. I hope that listing these issues is helpful. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO in-field upgrades
C. Scott Ananian wrote: > Has this already been discussed in more depth off-list? To some degree. I should have a concrete spec ready for discussion later today. -- Ivan Krstić <[EMAIL PROTECTED]> | GPG: 0x147C722D ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
XO in-field upgrades
In the weekend I just read about Alex' plan for XO in-field upgrades. The mailing list message is here: http://lists.laptop.org/pipermail/devel/2007-June/005381.html I think the proposal deserves more discussion; in particular there are some interactions with vserver and copy-on-write, as well as networking (I really don't want the laptops broadcasting information about what upgrade blobs they have on mDNS! We should use a multicast address at least...) Also, I think we've discussed shipping binary diffs rather than git's integral blobs, which may (or may not) be incompatible with the proposal so far. Fundamentally, I don't think that re-using git's design philosophy wholesale is actually the Right Thing for this application. Has this already been discussed in more depth off-list? I'd prefer to get the relevant folks together for a quick f2f, since it seems like a number of different groups are involved (networking, security, XO build), but I can post a more detailed critique if we want to continue the discussion on list instead. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
