Upgrades and image manifests
Hi, My name is Alexander Larsson, and I just started working on the field upgrade system of the olpc laptops. I have some ideas I'd like to explain and get feedback on. The olpc uses a full-image system, as opposed to the per-package versioning scheme of deb or rpms. So, an upgrade consists of replacing your system files with a full copy of a later (or earlier) version of the system image. We want to support both upgrading from a central server or from another laptop, and we want to minimize the data we have to download for an upgrade. To support this we create a manifest that describes each image, and we save a copy of the current image manifest installed on the laptop. The manifest format is very similar to the git "tree" object, and describes each file by listing name, metadata and sha1 hash value of the contents. Here is a small example: - link 777 500:500 foo bar blob 664 500:500 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 changed.txt blob 664 500:500 65ef9261ad6cfaa8403f2fa66d8edfada1445e5a common blob 664 500:500 f660633c832712eba3c6edd39151987e46bc3f87 image1.txt blob 711 500:500 empty logfs.pdf dir 775 500:500 subdir1 blob 664 500:500 empty subdir1/file1.txt blob 664 500:500 7448d8798a4380162d4b56f9b452e2f6f9e24e7a subdir1/file2.txt --- Given a manifest for the current image and a manifest for the image to upgrade to it is very simple and cheap to calculate what changes you need to make to transform the image (on the client), and what sha1 blobs you need to download. You can then download the file blobs from whatever source you have (they are self-verifying, since they are named by the sha1 hash of the content) and do the upgrade. It is very simple to host the blobs. We just put them all in a directory named by sha1 and export that with a http server. One can also host multiple versions of an image in the same directory, while minimizing the space used for common data. (Possibly one might want to gzip the blobs too.) I've written some code (attached) to generate and manipulate manifests like these. There are three tools: * generate-manifest: This generates a manifest files given a path to a directory containing the image tree. You can also make it populate a directory of sha1-named blobs by giving it a blob directory with -b. * diff-manifest: Gives a simple visual diff between two manifests. If you pass it -b it will instead give a list of all blobs required to update between the two manifests. * upgrade-manifest: Updates an image from one manifest to another, given a path of a directory with the required blobs. Using these tools I upgraded from olpc-redhat-stream-development-build-406-20070507_2141-devel_ext3-tree to olpc-redhat-stream-development-build-406-20070507_2157-ext3-tree in a directory on my development machine, so it seems to work so far. For finding and downloading updates I was thinking of using avahi to publish the image id + version, and then downloading the data and manifests using http. That way that clients can detect machines around them with later versions and auto-download (and apply automatically or later). The individual laptops can publish the version of the image they are using, and a school server can publish several versions. From the point of view of the individual laptop searching for upgrades they will look the same. Of course, we'd have to sign the manifests with some key to make sure you can't just auto-upgrade any machine to a rouge image. Now over to the questions I have: I'd like to put this code in a repository somewhere. Where should I put it? Does OLPC use selinux or xattrs? Because if so we have to extend the manifest format. We need a library to do http downloads. I see we're shipping libcurl on the laptop. Is it ok to use this library? (We also need a very simple http sever that maps sha1 -> file in system image for laptop-to-laptop upgrades, but I think we can do that without using a library.) Is using avahi/mDNS on the mesh network ok? manifest-0.1.tar.gz Description: application/compressed-tar ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote: > Does OLPC use selinux or xattrs? Because if so we have to extend the > manifest format. Not yet, but it's likely to in the near future when we ditch the short-term hacks and manage to implement the proper long-term security plan. So it's worth designing for it from the start. -- dwmw2 ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Mon, 2007-06-25 at 15:45 +0100, David Woodhouse wrote: > On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote: > > Does OLPC use selinux or xattrs? Because if so we have to extend the > > manifest format. > > Not yet, but it's likely to in the near future when we ditch the > short-term hacks and manage to implement the proper long-term security > plan. So it's worth designing for it from the start. > Yeah, I'm pretty sure that want to have support for at least describing xattrs (especially if we want to use this for regular filesystems as well - lots of people are using xattrs for all kinds of things.) --Chris ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote: > Hi, > > My name is Alexander Larsson, and I just started working on the field > upgrade system of the olpc laptops. I have some ideas I'd like to > explain and get feedback on. > > The olpc uses a full-image system, as opposed to the per-package > versioning scheme of deb or rpms. So, an upgrade consists of replacing > your system files with a full copy of a later (or earlier) version of > the system image. We want to support both upgrading from a central > server or from another laptop, and we want to minimize the data we have > to download for an upgrade. > > To support this we create a manifest that describes each image, and we > save a copy of the current image manifest installed on the laptop. The > manifest format is very similar to the git "tree" object, and describes > each file by listing name, metadata and sha1 hash value of the contents. > > Here is a small example: > - > link 777 500:500 foo bar > blob 664 500:500 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 changed.txt > blob 664 500:500 65ef9261ad6cfaa8403f2fa66d8edfada1445e5a common > blob 664 500:500 f660633c832712eba3c6edd39151987e46bc3f87 image1.txt > blob 711 500:500 empty logfs.pdf > dir 775 500:500 subdir1 > blob 664 500:500 empty subdir1/file1.txt > blob 664 500:500 7448d8798a4380162d4b56f9b452e2f6f9e24e7a subdir1/file2.txt > --- > > Given a manifest for the current image and a manifest for the image to > upgrade to it is very simple and cheap to calculate what changes you > need to make to transform the image (on the client), and what sha1 blobs > you need to download. You can then download the file blobs from whatever > source you have (they are self-verifying, since they are named by the > sha1 hash of the content) and do the upgrade. Two questions here: 1) what does the scheme do in the case where the file it's about to replace on the local machine isn't the same as what the manifest on the local machine says? ie, local changes have changed the sha1 hash of the local file. That's essentially a policy decision, do we replace when there are local changes or don't we. A %config-type thing could work here to mark files that you may/may not always want to replace. 2) After downloading a blob, I assume the tool sha1s the downloaded file to ensure that it's content matches the name, right? > It is very simple to host the blobs. We just put them all in a directory > named by sha1 and export that with a http server. One can also host > multiple versions of an image in the same directory, while minimizing > the space used for common data. (Possibly one might want to gzip the > blobs too.) > > I've written some code (attached) to generate and manipulate manifests > like these. There are three tools: > > * generate-manifest: This generates a manifest files given a path to a > directory containing the image tree. You can also make it populate a > directory of sha1-named blobs by giving it a blob directory with -b. > > * diff-manifest: Gives a simple visual diff between two manifests. If > you pass it -b it will instead give a list of all blobs required to > update between the two manifests. > > * upgrade-manifest: Updates an image from one manifest to another, given > a path of a directory with the required blobs. > > Using these tools I upgraded from > olpc-redhat-stream-development-build-406-20070507_2141-devel_ext3-tree > to olpc-redhat-stream-development-build-406-20070507_2157-ext3-tree in a > directory on my development machine, so it seems to work so far. > > For finding and downloading updates I was thinking of using avahi to > publish the image id + version, and then downloading the data and > manifests using http. That way that clients can detect machines around > them with later versions and auto-download (and apply automatically or > later). The individual laptops can publish the version of the image they > are using, and a school server can publish several versions. From the > point of view of the individual laptop searching for upgrades they will > look the same. Of course, we'd have to sign the manifests with some key > to make sure you can't just auto-upgrade any machine to a rouge image. > > Now over to the questions I have: > > I'd like to put this code in a repository somewhere. Where should I put > it? > > Does OLPC use selinux or xattrs? Because if so we have to extend the > manifest format. > > We need a library to do http downloads. I see we're shipping libcurl on > the laptop. Is it ok to use this library? Yeah, though the ironic thing just occurred to me that libcurl is (along with Mozilla) one of the more frequently updated pieces of a linux distro :) > (We also need a very simple http sever that maps sha1 -> file in system > image for laptop-to-laptop upgrades, but I think we can do that without > using a library.) "simple" means lighttpd in my mind, but apache has a longer track record. > Is usi
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 08:22 -0400, Dan Williams wrote: > Two questions here: > > 1) what does the scheme do in the case where the file it's about to > replace on the local machine isn't the same as what the manifest on the > local machine says? ie, local changes have changed the sha1 hash of the > local file. That's essentially a policy decision, do we replace when > there are local changes or don't we. A %config-type thing could work > here to mark files that you may/may not always want to replace. At the moment it only touches the files that are affected by the diff in the manifest. If a file wasn't changed in the manifests we don't even look at it. However, if the file we're replacing has changed locally, we currently throw that away. > 2) After downloading a blob, I assume the tool sha1s the downloaded file > to ensure that it's content matches the name, right? Yeah, or rather, while downloading. > > We need a library to do http downloads. I see we're shipping libcurl on > > the laptop. Is it ok to use this library? > > Yeah, though the ironic thing just occurred to me that libcurl is (along > with Mozilla) one of the more frequently updated pieces of a linux > distro :) I ended up using urllib2 in python. > > (We also need a very simple http sever that maps sha1 -> file in system > > image for laptop-to-laptop upgrades, but I think we can do that without > > using a library.) > > "simple" means lighttpd in my mind, but apache has a longer track > record. simple means 300 lines of C code in serve-manifest.c... ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 14:21 +0200, Alexander Larsson wrote: > On Fri, 2007-06-29 at 08:22 -0400, Dan Williams wrote: > > Two questions here: > > > > 1) what does the scheme do in the case where the file it's about to > > replace on the local machine isn't the same as what the manifest on the > > local machine says? ie, local changes have changed the sha1 hash of the > > local file. That's essentially a policy decision, do we replace when > > there are local changes or don't we. A %config-type thing could work > > here to mark files that you may/may not always want to replace. > > At the moment it only touches the files that are affected by the diff in > the manifest. If a file wasn't changed in the manifests we don't even > look at it. However, if the file we're replacing has changed locally, we > currently throw that away. > > > 2) After downloading a blob, I assume the tool sha1s the downloaded file > > to ensure that it's content matches the name, right? > > Yeah, or rather, while downloading. > > > > We need a library to do http downloads. I see we're shipping libcurl on > > > the laptop. Is it ok to use this library? > > > > Yeah, though the ironic thing just occurred to me that libcurl is (along > > with Mozilla) one of the more frequently updated pieces of a linux > > distro :) > > I ended up using urllib2 in python. Right, python too :) If we get a factory python process (didn't we have one at one point?) we save some memory. > > > (We also need a very simple http sever that maps sha1 -> file in system > > > image for laptop-to-laptop upgrades, but I think we can do that without > > > using a library.) > > > > "simple" means lighttpd in my mind, but apache has a longer track > > record. > > simple means 300 lines of C code in serve-manifest.c... That may work even better, 300 lines is certainly 1,000,000 times easier to audit for security problems than all of apache or lighttpd. Dan ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote: > Hi, > > My name is Alexander Larsson, and I just started working on the field > upgrade system of the olpc laptops. I have some ideas I'd like to > explain and get feedback on. > > The olpc uses a full-image system, as opposed to the per-package > versioning scheme of deb or rpms. So, an upgrade consists of replacing > your system files with a full copy of a later (or earlier) version of > the system image. We want to support both upgrading from a central > server or from another laptop, and we want to minimize the data we have > to download for an upgrade. > > To support this we create a manifest that describes each image, and we > save a copy of the current image manifest installed on the laptop. The > manifest format is very similar to the git "tree" object, and describes > each file by listing name, metadata and sha1 hash value of the contents. > > Here is a small example: > - > link 777 500:500 foo bar > blob 664 500:500 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 changed.txt > blob 664 500:500 65ef9261ad6cfaa8403f2fa66d8edfada1445e5a common > blob 664 500:500 f660633c832712eba3c6edd39151987e46bc3f87 image1.txt > blob 711 500:500 empty logfs.pdf > dir 775 500:500 subdir1 > blob 664 500:500 empty subdir1/file1.txt > blob 664 500:500 7448d8798a4380162d4b56f9b452e2f6f9e24e7a subdir1/file2.txt > --- > > Given a manifest for the current image and a manifest for the image to > upgrade to it is very simple and cheap to calculate what changes you So the manifest you update to is an interesting piece, and it's easier because where you get it from is completely orthogonal to the update process. So, you could: 1) as others have suggested, XO periodically contacts the school server 2) XO sends it's current image, asks what to update to, if anything 3) school server makes policy decision about update 4) school server sends back new manifest 5) XO verifies manifests signature 6) XO downloads changesets This is cool, because it doesn't matter _where_ you get the changesets from, be it the school server (which should be seeding the changesets anyway), or from another XO. It optimizes for bandwidth, but not at the expense of any security. You already know what changesets are valid based on the signature-verified server-provided manifest, and on the self-verifying sha1 of the changesets that anyone can provide to you. All in all, simple, straightforward, and not very much code. Dan > need to make to transform the image (on the client), and what sha1 blobs > you need to download. You can then download the file blobs from whatever > source you have (they are self-verifying, since they are named by the > sha1 hash of the content) and do the upgrade. > > It is very simple to host the blobs. We just put them all in a directory > named by sha1 and export that with a http server. One can also host > multiple versions of an image in the same directory, while minimizing > the space used for common data. (Possibly one might want to gzip the > blobs too.) > > I've written some code (attached) to generate and manipulate manifests > like these. There are three tools: > > * generate-manifest: This generates a manifest files given a path to a > directory containing the image tree. You can also make it populate a > directory of sha1-named blobs by giving it a blob directory with -b. > > * diff-manifest: Gives a simple visual diff between two manifests. If > you pass it -b it will instead give a list of all blobs required to > update between the two manifests. > > * upgrade-manifest: Updates an image from one manifest to another, given > a path of a directory with the required blobs. > > Using these tools I upgraded from > olpc-redhat-stream-development-build-406-20070507_2141-devel_ext3-tree > to olpc-redhat-stream-development-build-406-20070507_2157-ext3-tree in a > directory on my development machine, so it seems to work so far. > > For finding and downloading updates I was thinking of using avahi to > publish the image id + version, and then downloading the data and > manifests using http. That way that clients can detect machines around > them with later versions and auto-download (and apply automatically or > later). The individual laptops can publish the version of the image they > are using, and a school server can publish several versions. From the > point of view of the individual laptop searching for upgrades they will > look the same. Of course, we'd have to sign the manifests with some key > to make sure you can't just auto-upgrade any machine to a rouge image. > > Now over to the questions I have: > > I'd like to put this code in a repository somewhere. Where should I put > it? > > Does OLPC use selinux or xattrs? Because if so we have to extend the > manifest format. > > We need a library to do http downloads. I see we're shipping libcurl on > the laptop. Is it ok to use this library? > (We also nee
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 09:33 -0400, Dan Williams wrote: > So the manifest you update to is an interesting piece, and it's easier > because where you get it from is completely orthogonal to the update > process. So, you could: > > 1) as others have suggested, XO periodically contacts the school server > 2) XO sends it's current image, asks what to update to, if anything > 3) school server makes policy decision about update > 4) school server sends back new manifest > 5) XO verifies manifests signature > 6) XO downloads changesets > > This is cool, because it doesn't matter _where_ you get the changesets > from, be it the school server (which should be seeding the changesets > anyway), or from another XO. It optimizes for bandwidth, but not at the > expense of any security. You already know what changesets are valid > based on the signature-verified server-provided manifest, and on the > self-verifying sha1 of the changesets that anyone can provide to you. > > All in all, simple, straightforward, and not very much code. Yeah, its quite flexible. One can upgrade between any two images (forwards, backwards, cross images, etc). I've got the code mostly working now, and I managed to update a qemu instance of build 406 to the devel build 406 using something like: ./updatinator.py -u http://10.x.y.z/updates --get-manifest olpc-ext3 406 ./updatinator.py -u http://10.x.y.z/updates -i -m manifests/olpc-ext3.406 --download olpc-devel-ext3 406 ./upgrade-manifest -b blobs/ -d / manifests/olpc-ext3.406 manifests/olpc-devel-ext3.406 Where http://10.x.y.z/updates is an apache server with manifests and blobs generated with generate-manifests from the -tree OLPC images. However, then things failed when I upgraded to olpc-devel-ext3.445, due to out of diskspace. I need to look into using gzip of blobs and removing blobs once they have been applied (this is harder than it sounds because blobs can be used multiple times in one manifest). ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote: > Hi, > > My name is Alexander Larsson, and I just started working on the field > upgrade system of the olpc laptops. I have some ideas I'd like to > explain and get feedback on. One more thing :) I'm sure you know, but something people haven't brought up in the whole updates discussion yet is the whole picture. It's not just about the mechanical process of getting updates from point A to point B, since that completely forgets the users who's XO we're trying to update. It's critical to give some user visibility into the update process. While the tool needs to be resilient against users moving around and downloads timing out, the user also needs the _ability_ to see what the tool is doing, even if the UI or the user don't always want to or care. Sugar may want to notify the user when an update starts so the user knows why their machine is now slower, and why their WLAN light is always on. We should have at least the ability to do the Sidekick thing and allow the user to postpone certain updates, but after a week it gets applied automatically. We may want to show the progress of the update, to give users visibility into the process and when to expect it to get done. We need to notify the user than an update is done, and we would suggest that they reboot (possibly based on a metadata flag in the manifest for "suggest-reboot=yes"). The update daemon must provide a fair amount read-only status information before and during the update process to allow the GUI bits the flexibility to present that information to the user in the way it sees fit. In the ideal world, that means a D-Bus service that sends signals at various stages and has methods Sugar can call to get more information about the update process. However, that links in more libraries and adds vulnerability window to a critical piece of the system. So in that sense, even though a D-Bus server model is a very good one from an API point of view, perhaps the most responsible method would be simple text files that Sugar can inotify and get the required information. Oh, and we should probably rate-limit or connection-limit the local HTTP server that's serving updates so we don't hammer one particular XO. This could mean putting an _advisory_ "try-me=true/false" flag in the mDNS advertisement that we switch depending on the current connection load. The server would obviously still boot users if there were too many connections, but at least using the 'try-me' gives other laptops more information about whether they should try to connect right away, or try a different XO that may advertise the same blob. Have you thought about what information the mDNS record would advertise? In the best case, it's as large of a UDP packet as we can push out. And since we can probably deal easily with packets up to the wireless MTU (which is something > 2000 bytes), there's more room than the traditional 512 byte UDP packet. However, the space in the mDNS announcement is not unlimited, so I don't think we could put the entire blob list in there :) There's certainly enough room to put a build number, HTTP server port number, manifest hash, etc. Dan > The olpc uses a full-image system, as opposed to the per-package > versioning scheme of deb or rpms. So, an upgrade consists of replacing > your system files with a full copy of a later (or earlier) version of > the system image. We want to support both upgrading from a central > server or from another laptop, and we want to minimize the data we have > to download for an upgrade. > > To support this we create a manifest that describes each image, and we > save a copy of the current image manifest installed on the laptop. The > manifest format is very similar to the git "tree" object, and describes > each file by listing name, metadata and sha1 hash value of the contents. > > Here is a small example: > - > link 777 500:500 foo bar > blob 664 500:500 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 changed.txt > blob 664 500:500 65ef9261ad6cfaa8403f2fa66d8edfada1445e5a common > blob 664 500:500 f660633c832712eba3c6edd39151987e46bc3f87 image1.txt > blob 711 500:500 empty logfs.pdf > dir 775 500:500 subdir1 > blob 664 500:500 empty subdir1/file1.txt > blob 664 500:500 7448d8798a4380162d4b56f9b452e2f6f9e24e7a subdir1/file2.txt > --- > > Given a manifest for the current image and a manifest for the image to > upgrade to it is very simple and cheap to calculate what changes you > need to make to transform the image (on the client), and what sha1 blobs > you need to download. You can then download the file blobs from whatever > source you have (they are self-verifying, since they are named by the > sha1 hash of the content) and do the upgrade. > > It is very simple to host the blobs. We just put them all in a directory > named by sha1 and export that with a http server. One can also host > multiple versions of an image in the same
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 10:21 -0400, Dan Williams wrote: > On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote: > The update daemon must provide a fair amount read-only status > information before and during the update process to allow the GUI bits > the flexibility to present that information to the user in the way it > sees fit. auto-updatinator (the mDNS background downloader) emits dbus signals when there is a new version availible, and when it downloaded a new blob. The actual applying of the update once its done downloading should be triggered by something listening to that signal. > Oh, and we should probably rate-limit or connection-limit the local HTTP > server that's serving updates so we don't hammer one particular XO. > This could mean putting an _advisory_ "try-me=true/false" flag in the > mDNS advertisement that we switch depending on the current connection > load. The server would obviously still boot users if there were too > many connections, but at least using the 'try-me' gives other laptops > more information about whether they should try to connect right away, or > try a different XO that may advertise the same blob. Yeah. We currently only handle one file at a time in the local http server. However, some rate limiting on that sounds fair. At the moment we pick a new server to download each blob for randomly from the set of availible servers that has the version we want. > Have you thought about what information the mDNS record would advertise? > In the best case, it's as large of a UDP packet as we can push out. And > since we can probably deal easily with packets up to the wireless MTU > (which is something > 2000 bytes), there's more room than the > traditional 512 byte UDP packet. However, the space in the mDNS > announcement is not unlimited, so I don't think we could put the entire > blob list in there :) There's certainly enough room to put a build > number, HTTP server port number, manifest hash, etc. At the moment we have image id, a list of image version ranges availible, a base directory for the http uri, a hint that the server has diffs and a "prio" hint which is unused atm. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 15:49 +0200, Alexander Larsson wrote: > I've got the code mostly working now, and I managed to update a qemu > instance of build 406 to the devel build 406 using something like: > > ./updatinator.py -u http://10.x.y.z/updates --get-manifest olpc-ext3 406 > ./updatinator.py -u http://10.x.y.z/updates -i -m manifests/olpc-ext3.406 > --download olpc-devel-ext3 406 > ./upgrade-manifest -b blobs/ -d / manifests/olpc-ext3.406 > manifests/olpc-devel-ext3.406 > > Where http://10.x.y.z/updates is an apache server with manifests and > blobs generated with generate-manifests from the -tree OLPC images. > > However, then things failed when I upgraded to olpc-devel-ext3.445, due > to out of diskspace. I need to look into using gzip of blobs and > removing blobs once they have been applied (this is harder than it > sounds because blobs can be used multiple times in one manifest). I added --delete to upgrade-manifest that does delete-behind (handling multiple uses of the same hash in one upgrade). So now it should be possible to do the upgrade using less space. It worked when running on my local machine, but I haven't tried it in a qemu instance yet. However, I gotta leave now, and I won't be able to do any OLPC work at all next week. It would be cool if someone else could play a bit with this, test it a bit more, get the blobs on a public service, test on a real XO, etc. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
Please read Ivan's update proposal. He addresses many of the peripheral issues that have arisen, re user notification, etc. At this point, as far as I'm concerned, the only open question is how we get a filesystem image onto the laptop once we know which one to get. Ivan suggests rsync; I refined that somewhat with a directory-at-a-time rsync using manifests; someone else (Mike?) proposed bittorrent, and previously I proposed a bespoke solution using binary diffs and multicast. Alex's work is interesting, but we need to see benchmarks if we're not going to make a decision based on just fiat and personality. I've posted up/download bandwidth for upgrades between 464-465 and 465-466. Following up on Mike's suggestion, I've also patched bittorrent to allow using it for upgrades: the net bandwidth usage is something like 30MB, because it does not use block compression. I can post more detailed comparisons if people are interested. Please perform the equivalent benchmarks on the updatinator, before we continue this discussion. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Fri, 2007-06-29 at 12:26 -0400, C. Scott Ananian wrote: > Please read Ivan's update proposal. He addresses many of the > peripheral issues that have arisen, re user notification, etc. > > At this point, as far as I'm concerned, the only open question is how > we get a filesystem image onto the laptop once we know which one to > get. Ivan suggests rsync; I refined that somewhat with a > directory-at-a-time rsync using manifests; someone else (Mike?) > proposed bittorrent, and previously I proposed a bespoke solution > using binary diffs and multicast. > > Alex's work is interesting, but we need to see benchmarks if we're not > going to make a decision based on just fiat and personality. I've > posted up/download bandwidth for upgrades between 464-465 and 465-466. > Following up on Mike's suggestion, I've also patched bittorrent to > allow using it for upgrades: the net bandwidth usage is something like > 30MB, because it does not use block compression. I can post more > detailed comparisons if people are interested. Please perform the > equivalent benchmarks on the updatinator, before we continue this > discussion. With updatinator, bsdiff-ing the changes between files and gzipping new files is the sweet spot. I tested plain blobs, gzipped blobs, bzip2-ed blobs, bsdiff + gzip, and bsdiff + bzip2. Unfortunately, bzip2's memory consumption on decompress isn't all that great according to other people's research, and gzip gives us the best balance between compression ratio and decompression mem/cpu usage. Scott, you may also want to re-test the rsync benchmarks using rsync compression to make the bandwidth numbers in the rsync benchmarks go down. You didn't mention anything specifically about using wire compression, but the numbers look like you hadn't used it? This patch adds bsdiff + gzip compression to updatinator: http://people.redhat.com/dcbw/updatinator-bsdiff-gzip.patch The difference sizes, using "du -csb" on the difference blob directory are as follows. This is the amount of data that would be transferred over the network, not including HTTP headers. 464->465: 6,869,799 bytes 465->466: 18,870,574 bytes The resulting image directories verify with both verify-manifest and diff -rua. As an improvement, we should provide a manifest-diff file for each update path (along with the manifests for the actual image) that lists the blobs with their own sha1 sum so that they are self-verifying. This would also simplify the patch quite a bit because there would be less path-munging. The diff-manifest tool that generates the blob diffs will output this data in a somewhat suitable format already. Cheers, Dan Procedure: 1) get the 465 and 466 "tree" files 2) unpack those trees 3) diff-manifest -g 465-466 465 466 465-manifest.txt 466-manifest.txt 4) cp -r 465 465-test 5) upgrade-manifest -b 465-466 -d 465-test 465-manifest.txt 466-manifest.txt 6) verify-manifest 466-manifest.txt 465-test ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Upgrades and image manifests
On Mon, 2007-07-02 at 07:47 -0400, Dan Williams wrote: > > With updatinator, bsdiff-ing the changes between files and gzipping new > files is the sweet spot. I tested plain blobs, gzipped blobs, bzip2-ed > blobs, bsdiff + gzip, and bsdiff + bzip2. Unfortunately, bzip2's memory > consumption on decompress isn't all that great according to other > people's research, and gzip gives us the best balance between > compression ratio and decompression mem/cpu usage. > > Scott, you may also want to re-test the rsync benchmarks using rsync > compression to make the bandwidth numbers in the rsync benchmarks go > down. You didn't mention anything specifically about using wire > compression, but the numbers look like you hadn't used it? > > This patch adds bsdiff + gzip compression to updatinator: > > http://people.redhat.com/dcbw/updatinator-bsdiff-gzip.patch Cool. I'll merge it. > The difference sizes, using "du -csb" on the difference blob directory > are as follows. This is the amount of data that would be transferred > over the network, not including HTTP headers. > > 464->465: 6,869,799 bytes > 465->466: 18,870,574 bytes > > The resulting image directories verify with both verify-manifest and > diff -rua. > > As an improvement, we should provide a manifest-diff file for each > update path (along with the manifests for the actual image) that lists > the blobs with their own sha1 sum so that they are self-verifying. This > would also simplify the patch quite a bit because there would be less > path-munging. The diff-manifest tool that generates the blob diffs will > output this data in a somewhat suitable format already. As a different approach I was thinking of making the "manifest file" just contain a sha1 has and a gpg sign, and then make the actual manifest data a blob. That way we'd automatically reuse gzip and bsdiff features from the blobs. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
