On Fri, 2007-06-29 at 12:26 -0400, C. Scott Ananian wrote: > Please read Ivan's update proposal. He addresses many of the > peripheral issues that have arisen, re user notification, etc. > > At this point, as far as I'm concerned, the only open question is how > we get a filesystem image onto the laptop once we know which one to > get. Ivan suggests rsync; I refined that somewhat with a > directory-at-a-time rsync using manifests; someone else (Mike?) > proposed bittorrent, and previously I proposed a bespoke solution > using binary diffs and multicast. > > Alex's work is interesting, but we need to see benchmarks if we're not > going to make a decision based on just fiat and personality. I've > posted up/download bandwidth for upgrades between 464-465 and 465-466. > Following up on Mike's suggestion, I've also patched bittorrent to > allow using it for upgrades: the net bandwidth usage is something like > 30MB, because it does not use block compression. I can post more > detailed comparisons if people are interested. Please perform the > equivalent benchmarks on the updatinator, before we continue this > discussion.
With updatinator, bsdiff-ing the changes between files and gzipping new files is the sweet spot. I tested plain blobs, gzipped blobs, bzip2-ed blobs, bsdiff + gzip, and bsdiff + bzip2. Unfortunately, bzip2's memory consumption on decompress isn't all that great according to other people's research, and gzip gives us the best balance between compression ratio and decompression mem/cpu usage. Scott, you may also want to re-test the rsync benchmarks using rsync compression to make the bandwidth numbers in the rsync benchmarks go down. You didn't mention anything specifically about using wire compression, but the numbers look like you hadn't used it? This patch adds bsdiff + gzip compression to updatinator: http://people.redhat.com/dcbw/updatinator-bsdiff-gzip.patch The difference sizes, using "du -csb" on the difference blob directory are as follows. This is the amount of data that would be transferred over the network, not including HTTP headers. 464->465: 6,869,799 bytes 465->466: 18,870,574 bytes The resulting image directories verify with both verify-manifest and diff -rua. As an improvement, we should provide a manifest-diff file for each update path (along with the manifests for the actual image) that lists the blobs with their own sha1 sum so that they are self-verifying. This would also simplify the patch quite a bit because there would be less path-munging. The diff-manifest tool that generates the blob diffs will output this data in a somewhat suitable format already. Cheers, Dan Procedure: 1) get the 465 and 466 "tree" files 2) unpack those trees 3) diff-manifest -g 465-466 465 466 465-manifest.txt 466-manifest.txt 4) cp -r 465 465-test 5) upgrade-manifest -b 465-466 -d 465-test 465-manifest.txt 466-manifest.txt 6) verify-manifest 466-manifest.txt 465-test _______________________________________________ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel