On Wed, 2007-06-27 at 16:02 -0400, C. Scott Ananian wrote: > Anyway, here are some quick stats, for upgrades from build 464 to 465 to 466:
I've been doing some more work on updatinator. Now it stores the "release" info as a simple key-value file that contains the manifest sha1 value, and the actual manifest is just a blob like the other. This way we automatically get gzip and bsdiff support for the manifest files (which tend to be around 2 meg for a full olpc devel image). Here are some stats using bsdiffs and gzip. I don't have any byte counter in the app, so i just looked at interface RX/TX stats. It counts the full packet size instead of the tcp stream byte count, and it might have data from other apps running on the system, but it should give a somewhat correct maximum limit at least. > 464-to-465: > Full contents of tree: 429,302,610 bytes > rsync --whole-file: sent 32,034 bytes; received 13,153,813 bytes. > Standard rsync: sent 96,720 bytes; received 10,047,524 bytes. 9.0s > user, 64.9s real > Rsync-with-manifest: sent 79,192 bytes; received 9,139,515 bytes. 4.3 > user, 11.5s real. Updatinator: sent 596,681 bytes recieved 7,545,281 bytes > 465-to-466: > Full contents of tree: 458,828,206 bytes > rsync --whole-file: sent 21,990 bytes; received 25,826,516 bytes. > Standard rsync: sent 141,948 bytes; received 23,981,423 bytes. 14.0s > user, 68.5s real > Rsync-with-manifest: sent 125,158 bytes; received 23,171,430 bytes. > 10.4 user, 28.5s real. Updatinator: sent 1,263,885 bytes recieved 20,718,744 bytes I haven't looked into why we're sending so much data, perhaps its due to other traffic on my box, or perhaps the http accesses are just too chatty. But even taking that into consideration we're doing 1.2 meg less total network traffic in the 465-466 case than the best rsync case (and we're not doing a lot of local disk i/o). I'm also creating an updatinator repository at http://olpc1.redhat.com/updatinator/ so that people can start test it, and eventually use it to easily update their XOs. Another interesting piece of stats here is the size of this repository. A repository with all the ext3 and jffs2 developer images from version 400 to 450 has a blobs directory of 1.3 gigs. The estimated size for just the tarballs of these revisions is around 12 gigs. This is a pretty good compression ratio. This is good for the school server disk use and means it can easily store all old releases. _______________________________________________ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel