Re: RW Compressed FUSE FSs? (Re: XS - XO archiving and backup)
On Mon, Nov 10, 2008 at 6:00 PM, Martin Langhoff [EMAIL PROTECTED] wrote: On Mon, Nov 10, 2008 at 11:17 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: I would get a real journal and check for real which is the compressibility of the data that is actually there. My uninformed guess is that most of the content is already compressed: png, zip, pdf (some/most), odt, ogg, etc Interesting. I was under the impression that at least some of the file formats were uncompressed and taking advantage of jffs2's compressibility. Well, I'm sure some activities save in uncompressed formats, but my impression is that they aren't that many. That said, I think this should be checked. Regards, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: RW Compressed FUSE FSs? (Re: XS - XO archiving and backup)
On Nov 10 2008, at 11:08, Martin Langhoff was caught saying: On Mon, Nov 10, 2008 at 10:49 AM, Greg Smith [EMAIL PROTECTED] wrote: On backup and restore, aside from the comments already mentioned, I suggest you pay careful attention to the available space on your XS. You should have about 2GB free space on your XS for each XO. If you don't have enough space it may move some of the files over your wireless LAN then not save them on the XS. Excellent point. Which leads to a request - good references on a well behaved compressed FUSE FS that - supports RW - behaves well with rsync (which I presume mmaps files liberally) - supports hardlinks and ACLs As a happy user of large hard drives, I haven't needed a compressed FS since the unhappy days of DRDOS so I'm rusty on this front. There's a listing at http://fuse.sourceforge.net/wiki/index.php/CompressedFileSystems but I know nothing about the quality, reliability and performance. I took a look at these and in summary, only the following options seem useable for the XS scenario as the remainder are for cramfs, loopback, read-only or other non-general purpose use. compFUSEd Not 100% of status. Last update was May 1, 2007. LZOlayerFS Has not been updated since Feb 2006 so I say that is an immediate NO IMHO unless we want to take over the project. fuseCompress Looks to be in active development. Last Git update was 11/02 and there is a steady stream of updates for the last two months. I care mainly about the reliability -- but it better be fast too. Compression ratio is perhaps more negotiable, the other two arent :-) I think the next step in determing which (if any) of the above would meet the XS needs is to start pounding on them with LTP, iozone and other performance/stress/compliance tests to see how well they all work. Beyond performance, I'm worried about stability/maintainability. Regular filesytems get a lot of testing by a whole lot of people and when a major bug hits, we know it will be fixed ASAP. I don't know if the same can be said of externally maintained filesystems. I suggest we also engage with the developers of the three alternatives above and let them know that we're considering deploying them on the XS. Note that there is also an ext3 compression extension that is probably worth investigating, though this requires custom kernel packaging. That being said, I think we also need to get a really good idea of just how much data is already compressed. (We need same data to determine whether to enable or disable compression on /home/olpc on the XO itself). Running a FUSE layer will defintely result in performance degradation (which also means increased power consumption) and to go through the whole process of compressing blocks to discard the results for the majority of blocks would be a resource waste. ~Deepak -- Deepak Saxena - Kernel Developer, One Laptop Per Child _ __o (o ---\, Give One Laptop, Get One Laptop //\ - ( )/ ( ) http://www.amazon.com/xoV_/_ ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: RW Compressed FUSE FSs? (Re: XS - XO archiving and backup)
On Mon, Nov 10, 2008 at 11:17 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: I would get a real journal and check for real which is the compressibility of the data that is actually there. On a test machine with a good cross-section of activity documents, I gzipped all the files in the datastore directory. Only 6% storage savings according to the gzip stats - higher if I only look at the metadata files (avg 12%). Several files had negative compression ratios. So your suspicions seem correct - compression is a marginal win at best for ds-backup. cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: RW Compressed FUSE FSs? (Re: XS - XO archiving and backup)
martin wrote: On Mon, Nov 10, 2008 at 11:17 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: I would get a real journal and check for real which is the compressibility of the data that is actually there. On a test machine with a good cross-section of activity documents, I gzipped all the files in the datastore directory. Only 6% storage savings according to the gzip stats - higher if I only look at the metadata files (avg 12%). Several files had negative compression ratios. So your suspicions seem correct - compression is a marginal win at best for ds-backup. i assume we're already doing duplicate checks in order to avoid storing the same present-on-every-client executable N times, right? paul =- paul fox, [EMAIL PROTECTED] give one laptop, get one laptop --- http://www.amazon.com/xo ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: RW Compressed FUSE FSs? (Re: XS - XO archiving and backup)
On Mon, Nov 10, 2008 at 2:34 PM, [EMAIL PROTECTED] wrote: martin wrote: On Mon, Nov 10, 2008 at 11:17 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: I would get a real journal and check for real which is the compressibility of the data that is actually there. On a test machine with a good cross-section of activity documents, I gzipped all the files in the datastore directory. Only 6% storage savings according to the gzip stats - higher if I only look at the metadata files (avg 12%). Several files had negative compression ratios. So your suspicions seem correct - compression is a marginal win at best for ds-backup. i assume we're already doing duplicate checks in order to avoid storing the same present-on-every-client executable N times, right? Ds-backup only backs up files in the XO users journal so except for downloaded *.xo bundles there won't be any?? 'executables'. i.e. You won't get the large reduction that you see in backups of Windows/Linux clients OS images where 4 or 5 gigs of storage is replicated across all of the machines you are backing up. Bill Bogstad ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: RW Compressed FUSE FSs? (Re: XS - XO archiving and backup)
bill wrote: On Mon, Nov 10, 2008 at 2:34 PM, [EMAIL PROTECTED] wrote: martin wrote: On Mon, Nov 10, 2008 at 11:17 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: I would get a real journal and check for real which is the compressibility of the data that is actually there. On a test machine with a good cross-section of activity documents, I gzipped all the files in the datastore directory. Only 6% storage savings according to the gzip stats - higher if I only look at the metadata files (avg 12%). Several files had negative compression ratios. So your suspicions seem correct - compression is a marginal win at best for ds-backup. i assume we're already doing duplicate checks in order to avoid storing the same present-on-every-client executable N times, right? Ds-backup only backs up files in the XO users journal so except for downloaded *.xo bundles there won't be any?? 'executables'. i.e. You ah, thanks. paul =- paul fox, [EMAIL PROTECTED] give one laptop, get one laptop --- http://www.amazon.com/xo ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
