Re: [gentoo-user] What can I use for a compressed file system?
Am Freitag, 12. Mai 2006 23:21 schrieb ext Richard Fish: What I would really like to see is something like reiser4's plugin scheme brought up to the VFS layer in the kernel, so that any filesystem could gain transparent compression. Hmm, wouldn't that be a device mapper task, just like dm-crypt? Bye... Dirk -- Dirk Heinrichs | Tel: +49 (0)162 234 3408 Configuration Manager | Fax: +49 (0)211 47068 111 Capgemini Deutschland | Mail: [EMAIL PROTECTED] Hambornerstraße 55 | Web: http://www.capgemini.com D-40472 Düsseldorf | ICQ#: 110037733 GPG Public Key C2E467BB | Keyserver: www.keyserver.net pgph0OHCJplYa.pgp Description: PGP signature
Re: [gentoo-user] What can I use for a compressed file system?
On 5/14/06, Dirk Heinrichs [EMAIL PROTECTED] wrote: What I would really like to see is something like reiser4's plugin scheme brought up to the VFS layer in the kernel, so that any filesystem could gain transparent compression. Hmm, wouldn't that be a device mapper task, just like dm-crypt? I don't think so, because how would the filesystem know how many blocks are available to it? If I create a 10G filesystem on a 10G compressing dm that averages 2:1 compression, I could store 20G of files, but the filesystem would provide only 10G of space. Ok, so you create a 20G filesystem instead...now what happens if you store a bunch of already compressed files there, and run out of room on the dm volume before the filesystem thinks you should? My guess is Bad Things. But if the compression is done by the filesystem, it can know exactly how many real blocks a compressed file takes, so it can maintain it's accounting for free blocks appropriately. Thus I think the filesystem or VFS layer is the only sane place to implement any compression. -Richard -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
On Thursday 11 May 2006 19:51, W.Kenworthy wrote: What can I use for a compressed file system? I am looking at setting up a loopback mounted filesystem that I want to use to store backups into. Compression is needed as space will become a limitation in the future (I want to do a whole system backup that so far is 2:1 compressed via tar.bzip2. I am thinking of using dirvish into a compressed loopback mount - but how do I set up a compressed fs? Have you tried reiserfs? As long as it is NOT mounted with the notail option it can sometimes save 50% on space compared to ext3/jfs/xfs depending on your usage. There is also a possiblility of using LVM2 snapshots also if you have LVM2 devices already set up. I'm not sure how dirvish is for backup and I'm not sure how good a loopback backup to a file really is anyway. That depends on the consistency of at least a partition anyway. Maybe you are trying to solve the wrong problem? -- Zac Slade [EMAIL PROTECTED] ICQ:1415282 YM:krakrjak AIM:ttyp99 -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
since you are not looking at writing to this fs, then you can use cloop or squashfs for example, gentoo uses squashfs for its live cd/dvd squashfs is considered better, but both are in use on live cd/dvd, cloop was (At least partially) written by the knoppix dude. typically you get 2.5:1 compression with these over a general linux distro file average. either one will put all files starting at a root path into the compressed structure. The only real difference between doing it cloop/squashfs and tar.*z is that cloop/squashfs can be directly accessed (once mounted), which might be of some use. big negative (unless fixed in recent releases) is you need enough ram/VM to hold the entire fs (to be compressed) in memory. So if you have 512MB ram and a 1GB VM allocation, the biggest fs you can archive using cloop/squashfs would be 2.5GB (approx), that compresses down to the 1GB to fit into your VM. pretty recent cloop souce is at knoppix web site, squashfs, IIRC is at kernel.org squashfs would also be available in gentoo, as gentoo uses it in their live cd. -tl On Fri, 12 May 2006 02:47:56 -0500 Zac Slade [EMAIL PROTECTED] wrote: On Thursday 11 May 2006 19:51, W.Kenworthy wrote: What can I use for a compressed file system? I am looking at setting up a loopback mounted filesystem that I want to use to store backups into. Compression is needed as space will become a limitation in the future (I want to do a whole system backup that so far is 2:1 compressed via tar.bzip2. I am thinking of using dirvish into a compressed loopback mount - but how do I set up a compressed fs? Have you tried reiserfs? As long as it is NOT mounted with the notail option it can sometimes save 50% on space compared to ext3/jfs/xfs depending on your usage. There is also a possiblility of using LVM2 snapshots also if you have LVM2 devices already set up. I'm not sure how dirvish is for backup and I'm not sure how good a loopback backup to a file really is anyway. That depends on the consistency of at least a partition anyway. Maybe you are trying to solve the wrong problem? -- Zac Slade [EMAIL PROTECTED] ICQ:1415282 YM:krakrjak AIM:ttyp99 -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
This is what I currently use: But I dont have room for two archives, and this method doesnt keep versions. Trying to keep incrementals using this has proven to be a disaster. BillK On Thu, 2006-05-11 at 23:25 -0700, Richard Fish wrote: On 5/11/06, W.Kenworthy [EMAIL PROTECTED] wrote: What can I use for a compressed file system? I am looking at setting up a loopback mounted filesystem that I want to use to store backups into. From what I can tell, there are no really good compressing filesystems available currently. But why do you need to do this in the filesystem? Why not use a compressible format for your backups like tar, cpio, or (my favorite) dar? -Richard -- William Kenworthy [EMAIL PROTECTED] Home! -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
I already use reiserfs with notail, but potentially 60G wont go into 40G of space without compression, and then there is trying to keep versions ... Its notail is also irrelevant if you backup into a single file. Same for LVM snapshots (though in this case its a non-LVM laptop that I want disaster protection for - i.e., in case I drop it!). My working data is actually handled well by unison onto a solaris system at work, but again, space/time and data transfer costs are a problem when trying to do whole systems remotely, so it would be nice to handle it myself, I just dont want to go buy another disk to do it unless I have to. BillK On Fri, 2006-05-12 at 02:47 -0500, Zac Slade wrote: On Thursday 11 May 2006 19:51, W.Kenworthy wrote: What can I use for a compressed file system? I am looking at setting up a loopback mounted filesystem that I want to use to store backups into. Compression is needed as space will become a limitation in the future (I want to do a whole system backup that so far is 2:1 compressed via tar.bzip2. I am thinking of using dirvish into a compressed loopback mount - but how do I set up a compressed fs? Have you tried reiserfs? As long as it is NOT mounted with the notail option it can sometimes save 50% on space compared to ext3/jfs/xfs depending on your usage. There is also a possiblility of using LVM2 snapshots also if you have LVM2 devices already set up. I'm not sure how dirvish is for backup and I'm not sure how good a loopback backup to a file really is anyway. That depends on the consistency of at least a partition anyway. Maybe you are trying to solve the wrong problem? -- Zac Slade [EMAIL PROTECTED] ICQ:1415282 YM:krakrjak AIM:ttyp99 -- William Kenworthy [EMAIL PROTECTED] Home! -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
On 5/12/06, William Kenworthy [EMAIL PROTECTED] wrote: This is what I currently use: But I dont have room for two archives, and this method doesnt keep versions. Trying to keep incrementals using this has proven to be a disaster. Again, checkout dar. It is specifically designed for doing differential backups. As an example: # full backup dar --create /media/backups/20060513 --empty-dir -H0 --nodump -O -N -g / # differntial/incremental backup dare --create /media/backups/20060514 --empty-dir -H0 --nodump -O -N \ --ref /media/backups/20060513 -g / -Richard -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
William Kenworthy wrote, regarding Squashfs: and you need at least the uncompressed space to create the image ... not useful here. Wrong, you need sufficient disk space to create the compressed filesystem, that is all. Phillip Lougher -- View this message in context: http://www.nabble.com/What-can-I-use-for-a-compressed-file-system--t1604870.html#a4362648 Sent from the gentoo-user forum at Nabble.com. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
ted leslie wrote: big negative (unless fixed in recent releases) is you need enough ram/VM to hold the entire fs (to be compressed) in memory. So if you have 512MB ram and a 1GB VM allocation, the biggest fs you can archive using cloop/squashfs would be 2.5GB (approx), that compresses down to the 1GB to fit into your VM. This isn't the case for Squashfs (and never has been). Prior to version 3.0, mksquashfs could create a 4GB compressed filesystem irrespective of the amount of free memory/VM in the host computer. In version 3.0, the 4 GB filesystem limit has been removed, and filesystem sizes are potentially unlimited. Phillip Lougher -- View this message in context: http://www.nabble.com/What-can-I-use-for-a-compressed-file-system--t1604870.html#a4362494 Sent from the gentoo-user forum at Nabble.com. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
William Kenworthy wrote This is what I currently use: But I dont have room for two archives, and this method doesnt keep versions. Trying to keep incrementals using this has proven to be a disaster. Even though Squashfs is read-only (and so is tar, cpio etc.), you can append to pre-existing Squashfs filesystems without needing to decompress and then recompress the filesystem. Because Squashfs detects duplicates, and renames duplicated files (in the top level directory) at appending, this supports simple incremental versioning. For example, you could archive directory data, delete it, and later add directories a and b to the archive without needing the disk space to decompress directory data. If you kept the data directory, and later added new files to it, or modified files, adding the data directory to the Squashfs archive a second time would create two directories data, the first version, and data_1 the second version. Only files that have been added or have changed in the data directory will be added to the Squashfs archive, the other files (presumably the bulk) will be be detected as duplicates and not added. Phillip Lougher -- View this message in context: http://www.nabble.com/What-can-I-use-for-a-compressed-file-system--t1604870.html#a4362944 Sent from the gentoo-user forum at Nabble.com. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
Richard Fish write: From what I can tell, there are no really good compressing filesystems available currently. I would disagree, Squashfs is an advanced read-only compressing filesystem, which uses numerous techniques to obtaIn high compression ratios while also being fast. Some of the techniques (compressed metadata, use of fragment blocks, indexed compressed directories) I doubt you'll find many places elsewhere irrespective of the operating system. What I would agree with is there is no commercial support for compressing filesystems, which at a time where the major improvements to the Linux kernel are (arguably) being driven by the Linux distribution vendors, is a major limitation. Unfortunately, embedded systems vendors tend to simply use what is there, and the others are mainly focussed on the enterprise which is why there's a lot of enterprise scale and clustering filesystems about. But why do you need to do this in the filesystem? Why not use a compressible format for your backups like tar, cpio, or (my favorite) dar? So you can mount the filesystem and transparently access the files as if they were uncompressed. Phillip Lougher -- View this message in context: http://www.nabble.com/What-can-I-use-for-a-compressed-file-system--t1604870.html#a4363501 Sent from the gentoo-user forum at Nabble.com. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] What can I use for a compressed file system?
On 5/12/06, plougher [EMAIL PROTECTED] wrote: Richard Fish write: From what I can tell, there are no really good compressing filesystems available currently. I would disagree, Squashfs is an advanced read-only compressing filesystem, I should have said read-write filesystem. What I would really like to see is something like reiser4's plugin scheme brought up to the VFS layer in the kernel, so that any filesystem could gain transparent compression. I have no use for compression on backup disks, since I use programs that support compression internally. But I would love to be able to compress my normal filesystems /, /var, and /usr/portage. Being a laptop user, I could actually get more speed from those volumes if they were compressed, depending upon the algorithm used. -Richard -- gentoo-user@gentoo.org mailing list
