Re: possible to expand a file for vn-device FS usage ?
Patrick Thomas wrote: > Thank you for the very clear explanation. Does there exist a utility to > immediately take a partition that has been growfs'd and "fix" it so that > it does not experience this performance penalty ? > > That is, I am willing to sit and wait 10 minutes while some utility > rearranges and reorganizes the unmounted filesystem if it means I don't > have to dump/restore/blah/blah and if it allows me to avoid the > performance penalty you mentioned... You are talking about a defragmenter. It's possible to write one of these, but no one has written one yet, because no one really uses growfs(8) often enough that it ever becomes an issue. So the short answer is "no". The longer answer is that all a defragmenter would do is "make this disk look like it would have looked had i been this size all along". The easiest way to do this is a backup and restore. Note that this does not require a tape drive, if you have enough disk space for the dump image, so it's not like this will be an incredibly time consuming process or anything. How long it would take really depends on the size of your disks, and whether you can use a different drive for the backup image or not (concurrent access to the same disk with a lot of seeks will slow the process by a factor of ~3 on most IDE based systems). The even longer answer is that this sort of thing could forcibly migrate files away from an area of the disk, so it could be used to resize a disk *smaller* (something that isn't supported at all right now), or support relocation and resizing (ala the Power Quest "Partition Magic" program). I actually volunteered to write this code for "$1 and other valuable considerations" for Power Quest, Inc., to get them to support FFS with their tools, several times, but they were not interested. The offer still stands for them; without a payoff like FreeBSD support in the best commercial partitioning product, though, it's probably not worth it just as a standalone project, without cash being involved. Conceptually, if you want to write this code, of it's very easy; all you have to do is create an image of the metadata allocations as they would look on a disk of the appropriate size, if all the files were newly created for the first time, and then rewrite the disk contents to match the image your program has in mind. You can even make it incrementally restartable, by using a file to store the image, and another small file for an intention log, in case there is a power failure during the operation. This assumes that there is enough slack disk space around for you to store the log, and that you can make the log non-colliding with the rest of the FS contents (fairly easy, actually: treat it as an already relocated file in the new FS, and temporarily move file contents around, "fragmenting them worse" to clear the area where you plan to put the log). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: possible to expand a file for vn-device FS usage ?
Thank you for the very clear explanation. Does there exist a utility to immediately take a partition that has been growfs'd and "fix" it so that it does not experience this performance penalty ? That is, I am willing to sit and wait 10 minutes while some utility rearranges and reorganizes the unmounted filesystem if it means I don't have to dump/restore/blah/blah and if it allows me to avoid the performance penalty you mentioned... thanks, PT To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: possible to expand a file for vn-device FS usage ?
Patrick Thomas wrote: > What is the negative effect of this fragmentation, and does it mean I > won't be able to use all of the space that I added ? Old disk: [X X ][XX ][ XX ][X X][ XX] New disk (initial state): [X X ][XX ][ XX ][X X][ XX][][][][][] New disk (after 10 allocations): [XXX ][XX X][XXX ][XX X][ XXX][X ][ X ][ X][ X ][ X ] New disk (after 20 allocations): [][][][][][X X ][XX ][ X X][ XX ][ XX] Result: slowed access times, very long allocation cycle, even though there is a lot of free space (100% fill on bottom half, 50% fill on top half, when it should have an overall 50% fill). You get to spend all of your time generating random numbers until you find a cylinder group that isn't full. 50/50 chance of getting one of the bottom instead of top sectors each time. Performance falls off exponentially, just like the whole disk was almost full, even if the top half is really half empty. In the graphic example above, the fill on the disk is an *average* 75%; selection of disk space is a hash function, which, if you read Knuth: "Seminumerical algorithms: Sorting and Searching", would not have a performance hit at all until 85% full... IF the distribution was perfectly random, which it isn't because you added disk space without reallocating already allocated "random" allocations -- thus introducing an artificial selection bias "history". So: you'll be able to use the space, but your performance will suck, probably sooner, rather than later. FWIW, LFS and EXT2FS and EXT3FS and Reiser and NTFS all have this issue, but some of those have "cleaners"/defragmenters. Note: this example started with the bottom disk 50% full; you are much more likely to add disk space only after you really need it, and it's not likely to be double. Therefore (1) your performance will suck almost immediately, rather than the suck being delayed, and (2) the base of the exponential curve will be much, much higher, and (3) you will spend all your CPU on looking for an empty cluster in a cylinder group somewhere. In normal operation, FFS never has this problem because the free reserve guarantees that the hash never gets full enough to have a problem (well, except FreeBSD has reduced the default free reserve below the 10% speed/space tradeoff level to 8%, when 15% would be optimal [100% - 85% = 15%]), but then growing your disk space available is an abnormal operation not considered in the original design of FFS (all disk packs are 4M, and that's *luxury*). See the "newfs" man page, and the FFS design paper for details. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: possible to expand a file for vn-device FS usage ?
What is the negative effect of this fragmentation, and does it mean I won't be able to use all of the space that I added ? On Thu, 15 Aug 2002, Terry Lambert wrote: > Daniel O'Connor wrote: > > On Thu, 2002-08-15 at 17:04, Patrick Thomas wrote: > > > Any suggestions on how to expand that file without doing the dump/restore > > > steps ? > > > > man 8 growfs perchance? :) > > You can unmount it, grow the underlying file with: > > dd if-/dev/zero bs=XXX,count=XXX >> filename > > and *THEN* use growfs(8) on it. > > Doing this will leave the allocation layout in the same state > that it is at present, so the bottom half of the FS will end > up fragmented, even though there is free space at the top (FS > growing does not equally redistribute the FS content into the > newly enlarged space). > > The best approach is the same as it would be for a device: > dump and restore the FS from the old image to the new. In > the vn device case, you could just create a new empty FS of > the necessary size, and dump from the old piped to a restore > of the new. > > If you can live with the internal fragmentation, use growfs(8); > if you can't, use dump/restore. IMO, you will have less > potential for future problems if you use dump/restore. > > -- Terry > To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: possible to expand a file for vn-device FS usage ?
Daniel O'Connor wrote: > On Thu, 2002-08-15 at 17:04, Patrick Thomas wrote: > > Any suggestions on how to expand that file without doing the dump/restore > > steps ? > > man 8 growfs perchance? :) You can unmount it, grow the underlying file with: dd if-/dev/zero bs=XXX,count=XXX >> filename and *THEN* use growfs(8) on it. Doing this will leave the allocation layout in the same state that it is at present, so the bottom half of the FS will end up fragmented, even though there is free space at the top (FS growing does not equally redistribute the FS content into the newly enlarged space). The best approach is the same as it would be for a device: dump and restore the FS from the old image to the new. In the vn device case, you could just create a new empty FS of the necessary size, and dump from the old piped to a restore of the new. If you can live with the internal fragmentation, use growfs(8); if you can't, use dump/restore. IMO, you will have less potential for future problems if you use dump/restore. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: possible to expand a file for vn-device FS usage ?
On Thu, 2002-08-15 at 17:04, Patrick Thomas wrote: > Any suggestions on how to expand that file without doing the dump/restore > steps ? man 8 growfs perchance? :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 9A8C 569F 685A D928 5140 AE4B 319B 41F4 5D17 FDD5 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
