We seem to be communicating at cross purposes here, so I'll edit heavily and do a partial reboot.
On Wed, Dec 16, 2009 at 10:30 AM, Peter Grandi <[email protected]> wrote: >> Several filesystems I tested with had over 40% fragmentation >> (as reported by e2fsck) > > Weren't we talking about JFS here? Yes. The statement was intended to show that, as a technique, either truncation (which can create sparse files), followed by a start-to-end copy of the data (essentially small small number of large writes) does a reasonable job of producing less-fragmented files. Since JFS doesn't have a good way for me to determine overall filesystem fragmentation levels, I used ext3 and jfs in my tests. In my tests, the results were promising. .. > But then if you already suspected that "heavily discontiguous > most likely because it was written incrememtally in many tiny > pieces", why suggest below that "write a \0 every 4K" is a good > idea? The algorithm of writing a \0 every 4K (or so) is a well-established mechanism for forcing the complete allocation of a file, and since one isn't writing /full/ blocks, it is frequently (but not always) substantially faster. There isn't any seeking because the direction is linear, and I'm not writing the data synchronously. The only potential downside would be if raid (with parity bits) was involved, in which case the read-update-write cycle would come into effect - a good reason to write full blocks of zeroes here (avoid the read and update as much as possible). Linux MD does a very reasonable job of caching blocks in stripes, potentially allowing full stripe writes without having to resort to the read and update steps whatsoever. With JFS that is unlikely due to the arrangement of the filesystem as compared to the raid striping. >> What I'm asking is what filesystem operations best suit large >> JFS allocations. > > I think that the answers are "continuous write of largish bits", > ideally to a "freshly formatted filesystem". The reasons why > may be inferred from the disk layout, described here: > > http://en.wikipedia.org/wiki/JFS_(file_system) > http://www.sabi.co.uk/Notes/linuxFS.html#jfsStruct > http://jfs.sourceforge.net/project/pub/jfslayout.pdf > >> Should I allocate the entire file by way of truncate? > > That should not allocate anything: > > base# grep /dev/root /proc/mounts > /dev/root / jfs rw 0 0 > base# perl -e 'truncate STDOUT,1000*1000*1000' > /1G > base# ls -lsd /1G > 4 -rw------- 1 root root 1000000000 Dec 16 16:27 /1G Sadly, I knew this and managed to forget all about it. > Again, that would be "continuous write of largish bits" and "on > a less busy filesystem", but pre-creating files is often > regrettably fairly pointless as many applications don't > overwrite files, but just open them with "O_CREAT", and that > truncates them to 0 before writing. Ideally applications would > (optionally) open and overwrite and truncate at the end, but > that is quite rare. What other applications do is somewhat beyond the scope, here. I'm really only interested in rewriting existing files, or the creation of new files, in the most optimal way. There is no need to respond, it's clear that there isn't much to say on the subject that is useful to me. -- Jon ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev _______________________________________________ Jfs-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/jfs-discussion
