I have been running a lot of large compression tests to see how well Martijn van Beurden's new presets do and was once again reminded how real the fragmentation problem still is.
To have decent speed it's necessary to run multiple encoders in parallel. In my setup FLAC was compressing four files at the same time and each instance writes tiny bits of data to disk at once. A fragmentation analyzing tool reported that my files were in 300 fragments on average. Thanks to this decoding the files happened in a fraction of the speed that I should be getting. There would be one very simple solution to help against this issue, the use of setvbuf() command to buffer the file writes to larger chunks. Unfortunately it looks like the call has to be done inside libflac. The simple test change shown at the end of the mail reduced average fragmentation to two fragments per file and decode speeds were as good as without fragmentation. I don't know how big issue this is in *nix environments but added buffering would definitely be a nice change for the Windows frontend. diff --git a/src/libFLAC/stream_encoder.c b/src/libFLAC/stream_encoder.c index 6f46d78..bd95634 100644 --- a/src/libFLAC/stream_encoder.c +++ b/src/libFLAC/stream_encoder.c @@ -1288,6 +1288,8 @@ static FLAC__StreamEncoderInitStatus init_FILE_internal_( if(file == stdout) file = get_binary_stdout_(); /* just to be safe */ + setvbuf(file, NULL, _IOFBF, 10*1024*1024); /* 10MB output buffer to help reduce disk fragmentation */ + encoder->private_->file = file; encoder->private_->progress_callback = progress_callback; _______________________________________________ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev