Re: [Fwd: [linux-audio-dev] info point on linux hdr]
I can only say that I am not using any form of preallocation and still get acceptable performance, even if I am forced to use quite big buffers to overcome to filesystem latencies. I tried to compare ext2 to FAT, but I do not get visible differences in terms of buffer usage spikes (= disk thread blocks for long time due to the filesystem being busy). Therefore I suspect it's not a matter of filesystem ( ext2 or FAT), but buffer flushing I (update,bdlush) and metadata updating. Anyway I hope that one day we will be able to simply use O_DIRECT and get RAW I/O access on a per-file basis, doing directly disk from/to userspace transfers. I think that this would help us to greatly reduce the needed buffersizes when doing audio multitrack recording. does anyone know if/when this is planned ? (2.3.x kernels already contain the O_DIRECT define, but it's currently ignored) PS: Does anyone know how to make a RAW I/O device on a spare disk partition, and then put an ext2 over it (running the whole partition in RAWIO mode) ? Is that possible ? If yes I could run my multitrack benchmarks to see how much we can gain in terms of needed buffersizes (should get smaller by at least 1/4 or so). Benno. On Fri, 14 Apr 2000, Andrew Clausen wrote: Hi all, Any comments? Received: by alpha (mbox clausen) (with Cubic Circle's cucipop (v1.31 1998/05/13) Sat Apr 15 07:57:05 2000) X-From_: [EMAIL PROTECTED] Fri Apr 14 22:17:10 2000 Received: from ginette.musique.umontreal.ca (ginette.MUSIQUE.UMontreal.CA [132.204.178.34]) by mx.alphalink.com.au (8.9.3/8.9.3) with SMTP id WAA08844 for [EMAIL PROTECTED]; Fri, 14 Apr 2000 22:17:08 +1000 Received: (from majordom@localhost) by ginette.musique.umontreal.ca (950413.SGI.8.6.12/950213.SGI.AUTOCF) id IAA19455 for linux-audio-dev-list; Fri, 14 Apr 2000 08:16:48 -0400 Received: from renoir.op.net (renoir.op.net [207.29.195.4]) by ginette.musique.umontreal.ca (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id IAA19452 for [EMAIL PROTECTED]; Fri, 14 Apr 2000 08:16:46 -0400 Received: from op.net (d-bm2-0b.ppp.op.net [209.152.194.43]) by renoir.op.net (o1/$Revision: 1.18 $) with ESMTP id IAA22739 for [EMAIL PROTECTED]; Fri, 14 Apr 2000 08:11:44 -0400 (EDT) Received: (from pbd@localhost) by op.net ($Revision: 1.2 $) id IAA11954; Fri, 14 Apr 2000 08:10:10 -0400 Date: Fri, 14 Apr 2000 08:10:10 -0400 Message-Id: [EMAIL PROTECTED] From: Paul Barton-Davis [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: [linux-audio-dev] info point on linux hdr Sender: [EMAIL PROTECTED] Precedence: bulk X-Mozilla-Status2: i mentioned in some remarks to benno how important i thought it was to preallocate the files used for hard disk recording under linux. i was doing more work on ardour yesterday, and had the occasion to create some new "tapes", of lengths from 2 to 40 minutes. the simple act of creating 24 5 minute WAV files on ext2, files in which every block has been allocated, takes at least 1 minute, perhaps as much as 2. i create the files by first opening them, then writing a zero byte every blocksize bytes (blocksize comes from stat(2)), except for the WAV header at the beginning. i hadn't done this in a while, but it reminded me of the near-impossibility of expecting the FS code to allocate this much disk space in real time under ext2. If someone knows of a faster way to allocate the disk space, please let me know. Its not that I have a problem right now, just wanted to point out this behaviour. --p
Re: [Fwd: [linux-audio-dev] info point on linux hdr]
On Sat, Apr 15, 2000 at 12:24:16AM +0200, Andrew Clausen wrote: i mentioned in some remarks to benno how important i thought it was to preallocate the files used for hard disk recording under linux. [...] Unfortunately efficient preallocation is rather hard with the current ext2. To do it efficiently you just want to allocate the blocks in the bitmaps without writing into the actual allocated blocks (otherwise it would be as slow as the manual write-every-block-from-userspace trick) Now in these blocks there could be data from other files, the new owner of the block should not be allowed to see the old data for security reasons. You suddenly get a new special kind of block in the file system with different semantics: ignore old data and only supply zeroes to the reader, unless the block has been actually writen. This ``ignore old data until writen'' information about the block would need to be persistent on disk -- you cannot just hold it in memory, otherwise it would not be known anymore after a reboot/crash. Filling the unwriten blocks with zeroes on shutdown would be too slow[1]. The problem is that ext2 has no space to store this information. The blocks are allocated using simple bitmaps, and you cannot express three states (free, allocated, write-only) in only a single bit. If you were using a extent based file system there would be probably enough space (you usually find space for a single bit somewhere in the extent tree), but with bitmaps it is tricky and would require on disk format changes. Apparently extent based ext2 is planned, maybe it would be useful to include that feature then, but before that it looks too hairy. JFS and XFS seem to support these things already. -Andi [1] Imagine your system starting a 100MB write when the UPS tries to force a quick shutdown on power fail -- you really don't want that.
[Fwd: [linux-audio-dev] info point on linux hdr]
Hi all, Any comments? i mentioned in some remarks to benno how important i thought it was to preallocate the files used for hard disk recording under linux. i was doing more work on ardour yesterday, and had the occasion to create some new "tapes", of lengths from 2 to 40 minutes. the simple act of creating 24 5 minute WAV files on ext2, files in which every block has been allocated, takes at least 1 minute, perhaps as much as 2. i create the files by first opening them, then writing a zero byte every blocksize bytes (blocksize comes from stat(2)), except for the WAV header at the beginning. i hadn't done this in a while, but it reminded me of the near-impossibility of expecting the FS code to allocate this much disk space in real time under ext2. If someone knows of a faster way to allocate the disk space, please let me know. Its not that I have a problem right now, just wanted to point out this behaviour. --p
Re: [Fwd: [linux-audio-dev] info point on linux hdr]
Andrew Clausen [EMAIL PROTECTED] writes: Hi all, Any comments?From: Paul Barton-Davis [EMAIL PROTECTED] Subject: [linux-audio-dev] info point on linux hdr To: [EMAIL PROTECTED] Date: Fri Apr 14 07:10:10 2000 -0500 i mentioned in some remarks to benno how important i thought it was to preallocate the files used for hard disk recording under linux. i was doing more work on ardour yesterday, and had the occasion to create some new "tapes", of lengths from 2 to 40 minutes. the simple act of creating 24 5 minute WAV files on ext2, files in which every block has been allocated, takes at least 1 minute, perhaps as much as 2. i create the files by first opening them, then writing a zero byte every blocksize bytes (blocksize comes from stat(2)), except for the WAV header at the beginning. I'm confused. You wrote what is conceptually 24x5=120 minutes of audio data in 2 minutes. So you have 60 times the performance you need. What is the problem? i hadn't done this in a while, but it reminded me of the near-impossibility of expecting the FS code to allocate this much disk space in real time under ext2. If someone knows of a faster way to allocate the disk space, please let me know. Its not that I have a problem right now, just wanted to point out this behaviour. If you blocksize was 1k increasing it to 4k could help. Eric
Re: [Fwd: [linux-audio-dev] info point on linux hdr]
Unfortunately efficient preallocation is rather hard with the current ext2. To do it efficiently you just want to allocate the blocks in the bitmaps without writing into the actual allocated blocks (otherwise it would be as slow as the manual write-every-block-from-userspace trick) yes, its slow, but its not hard to design an application so that its rare that it needs to be done. that said, when i was creating a 24 track, 40 minute "tape" for ardour the other day, i managed to reheat lunch, eat it, and play with my friend's son for a while during the creation process :) --p
Re: [Fwd: [linux-audio-dev] info point on linux hdr]
Date:Sat, 15 Apr 2000 01:35:10 +0200 From: Andi Kleen [EMAIL PROTECTED] On Sat, Apr 15, 2000 at 12:24:16AM +0200, Andrew Clausen wrote: i mentioned in some remarks to benno how important i thought it was to preallocate the files used for hard disk recording under linux. [...] Unfortunately efficient preallocation is rather hard with the current ext2. To do it efficiently you just want to allocate the blocks in the bitmaps without writing into the actual allocated blocks (otherwise it would be as slow as the manual write-every-block-from-userspace trick) Now in these blocks there could be data from other files, the new owner of the block should not be allowed to see the old data for security reasons. It depends on what you mean by preallocation. If you mean preallocation for the purposes of making sure you don't run out of disk space, then it's just a matter of keeping track of how many committed (but not yet allocated) blocks, so you can return ENOSPACE at the right time. If you mean preallocation to avoid fragmentation, ext2 does that already. When you allocate a block for an inode, if the following blocks are free, they will be grabed in the free block bitmap, although they won't be assigned to the inode. The in-core inode structure stores the first preallocated block address, and how many blocks were grabbed. If the file is closed without using all of the preallocated blocks, they are released back to the filesystem. You can control the maximum number of blocks that are grabbed for preallocation purposes changing a s_prealloc_blocks stored in the ext2 superblock. - Ted
