Re: [LAD] [LAU] cancelling I/O with libsndfile
To me that suggests that either you have a completely unique problem to solve or that they are other solutions to your problem that other people use to get around the same problem. My guess is that your problem is not completely unique. Certainly not unique. My guess is that nobody worried about a (possibly small) performance penalty, and instead used simpler strategies -- or they weren't using sndfile in the first place. Consider this: for any request size you choose, NOT cancelling could double your latency in the worst case if you're streaming from a slow NFS server. Now, I've been able to implement my strategy by storing fd's and using sf_open_fd() as necessary. I.e. after a thread is cancelled in the middle of I/O (rendering the SNDFILE invalid), I re-open fd's using sf_open_fd(). This begs a question -- is it possible to reset the SNDFILE less dramatically? E.g. would a seek back to offset 0, or a pair of seeks, or some other trick also reset the data structures? If not, it might be nice to add a sf_reset() function or a SF_RESET sf_command() parameter. On second thought, though, this proposal could not possibly work without a userspace (libsndfile) cache, because read() might return incomplete frames, which would need to be processed in a later call. Modifying libsndfile to do fread/fwrite style buffering would be As I wrote above, and as Paul noticed, the virtual I/O layer (which I did not initially notice) is a good solution for this... Best, Dan ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
Dan Muresan wrote: Certainly not unique. My guess is that nobody worried about a (possibly small) performance penalty, and instead used simpler strategies -- or they weren't using sndfile in the first place. Consider this: for any request size you choose, NOT cancelling could double your latency in the worst case if you're streaming from a slow NFS server. Err, if you're worried about latency, why are you streaming from a slow NFS server? This begs a question -- is it possible to reset the SNDFILE less dramatically? E.g. would a seek back to offset 0, or a pair of seeks, or some other trick also reset the data structures? Really not sure. You should try it and report back. If not, it might be nice to add a sf_reset() function or a SF_RESET sf_command() parameter. What problem does that fix? Do you even know if there is a problem? Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/ ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
Err, if you're worried about latency, why are you streaming from a slow NFS server? The point was that NOT cancelling introduces extra latency unnecessarily, and short requests introduce CPU overhead. I gave the NFS example to preempt the hard disks are really fast objection that some people are always tempted to make... For the given example, if the NFS server can stream at even 110% of the playback speed without jitter, you can still achieve glitch-free, continuous playback with as low a *skip* latency as the CPU (and chosen request size) permit. *If* you can cancel active requests, that is -- otherwise you need to double that latency. And yes, I actually do stream from a fileserver (over NFS) myself quite often. be nice to add a sf_reset() function or a SF_RESET sf_command() parameter. What problem does that fix? Do you even know if there is a problem? The problem of a SNDFILE becoming invalid after a cancelled I/O. Yes, there is a problem -- valgrind reports nasty memory errors (and eventually I get a segfault) if I don't re-open the SNDFILE. [ If you meant I should test the sf_seek() trick first -- I doubt that testing a few sf_seek()s on a few FLAC files will reliably establish whether sf_seek() really does reset all encoder and decoder structure under all circumstances. Especially since you said you weren't sure yourself. ] -- Dan ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
b) Work around it. This is what I was trying to do via pthread_cancel + sf_open_fd. It seems to work with FLAC. However, there could be memory leaks if any decoder / encoder function (in sndfile or one of the external libs) allocates temporary memory that it expects to free before returning. Do you know if this might be the case (e.g. for the native formats, libFLAC, libvorbis)? I may need to fall back on the vio stuff. How does a sf_vio_read or sf_vio_write signal I/O errors? How does a vio_read signal EOF? The API docs don't seem to say anything. -- Dan ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
Dan Muresan wrote: b) Work around it. This is what I was trying to do via pthread_cancel + sf_open_fd. It seems to work with FLAC. However, there could be memory leaks if any decoder / encoder function (in sndfile or one of the external libs) allocates temporary memory that it expects to free before returning. Do you know if this might be the case (e.g. for the native formats, libFLAC, libvorbis)? No idea. I would be surpised if this works. I may need to fall back on the vio stuff. How does a sf_vio_read or sf_vio_write signal I/O errors? I never foresaw the possiblity of anybody doing this. I suspect they should return -1. How does a vio_read signal EOF? Just like the read() system call, by returning zero. Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/ ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
On Tue, Jun 14, 2011 at 01:01:09PM +0300, Dan Muresan wrote: b) Work around it. This is what I was trying to do via pthread_cancel + sf_open_fd. It seems to work with FLAC. 'Work around it' coild mean something entirely different : not trying to cancel anything, by using a system that doesn't require it. More below. It's probably a Good Thing (TM) that libsndfile doesn't do its own buffering. First, the file system will do buffering, and if you're playing a sound file probably decide it should read ahead. Second, any application that wants to use audio data in real time will need to do its own buffering, typically a few seconds. How exactly this is organised will be application dependent. Having a third layer in between this and the system is not going to help. If you support things like loops, random locates, shutthling, scrubbing, etc. the buffer management logic required to support all of this in an efficient way may become complicated. But only the application can do it. One way to organise the buffer is to divide it in fragments of say 1/4 second, and have the start offset of each one quantised to the same value. The syscall overhead becomes trivial for such a lenght. When you need new fragments, you send commands to the reader thread to read them. If the requirements change before they are read, you can still cancel all but one of them (you cancel the requests, not the actual read() call). The one you can't cancel is no big deal, just let it happen. It will take little time compared to what has to be done anyway. Ciao, -- FA ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
Hi Fons, It's probably a Good Thing (TM) that libsndfile doesn't do its own buffering. First, the file system will do buffering, and if I agree, I prefer the flexibility of implementing my own user-space cache in an app-dependent way, too. exactly this is organised will be application dependent. Having a third layer in between this and the system is not going to help. For sure, two user-space caches add a useless extra layer of copying. One way to organise the buffer is to divide it in fragments of say 1/4 second, and have the start offset of each one Yes, that's exactly how my implementation works. My ringbuffer is divided in fragments. quantised to the same value. The syscall overhead becomes trivial for such a lenght. When you need new fragments, you Yes, that's why I argued for long requests over short ones. the actual read() call). The one you can't cancel is no big deal, just let it happen. It will take little time compared And this is why I gave the NFS example earlier in the discussion... What if it the data comes from the network, e.g. a streaming server that has just 110% the bandwidth of the actual real-time playback? As I said, in this case you almost double the latency if you can't cancel the request. It might be an interesting idea to do new requests in parallel with cancelled ones, but this requires another layer of ringbuffer / memory management. -- Dan ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
that it expects to free before returning. Do you know if this might be the case (e.g. for the native formats, libFLAC, libvorbis)? No idea. I would be surpised if this works. It seems to works for FLAC, but I can't trust it either. Besides memory leaks, mutexes might remain locked, and indirectly deadlock sf_free(). I may need to fall back on the vio stuff. How does a sf_vio_read or sf_vio_write signal I/O errors? I never foresaw the possiblity of anybody doing this. I suspect they should return -1. Well, forget my cancelling strategy -- errors could occur even if one writes a trivial layer (vio_read == read). An I/O API *must* specify error handling. This is not some exotic use case... From TFC, psf_fread (): if (psf-virtual_io) return psf-vio.read ... (i.e. no processing), while the non-virtual branch returns the partial count if an error ever occurs. So I guess I should return some short count on error. Anyway, for cancelling requests EOF works -- indeed as long as one can get out of the sf_read jail free anything works. -- Dan ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
On Tue, Jun 14, 2011 at 02:42:18PM +0300, Dan Muresan wrote: For sure, two user-space caches add a useless extra layer of copying. Not only copying (which is cheap anyway), but also logic. Your application layer cache knows what you want to achieve, will have some ad-hoc strategies and try to do the right thing. The intermediate one doesn't have that and may well work against you. The same problem exists with the system level buffering, but you can't do much about that. One way to organise the buffer is to divide it in fragments of say 1/4 second, and have the start offset of each one Yes, that's exactly how my implementation works. My ringbuffer is divided in fragments. quantised to the same value. The syscall overhead becomes trivial for such a lenght. When you need new fragments, you Yes, that's why I argued for long requests over short ones. the actual read() call). The one you can't cancel is no big deal, just let it happen. It will take little time compared And this is why I gave the NFS example earlier in the discussion... What if it the data comes from the network, e.g. a streaming server that has just 110% the bandwidth of the actual real-time playback? As I said, in this case you almost double the latency if you can't cancel the request. I don't know what your app is doing, so I'll assume it's some sort of player. Now if you relocate, you send the commands to read the data at the new position to your reader thread. Assume your buffer is 2 seconds, so that's 8 commands to read 1/4 of second. You can't safely start playback again unless you have at least a second or so buffered. No assume you have a new relocate before that time. Again you send the commands to read 8 blocks of 1/4 second. There is some logic in the app that makes these cancel the previous ones that have not yet started. So you end up with 1 you can't cancel against 8 that have to be done anyway. That's not big loss. One of my players works this way. From a user point of view a relocate happens almost instantly. If the read bandwidth is just above what is required for continuous streaming, then very probably you can't support this sort of thing without extra delays. But even in this case they don't need to be very big. In the example above one extra fragment is read (the one you can't cancel) compared to the four or so you need to have done anyway before you can resume playing safely. So it takes as most 25% more time. If the reads happen on an NFS volume then even if the cancel would work on the client side that doesn't imply that the data won't be transmitted by the server anyway. So nothing would be gained by cancelling. Ciao, -- FA ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [LAU] cancelling I/O with libsndfile
The intermediate one doesn't have that and may well work against you. The same problem exists with the system level buffering, Yes. Well, the worse clash is between the kernel-space vs. the user-space caching strategy, but indeed you can't do much about it (unless you go the O_DIRECT way like databases). is 2 seconds, so that's 8 commands to read 1/4 of second. You can't safely start playback again unless you have at least a second or On transport relocation, I was planning to be un-safe, compromise on the cache pre-fill percentage and accept the possibility of a few application-level (not Jack-level) under-runs -- in the interest of (likely) gapless payback. The cache should catch up gradually, depending on the streaming-to-playback bandwidth ratio. But otherwise you are right. so buffered. No assume you have a new relocate before that time. I'm not considering this yet, but I see your point about there always existing a single un-cancellable request. If the reads happen on an NFS volume then even if the cancel would work on the client side that doesn't imply that the data won't be transmitted by the server anyway. So nothing would be gained This depends on whether it's a WAV or a compressed file, and on the NFS rsize and wsize mount params I guess (or I'm missing something). Thanks for the analysis. Best Dan ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
[LAD] Updates of zita-rev1 and zita-at1
Hello all, Some updates now available at http://kokkinizita.linuxaudio.org:/linuxaudio/downloads zita-rev1-0.2.1 * Nasty bug detected and fixed by Robin Gareus (thanks !) zita-rev1-0.2.1 and zita-at1-0.2.2 * Changed looks at bit (rotary knobs look different). * Added -pthread to linker flags, required for the GNU gold linker (suggested by Alessio Treglia, thanks !). * s/kokkinizita.net/linuxaudio.org/ everywhere. Ciao, -- FA ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] Updates of zita-rev1 and zita-at1
On 06/15/2011 12:26 AM, Fons Adriaensen wrote: Hello all, Some updates now available at http://kokkinizita.linuxaudio.org:/linuxaudio/downloads yay. yummy software updates :) zita-rev1-0.2.1 zita-rev1-0.2.1 and zita-at1-0.2.2 it seems that, despite the ending .bz2, both files are actually gzipped... ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev
