Re: Bug#686777: netjack2 + opus custom modes + debian
Hi, So the background (that's been missing from the BTS up until now) is that shortly after Robin initially reported this bug, he also contacted us upstream and we had a fairly detailed discussion on IRC about all the various issues. Since Debian was in freeze there wasn't much going to happen with the package right then, and it wasn't clear at the end of the discussion whether netjack was still going to use the custom modes after all. But it is time now to decide on this soon, so I got in touch with Robin again yesterday to see what constraints we really have to work with. My understanding of the background prior to that is that Robin had some discussion with some of the developers at FOMS, who at the time suggested the custom modes probably would be appropriate for the use described to them. The later discussion on IRC however (including the developers who were at FOMS) was much less conclusive, and it wasn't at all clear after that, that the need to do this outweighed the costs, both in general, and to jack specifically. The custom modes are not interoperable with anything else, nor are they a part of the codec standard, but they do exist in the code for people with very specialised needs in 'closed' applications, where the need for oddball frame sizes strongly outweighs any other considerations of interoperability, or codec performance (the latter being both in the sense of processing resources *and* more importantly audio quality). My understanding at present is that the primary (only?) reason that netjack is using custom modes is so that it can use 64 sample frames which shaves ~1ms of latency off the usual 2.5ms (120 sample) minimum frame size for normal opus modes. We didn't quite get to the bottom of all of that before Robin had to leave, so at present my only understanding of the reason for that is that "pro audio equipment" can operate with lower latencies than normal sound cards which makes this desirable. What I still don't understand though is why if you are using Pro audio equipment the degradation in audio quality that this would bring (which is significant) would be acceptable for that use? You'd need to stream at much higher bitrates than normal to recover even some of that, and even then there are many quality enhancements in the encoder that only apply to the normal modes, which are completely bypassed in the custom modes. And even without those, quality will still suffer compared to the more normal analysis frame sizes used by this codec, just by virtue of the tiny window size. Even 2.5ms frames have notably lower quality than the default 20ms ones do. Non-standard 1.3ms frames are at a considerable disadvantage here for high fidelity reproduction. The 'normal' latency of Opus is orders of magnitude lower than anything which can even approach it for quality, but even it has both hands tied behind its back when it only has 64 samples to work its magic on. Which basically makes the question become: "If you are using Pro audio equipment and ~1ms of latency does make a difference to you, then wouldn't a lossless transport mode be more appropriate for that anyway?" Which isn't exactly the original question that we need to answer here, but it is relevant to that, since netjack is the only thing that I'm aware of that's likely to want support for custom modes from the distro packages. So the question of whether it's actually gaining any real benefit from this is the key to knowing whether we even need to consider supporting custom modes in the distro in the foreseeable future. The upstream developers have reaffirmed that they definitely do not want to enable the custom modes by default in what they release, so even if we do override that here for the .debs, there'll still be a question of our compatibility with other distros and users. Re Jonas's remarks: > In my opinion the best option so far is for libopus to enable custom > modes: Primary aim in Debian is to enable most possible features - being > fastest possible has lower priority so can wait until done properly. > > ...and convenience code copies is explicitly discouraged in Policy, so > range far lower on the list! I concur with this, which is why I revisited the question of whether we would even need to with Robin. And the other upstream developers agree that should we really need to, enabling this for the packages is probably the preferred option (unless we really can think of some better way instead). There isn't really a "wait until done properly" about this though, the penalties are largely inherent in the extra complexity this adds. Part of the catch here is that while enabling this will not break ABI, disabling it again will, so this is very much a one-way decision which I'd like to not make until we are certain there will be no turning back from it, or second thoughts, or regrets over mistakes that could be remedied now in better ways instead. On Sat, Jun 29, 2013 at 07:36:57PM +0200, Adrian Knoth
Re: Bug#686777: netjack2 + opus custom modes + debian
On 06/30/2013 03:11 AM, Ron wrote: > My understanding of the background prior to that is that Robin had some > discussion with some of the developers at FOMS, who at the time suggested > the custom modes probably would be appropriate for the use described to > them. correct. derf aka Tim Terriberry in this case. [..] > The custom modes are not interoperable with anything else, nor are they > a part of the codec standard, but they do exist in the code for people > with very specialised needs in 'closed' applications, where the need for > oddball frame sizes strongly outweighs any other considerations of > interoperability, or codec performance (the latter being both in the > sense of processing resources *and* more importantly audio quality). jack in particular was one of the use-cases for opus-devs to justify custom modes. > My understanding at present is that the primary (only?) reason that > netjack is using custom modes is so that it can use 64 sample frames > which shaves ~1ms of latency off the usual 2.5ms (120 sample) minimum > frame size for normal opus modes. We didn't quite get to the bottom > of all of that before Robin had to leave, so at present my only > understanding of the reason for that is that "pro audio equipment" > can operate with lower latencies than normal sound cards which makes > this desirable. not quite. netjack is using opus custom modes so that jack can use the same period-size across the complete jack system. Adding buffering on either side (sender + receiver) to align jack + opus buffers will always result in additional latency. For large jack buffersizes or long-distance communication that additional latency may be negligible, but it still is more latency. Furthermore, aligning non-audio jack-data (transport + MIDI) with sample accuracy to those opus-audio-buffers is far from trivial. It's not impossible, but it is quite complex because jack is not designed to cater for that case. > What I still don't understand though is why if you are using Pro audio > equipment the degradation in audio quality that this would bring (which > is significant) would be acceptable for that use? a) because some users demand it :) b) because celt is no longer available on most distros low, fixed latency is most important. There are countless solutions for high-quality streaming - where latency and jitter is irrelevant, but basically only netjack that provides synchroneous low latency. [..] > Which basically makes the question become: "If you are using Pro audio > equipment and ~1ms of latency does make a difference to you, then > wouldn't a lossless transport mode be more appropriate for that anyway?" on a LAN, yes lossless. Over Wifi it may make sense to compress lossy to accommodate more channels. On WAN there are e.g. remote jam-sessions, phone relays, live monitoring,.. - none of which requires high quality, but all require fixed low latency. [..] > The upstream developers have reaffirmed that they definitely do not > want to enable the custom modes by default in what they release, so > even if we do override that here for the .debs, there'll still be a > question of our compatibility with other distros and users. yes, the solution for that would be to add opus as git-submodule to jack and statically link netjack against it. That'd also accommodate windows, OSX and *BSD builds of jackd. [..] > - Can jack really make a case for needing this in a way that actually >delivers real benefits to jack users. (Robin has said that this is >also 'complicated', but I still don't fully understand why yet). see above. Sample-sync alignment with other data-types is not easy. Asynchronous (buffered) communication is orthogonal to everything else in jack. It will likely be rejected upstream. jack does not aim to do everything. JACK tries to address 95% and do that right and not care about the last 5% edge-cases. On top of of that, there are currently no volunteers to implement "vanilla opus" on netjack2 (and also no volunteer to implement that in netjack1). I was scratching my own itch with netjack2+opus. works for me. The only case for non-custom modes would be: 1) interoperability with other opus apps 2) higher quality encoding (1) is never going to work out. netjack consists of N audio-channels, M midi-channels. Both include per-port latencies (min,max). And netjack also comprises transport information (timecode, tempo, bar-beat-tick, audio-frames per video-frame, etc). It is not a data stream that will be consumed by non-jack. (2) if a user chooses lossy encoding s/he does not really care about quality anyway. jack's main features is no-copy zero-latency with local clients, being able to include remote clients on the network that align sample-sync and respond reliably is the main use-case. ciao, robin ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.
Re: Bug#686777: netjack2 + opus custom modes + debian
On 06/30/2013 03:11 AM, Ron wrote: Hi! I'll limit my response to the aspect of symbols, since Robin has already answered the other questions. >> Just sketching now: >> >> libopus0 will provide /usr/lib/libopus.so.0 (business as usual) >> libopus-custom-0 will provide /usr/lib/libopus-custom.so.0 > > The big problem with this is that both of those will provide all of > the functions that libopus.so.0 does, only some of the symbols with > the same names will have different implementations in the -custom one. > > Which means that when jack links to -custom, and jill links to -vanilla, > and then some high level audio app or desktop environment or whatever > links to both jack and jill ... hilarity is likely to ensue. I've seen colliding symbols with ardour via indirect linking, and it's really a PITA to diagnose. But here it seems to be very unlikely: only the jack server links against libopus(-custom), and this server is a standalone binary that's not linking or linked to anything else. All the clients link against libjack, and even if they do link against libopus, they're not interfering with the server's libopus-custom, since client-server communication is done via /dev/shm. So I think we can ignore the symbol aspect for all practical cases. (Correct me if I'm wrong). Cheers ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Re: Bug#686777: netjack2 + opus custom modes + debian
On Mon, Jul 01, 2013 at 11:15:10AM +0200, Robin Gareus wrote: > On 06/30/2013 03:11 AM, Ron wrote: > > > My understanding of the background prior to that is that Robin had some > > discussion with some of the developers at FOMS, who at the time suggested > > the custom modes probably would be appropriate for the use described to > > them. > > correct. derf aka Tim Terriberry in this case. Right, and it was Tim who ran through the math in the IRC discussion to point out that this might not have been the best suggestion after all. > > The custom modes are not interoperable with anything else, nor are they > > a part of the codec standard, but they do exist in the code for people > > with very specialised needs in 'closed' applications, where the need for > > oddball frame sizes strongly outweighs any other considerations of > > interoperability, or codec performance (the latter being both in the > > sense of processing resources *and* more importantly audio quality). > > jack in particular was one of the use-cases for opus-devs to justify > custom modes. Greg was probably the one who most often raised oddball cases that might use this, but even he was now scratching his head somewhat at what you are actually trying to achieve with using it for netjack as you are. Which is why I wanted to try to get to the bottom of your rationale here. > > My understanding at present is that the primary (only?) reason that > > netjack is using custom modes is so that it can use 64 sample frames > > which shaves ~1ms of latency off the usual 2.5ms (120 sample) minimum > > frame size for normal opus modes. We didn't quite get to the bottom > > of all of that before Robin had to leave, so at present my only > > understanding of the reason for that is that "pro audio equipment" > > can operate with lower latencies than normal sound cards which makes > > this desirable. > > not quite. > > netjack is using opus custom modes so that jack can use the same > period-size across the complete jack system. What gets to pick the period here? Even custom modes still limit you to some discrete number of choices, it isn't continuously variable, so there will surely still be cases where some part of the system needs to adapt to suit other parts if this is to be true, won't there? > Adding buffering on either side (sender + receiver) to align jack + opus > buffers will always result in additional latency. > > For large jack buffersizes or long-distance communication that > additional latency may be negligible, but it still is more latency. You realise that we are talking about a latency that is equivalent to moving your head approximately 15 inches from your speakers, right? Or in the case of a 'jam session', two players standing about a foot from each other ... with their instruments held a foot from each others chins. If they are any further apart than that, in the same room, then the latency difference that you are worrying over here becomes insignificant by comparison. If they stand a guitar length apart and put headphones on, even the normal mode of opus will get the sound to their ears faster than the air between them would. It doesn't take large buffer sizes or long distances to make this negligible. We're not talking coarse yak hairs here, it's the finest angora. > Furthermore, aligning non-audio jack-data (transport + MIDI) with sample > accuracy to those opus-audio-buffers is far from trivial. Elastic stores really are trivial. So I'm still not really sure what showstopper complexity you are worried about there. > > What I still don't understand though is why if you are using Pro audio > > equipment the degradation in audio quality that this would bring (which > > is significant) would be acceptable for that use? > > a) because some users demand it :) What exactly are users demanding here? They surely aren't demanding that members of the band stand on each others toes and play their guitars with their teeth. If they want to use oxygen free copper that's fine, but what _real_ gain are they demanding to get here? That's the important question we need to answer. > b) because celt is no longer available on most distros It arguably should have never been available on them in the first place :) But that was a learning experience for us too! > low, fixed latency is most important. > > There are countless solutions for high-quality streaming - where latency > and jitter is irrelevant, but basically only netjack that provides > synchroneous low latency. You can still have low latency with standard opus modes. If you take one small step toward your speaker it can even be lower than the custom mode you are currently using :) > > Which basically makes the question become: "If you are using Pro audio > > equipment and ~1ms of latency does make a difference to you, then > > wouldn't a lossless transport mode be more appropriate for that anyway?" > > on a LAN, yes lossless. Over Wifi it may make sense to com
Re: Bug#686777: netjack2 + opus custom modes + debian
On Mon, Jul 01, 2013 at 04:40:19PM +0200, Adrian Knoth wrote: > On 06/30/2013 03:11 AM, Ron wrote: > > Hi! > > I'll limit my response to the aspect of symbols, since Robin has already > answered the other questions. > > > >> Just sketching now: > >> > >> libopus0 will provide /usr/lib/libopus.so.0 (business as usual) > >> libopus-custom-0 will provide /usr/lib/libopus-custom.so.0 > > > > The big problem with this is that both of those will provide all of > > the functions that libopus.so.0 does, only some of the symbols with > > the same names will have different implementations in the -custom one. > > > > Which means that when jack links to -custom, and jill links to -vanilla, > > and then some high level audio app or desktop environment or whatever > > links to both jack and jill ... hilarity is likely to ensue. > > I've seen colliding symbols with ardour via indirect linking, and it's > really a PITA to diagnose. > > But here it seems to be very unlikely: only the jack server links > against libopus(-custom), and this server is a standalone binary that's > not linking or linked to anything else. > > All the clients link against libjack, and even if they do link against > libopus, they're not interfering with the server's libopus-custom, since > client-server communication is done via /dev/shm. > > So I think we can ignore the symbol aspect for all practical cases. > (Correct me if I'm wrong). Where does libjacknet.so.0.1.0 fit into all of that? I don't think we can guarantee that for the forever future something using the custom modified symbols would be compatible with the normal builds. Optimisation work is really only just beginning, and there are quite a few places where the normal code might diverge in newly incompatible ways from what is possible when custom is enabled, and where a symbol collision could be even worse than it is at present. This could turn into a snowball of ugly pretty easily I fear ... Which is why I'm really keen to be sure we're not going down this path for something so tiny that nobody will ever be able to hear it. Cheers, Ron ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Re: Bug#686777: netjack2 + opus custom modes + debian
On 07/01/2013 05:59 PM, Ron wrote: [..] > So I'm still not really sure what > showstopper complexity you are worried about there. Sample accurate alignment of buffered netjack streams with the rest of jack. updating port-latencies,.. Sounds easy, but it's not. The realshowstopper there is lack of manpower. No one volunteered to implement it. ciao, robin ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
Re: Bug#686777: netjack2 + opus custom modes + debian
On Wed, Jul 03, 2013 at 03:30:51AM +0200, Robin Gareus wrote: > On 07/01/2013 05:59 PM, Ron wrote: > [..] > > So I'm still not really sure what > > showstopper complexity you are worried about there. > > Sample accurate alignment of buffered netjack streams with the rest of > jack. updating port-latencies,.. Sounds easy, but it's not. You do know that you've _already_ got to deal with this if you care about it when using Opus right? The first few samples out of the decoder aren't the first few samples that you put into the encoder, whatever mode you use. And yeah, I'm not saying it's a one-liner, but it's not squaring the circle either, it is a fairly normal part of anything dealing with windowed codecs. > The realshowstopper there is lack of manpower. > No one volunteered to implement it. Ok, that's fine. There are no answers here that don't involve work that still needs to be done, which hasn't been done yet and that someone will need to do if they want this to happen. There's still no answer to a sensible way to distribute -custom builds in a shared environment yet either. Upstream is very much focussed on just the standardised modes, and custom is still a hack, that people with total control over a closed system can use if they really need it, and if they are prepared to deal with taking full responsibility for anything that may get broken with it (or be completely untested) over subsequent releases. That's not really a state of affairs that we want if we're going to expose it in general purpose distro libraries (either from the libopus package or embedded in other public libs). Sooner or later some change will bite people badly. So this needs to be resolved if there ever is anything that's to go in the distro that uses them. But while there isn't, it can also wait until more urgent things are completed too, and everything just settles down a bit in general and stops changing as rapidly as it currently is. What I'm concerned about here is that we figure out what the _best_ answer is, so that when someone does have time to do it, they don't waste it chasing up the wrong tree. Right now, it really is sounding like the best answer for what we have at present is for someone to look at the jack code and do the work that is needed there, to account for the initial padding and window latency. If that's the real showstopper, that's what people should focus on fixing first. I'm still open to suggestions for good ways to manage the how we might enable custom problem, but that seems orthogonal to also fixing jack in the way that will work with best results there, and less pressing until there is a definite need for it. Cheers, Ron ___ pkg-multimedia-maintainers mailing list pkg-multimedia-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers
