Re: [music-dsp] [admin] list etiquette
What is your problem? You can approach the mailing list and discuss whatever topic you want. Nothing is lost, don't make such a fuss. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [admin] list etiquette
You're speaking about an event that happened in the past. Which has nothing to do with the present, or the future, or the accessibility of this mailing list. You can learn from the mistakes or faults of others, can't you? For example, if someone wants to construct a windowed sinc FIR filter with 20,000 coefficients, it can be useful to learn *why* that idea is not the best. Without pointing out to *why* that idea may fail, you cannot learn from it, can you? As the discussion was about things like, FIR filters with lots of coefficients, upsampling, interpolation, convolution, frequency response of interpolators and various other polynomials... Topics that are all relevant here, I guess. I think at least 90%+ of the discussion was strictly about these topics. If you cannot dismiss the other 10% as noise, then you focus too much on the (for you) irrelevant details. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [admin] list etiquette
I decided to unsubscribe from this mailing list. I'm busy anyways working on a new Z-domain filter design method. Have a nice dup swap drop action. Bye. -P ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [admin] list etiquette
On 24/08/2015, Theo Verelst theo...@theover.org wrote: I'm not going to confuse etiquette with thinking straight, it's clear if people can be respected and have some things to learn or teach, or go on about emotionally, that a lot of reasonable communication can take place. Note: when I tell someone that he is wrong, in case when he is provably wrong, that has nothing to do with emotions. So do not confuse emotion with logic. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 22/08/2015, Ethan Duni ethan.d...@gmail.com wrote: So your whole point is that it's not *exactly* sinc^2, but a slightly noisy version thereof? My point was that there are no effects of resampling visible in the graphs. And you're wrong - all those 88 alias images are effects of resampling... That has nothing to do with exactly how the graphs were generated, nor does insisting that the graphs are slightly noisy address the point. Well, it was *you* who insisted that it displays a graphed sinc^2 curve, and not a resampled signal... And you were wrong. Indeed, you've already conceded that the resampling effects are not visible in the graphs several posts back. Aren't all those 88 alias images effects of resampling? What are those, if not effects of resampling? You claimed no upsampling is involved, yet when I upsample noise, I get exactly that graph. So it seems you were wrong. It seems like you're just casting about for some other issue that you can tell yourself you won, and then call me names, to feed your fragile ego. Well, if you do not see that the curve is NOT a graphed sinc^2, but rather, a noisy curve seemingly from resampled noise, then you have some underlying problem. Honestly, it's a pretty sad spectacle and I'm embarrassed for you. I'm embarrassed for you. It really would be better for everyone - including you - if you could interact in a good-faith, mature manner. Please make an effort to start doing so, or you're pretty soon going to find that nobody here will interact with you any more. Yet - for some reason - you keep interacting with me for the past 22 mails you wrote. Maybe to feed your fragile ego and prove that you won... (?) By the way, there's no reason for any jaggedness to appear in the plots, given the lengths of data you were talking about. There *is* reason for jaggedness to appear in the plots. If you don't believe, try it yourself - take some white noise sampled at 500 Hz, and resample it to 44.1 kHz. The shorter the length, the more jagged the spectrum will look. Besides, we do not know how much data Olli processed, so you cannot say there's no reason for jaggedness in his graph - as you do not know how he derived his graph. So your argument is invalid again. Producing a very smooth graph from a long enough segment of data is straightforward, if you use appropriate techniques (not just one big FFT of the whole thing, that won't ever get rid of the noisiness no matter how much data you throw at it). Exactly. And that's what I used (spectral averaging over a long segment), yet it is STILL noisy, if the white noise segment is not very long. So your argument is wrong again... -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] unsubscribe
On 22/08/2015, Alen akoe...@rogers.com wrote: Indeed. This debate is getting tiresome. That's what happens when someone does not accept that he is wrong, despite overwhelming evidence. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
So you claim that the graph depicts a sinc^2 graph, and it shows the frequency response of a continuous time linearly interpolated signal, and involves no resampling. That is false. That is not how Olli created his graph. First, the continuous time signal (which, by the way, already contains an infinite amount of aliases of the original spectrum) exists only in your imagination - I'm almost 100% certain Olli made his graph by resampling noise. The telltale signs of this are: - the curves on the graph are jagged/noisy, typical of averaged white noise spectrum - if you watch closely, the same jaggedness repeats at a 2*PI frequency interval, showing that they are aliases of the original spectrum, which was noisy. Therefore, Ollis graph does *not* depict a continuous time signal, but rather, a noisy signal that was resampled to 44.1 kHz. Therefore, what you see on the graph, is the artifacts from the resampling. Therefore, all your arguments are invalid. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
So let me get this straight - you have an *imaginary* graph in your head, depicting the frequency response of a continuous time linearly interpolated signal, and you keep arguing about this *imaginary* graph (maybe to feed your fragile ego and to prove that you won). That is *not* what you see on Olli's graph, as been discussed in depth. So what you're arguing about, is not Olli's graph that was presented, but rather, an *imaginary* graph, that exists only in your head. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
And besides, no one ever said that Olli's graph depicts analyitical frequency responses of continuous time interpolators. The graphs come from a musicdsp.org code entry: http://musicdsp.org/archive.php?classid=5#49 There's no comment whatsover, just the code and the graphs. If you read his 65 page long paper on interpolators, he doesn't discuss analytical continuous time interpolator frequency responses whatsoever. He just shows their graphs, and tells where they have zeros in the response. No formulas for analytical frequency responses, at all - seemingly he is not interested in that. I just skimmed through his paper again, and the closest thing that he has in it, are polynomial approximations for frequency responses in the passband. About that's all, other than that, no frequency response formulas. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 22/08/2015, Sampo Syreeni de...@iki.fi wrote: The conjugate sine to +1, -1, +1, -1, ... is 0, 0, 0, 0... Just phase shift the original sine at the Nyquist frequence. Let me ask what do you mean by conjugate sine ? If you mean complex conjugate, and assume the sine to be the real part of complex phasor rotating around the complex unit circle, then isn't the conjugate of that phasor also +1, -1, +1, -1,... ? The only difference is that the phasor is mirrored around the X axis (so the imaginary part +i becomes -i), so it rotates in the opposite direction (negative frequency). Since the frequency of that phasor is pi, the complex conjugate phasor rotating at the other direction is also +1, -1, +1, -1... Either direction, the phasor toggles between positions z=1 and z=-1. Maybe you meant quadrature sine ? That'll show you that that precise signal cannot be reconstructed without resorting to complex continuation of the signal, on the Fourier plane. Let me ask, what do you mean by Fourier plane? I never heard that term, and Google only gives me optics-related pages. Maybe you mean complex plane? ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Okay, I'll risk exceeding my daily message limit. If the administrators think it is inappropriate, dealing with that is at their discretion. Here is another proof that the alias images in the spectrum are caused by the sampling/upsampling, not the interpolation: Let's replace linear interpolation with simply stuffing zeros between samples. So that means, we upsample the signal without applying interpolation or filtering. Let's try this on an ~50 Hz sine wave sampled at 44100/88 ~= 501 Hz, upsampled to 44.1 kHz by stuffing 87 zeros between each sample. The resulting waveform looks like individual impulses, spaced 88 samples apart: http://morpheus.spectralhead.com/img/sine_upsampled_waveform.png Here is the spectrum: http://morpheus.spectralhead.com/img/sine_upsampled_spectrum.png We can see the usual alias frequencies at 450 Hz, 550 Hz, 950 Hz, 1050 Hz, 1450 Hz, 1550 Hz, 1950 Hz, 2050 Hz, ... This is because the upsampling causes the original spectrum to repeated infinite times, causing these alias frequencies to appear in the resulting spectrum. Therefore, it is NOT the interpolation that is causing these alias images, but rather, the upsampling... More precisely, they're already present in the original signal sampled at 500 Hz, the upsampling just makes them visible. I used no interpolation at all, yet all this aliasing appeared on the spectrum. All the interpolation does, is it filters out some of this aliasing... Since the impulse response of linear interpolation is a triangle, applying linear interpolation is equivalent to convolving the resulting upsampled signal with a triangular kernel filter. Since the Fourier transform of a rectangle is a sinc function, and a triangular kernel is equivalent to convolving two rectangular kernels, the Fourier spectrum of a triangular kernel will look like a sinc^2 function. But that's not what causes the aliasing... it's there already after the upsampling, before you apply the interpolation/convolution. You can take a discretized version of a continuous triangular kernel sampled at the upsampled rate, and convolving the upsampled signal with that kernel will be equivalent to linear interpolation. You do not actually need a continuous time signal to be present, and the aliasing/imaging is there already before doing the triangular convolution at the upsampled rate. Several authors discuss the equivalence of linear interpolation and convolution with a triangular filter, examples: 1) linear interpolation can be expressed as convolving the sampled function with a triangle function[1] http://morpheus.spectralhead.com/img/linear_interpolation1.png 2) The first-order hold [= linear interpolation] corresponds to an impulse response for the reconstruction filter that is a triangle of duration equal to twice the sampling period.[2] http://morpheus.spectralhead.com/img/linear_interpolation2.png 3) http://morpheus.spectralhead.com/img/linear_interpolation3.png [1] Oliver Kreylos, Sampling Theory 101 http://idav.ucdavis.edu/~okreylos/PhDStudies/Winter2000/SamplingTheory.html [2] Alan V. Oppenheim, Signals and Systems, ch. 17. Interpolation http://ocw.mit.edu/resources/res-6-007-signals-and-systems-spring-2011/lecture-notes/MITRES_6_007S11_lec17.pdf [3] Ruye Wang, Sampling Theorem, Reconstruction of Signal by Interpolation http://fourier.eng.hmc.edu/e101/lectures/Sampling_theorem/node3.html -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Also, you even contradict yourself. You claim that: 1) Olli's graph was created by graphing sinc(x), sinc^2(x), and not via FFT. 2) The artifacts from the resampling would be barely visible, because the oversampling rate is quite high. So, if - according to 2) - the artifacts are not visible because the oversampling is high and the graph doesn't focus on that, then how do you know that 1) is true? You claim that the resampling artifacts wouldn't be visible anyways. If that's true, then how would you prove that FFT was not used for creating Olli's graph? Also, even you yourself acknowledge that It shows the aliasing left by linear interpolation into the continuous time domain. So, we agree that the graph shows aliasing, right? I do not know where you get your idea of additional aliasing - it's the very same aliasing, except the resampling folds it back... ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 21/08/2015, Ethan Duni ethan.d...@gmail.com wrote: So you agree that the effects of resampling are not shown, and all we see is the spectrum of the continuous time polynomial interpolators. I claim that they are aliases of the original spectrum. Just as you also call them: It shows the aliasing left by linear interpolation into the continuous time domain. I never claimed anything about the folding back of alias frequencies from the resampling at 44.1k rate. If I were you, I'd quit haranguing people over irrelevancies and straw men, It's rather you who argue about irrelevant things and straw man arguments - for that matter, I never claimed that the folded back aliases from the resampling at 44.1k are visible on Olli's graph. It's you who is forcing this irrelevant argument. So maybe you should listen to your own advice, first. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
A sampled signal contains an infinte number of aliases: http://morpheus.spectralhead.com/img/sampling_aliases.png the spectrum is replicated infinitely often in both directions These are called aliases of the spectrum. You do not need to fold back the aliasing via resampling for them to become aliases... They're aliases already - when sampled at the original rate, they would all alias back to the original signal. This is because exp(i*x) is periodic, and after 2*PI radians, you get back to the same frequency... hence, frequencies that are 2*PI apart from each other, are all aliases... If you fail to understand that, I think you fail to understand even the basics of sampling theory. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Let's repeat the same with a 50 Hz sine wave, sampled at 500 Hz, then linearly interpolated and resampled at 44.1 kHz: http://morpheus.spectralhead.com/img/sine_aliasing.png The resulting alias frequencies are at: 450 Hz, 550 Hz, 950 Hz, 1050 Hz, 1450 Hz, 1550 Hz, 1950 Hz, 2050 Hz, 2450 Hz, 2550 Hz, ... I think it should be obvious that these are all alias frequencies of 50 Hz, since if you sample any of these sinusoids at 500 Hz rate, they will all alias to 50 Hz. Hence, they are - by definition - aliases of the 50 Hz sinusoid. Welcome to sampling theory 101. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 21/08/2015, Ethan Duni ethan.d...@gmail.com wrote: It shows *exactly* the aliasing It shows the aliasing left by linear interpolation into the continuous time domain. It doesn't show the additional aliasing produced by then delaying and sampling that signal. I.e., the images that would get folded back onto the new baseband, disturbing the sinc^2 curve. This image doesn't involve any fractional delay. Those differences would be quite small for resampling to 44.1kHz with no delay, since the oversampling ratio is considerable, so you'd have to look carefully to see them. I think they're actually on the image: http://morpheus.spectralhead.com/img/resampling_aliasing.png They're hard to notice, because the other aliasing masks it. This is a big hint that they are not portrayed: Ollie knows what he is doing, so if he wanted to illustrate the effects of the resampling, he would have constructed a scenario where they are easily visible. Since that image is not meant to illustrate the effects of resampling, but rather, to illustrate the effects of interpolation, *obviously* it doesn't focus on the aliasing from the resampling. Therefore, it is not a hint at all, and your argument is invalid. And probably mentioned a second sample rate, explicitly shown both the sinc^2 and its aliased counterpart, etc. The effect would be shown in a visible, explicit manner, if that was what the graph was supposed to show. The fact that this graph is not supposed to demonstrate the aliasing from the resampling, does not mean that 1) it's not there on the graph (it's just barely visible) 2) the images of the continuous time interpolated signal are not aliasing. That's also called aliasing!!! But all of those things depend on parameters like oversampling ratio and delay, so it would be a much more complicated picture. Yes, and that's all entirely irrelevant here... Because the images in the continuous time signal before the resampling are also called aliasing!!! They're all aliases of the original spectrum, and they all alias back to the original spectrum when sampled at the original sampling rate! They're called aliasing even before you resample them! What we're shown here is just the effects of polynomial interpolation to get to the continuous time domain. False. I've shown the FFT frequency spectra of actual upsampled signals. The additional effects of delaying and then sampling that signal back into the discrete time domain are not visible. There was no delaying involved at all. The effects of sampling that signal back are not visible, because there's 88x oversampling, just as I pointed out. If you want, you can repeat the same with less oversampling, and present us your results. It seems that you have assumed that some resampling must be happening because the graph only goes up to 22kHz. But that's just the range of the graph, you don't need to do any resampling of anything to graph sinc^2 over any particular range of frequencies. I never said you need do to resampling of the continuous time signal to graph sinc^2. I said: the images in the frequency spectrum of the continuous time signal are aliases of the original spectrum, and they alias back to the original spectrum when the continuous time signal is sampled at the original rate! But that's not quite the exact same graph. It's essentially the exact same graph. And why are you putting a sound card in the loop? That was the most convenient way to record the signal. This is all just digital processing in question here. You don't even need to process any signals, there are analytic expressions for all of the quantities involved. That's just one way of drawing fancy graphs. FFT is another way of drawing fancy graphs. Why would I restrict myself to one method? That's how Ollie generated graphs of them without reference to any particular signals. How do you know? Prove it! I'm convinced he generated it via numerical means and FFT. Again, the differences in question are small due to the high oversampling ratio, so it's going to be quite difficult to see them in macroscopic graphs like this. Let me point out again, that all those spectral images in the continunous time signal before the resampling, *are* aliasing, as they're aliases of the original spectrum, and are *very* visible on the graph! If you want to see the differences, just make a plot of both sinc^2 and its aliased versions (for whatever oversampling ratios and/or delays), and look at the differences. It won't be interesting for high oversampling ratios and zero delay - which is exactly why that scenario is a poor choice for illustrating the effects in question. And you're entirely missing the point what it is supposed to illustrate. The fact that sampling a continuous time signal at a very high rate results in a spectrum that closely resembles the continuous time spectrum (over the sampled bandwidth) is beside the point. Exactly. It's
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 21/08/2015, Ethan Duni ethan.d...@gmail.com wrote: Creating a 22000 Hz signal from a 250 Hz signal by interpolation, is *exactly* upsampling That is not what is shown in that graph. The graph simply shows the continuous-time frequency response of the interpolation polynomials, graphed up to 22kHz. No resampling is depicted, or the frequency responses would show the aliasing associated with that. It shows *exactly* the aliasing http://morpheus.spectralhead.com/img/interpolation_aliasing.png There are about 88 alias images visible on the graph. The linear interpolation curve is not smooth, so it contains aliasing. It's just showing the sinc^2 response of the linear interpolator, and similar for the other polynomials. If the signal you interpolate is white noise, and the spectrum of the signal is a flat spectrum rectangle like the one displayed, then after resampling, you get *exactly* the spectrum you see on the graph, showing 88 alias images. Proof: I created 60 seconds of white noise sampled at 500 Hz, then resampled it to 44.1 kHz using linear interpolation. After the upsampling, it sounds like this: http://morpheus.spectralhead.com/wav/noise_resampled.wav Its spectrum looks like this: http://morpheus.spectralhead.com/img/noise_resampled.png Looks familiar? Oh, it's the *exact* same graph! (Minus some difference above 20 kHz, due to my soundcard's anti-alias filter.) It is an FFT graph of the upsampled white noise, and it shows *exactly* the aliasing. Good morning! This is what you'd get if you used those interpolation polynomials to convert a 250Hz sampled signal into a continuous time signal, not a discrete time signal of whatever sampling rate. Nope. You get the same graph if you sample that continuous time signal at a 44.1 kHz sampling rate (with some further aliasing from the sampling). Just as I've shown. Besides, I think the graph was created via numerical means using FFT, because it has noise at the low ampliutes (marked on the image). Therefore, it doesn't show a continuous time sinc^2 graph, because that wouldn't be noisy. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Since you constantly derail this topic with irrelevant talk, let me instead prove that 1) Olli Niemiatalo's graph *is* equivalent of the spectrum of upsampled white noise. 2) Olli Niemitalo's graph does *not* depict sinc(x)/sinc^2(x). First I'll prove 1). Using palette modification, I extracted the linear interpolation curve from Olli's figure: http://morpheus.spectralhead.com/img/other001b.gif Then I sampled white noise at 500 Hz, and resampled it to 44.1 kHz using linear interpolation. I got this spectrum: http://morpheus.spectralhead.com/img/resampled_noise_spectrum.gif To do a proper A/B comparison between the two spectra, I tried to align and match them as much as possible, and created an animated GIF file that blinks between the two graphs at a 500 ms rate: http://morpheus.spectralhead.com/img/olli_vs_resampled_noise.gif Although the alignment is not 100% exact, to my eyes, they look like totally equivalent graphs. This proves that upsampled white noise has the same spectrum as the graph shown on Olli's graph for linear interpolation. Second, I'll prove 2). Have you actually looked at Olli Niemitalo's graph closely? Here is proof that it is NOT a graph of sinc(x)/sinc^2(x): http://morpheus.spectralhead.com/img/other001-analysis.gif It is NOT sinc(x)/sinc^2(x), and you're blind as a bat if you do not see that. Since I proved both 1) and 2), it is totally irrelevant what you say, because none of what you could ever say would disprove this. Sinc(x) does not have a jagged/noisy look, therefore it is 100% certain it is not what you see on Olli's graph. Point proven, end of discussion. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 22/08/2015, Ethan Duni ethan.d...@gmail.com wrote: We've been over this repeatedly, including in the very post you are responding to. The fact that there are many ways to produce a graph of the interpolation spectrum is not in dispute, nor is it germaine to my point. Earlier you disputed that there's no upsampling involved. Apparently you change your mind quite often... It's seems like you are trying to avoid my point entirely, in favor of some imaginary dispute of your own invention, which you think you can win. I claimed something, and you disputed it. I proved that what I claimed, is true. Therefore, all your further arguments are invalid... (and are boring) I have no idea what you think you are proving by scrutinizing graph artifacts like that I am proving that what you see on the graph is not sinc(x) / sinc^2(x), but rather some noisy curve, like the spectrum of upsampled noise. Therefore, my original argument is correct. It's also in extremely poor taste to use retard as a term of abuse. Well, if you do not see that the graph pictured on Olli's figure is not sinc(x), then you're retarded. Meanwhile, it seems that you are suggesting that the spectrum of white noise linearly interpolated up to a high oversampling rate is not sinc^2. Naturally, there's going to be some jaggedness in the spectrum because of the noise. So, obviously, that is not sinc^2 then. Are you claiming that those wiggles in the graph represent aliasing of the spectrum from resampling at 44.1kHz? If so, that is unlikely. Nope, the wiggles in the graph are from the noise. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Upsampling means, that the sampling rate increases. So if you have a 250 Hz signal, and create a 22000 Hz signal from it, that is - by definition - upsampling. That's *exactly* what upsampling means... You insert new samples between the original ones, and interpolate between them (using whatever interpolation filter of your preference). And that is often used synonymously with 'oversampling', and that's what happens in an oversampled D/A converter. (Though 'oversampling' has a different meaning in A/D context.) -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 20/08/2015, Ethan Duni ethan.d...@gmail.com wrote: Wasn't the premise that memory was cheap, so we can store a big prototype FIR for high quality 512x oversampling? So why are we then worried about the table space for the fractional interpolator? For the record, wasn't it you who said memory is often a constraint? Quote from you: There are lots of dsp applications where memory is very much the main constraint. So apparently your premise is that memory can be expensive and a constraint, and now you ask why are we worried about using extra memory. At least make up your mind, whether you consider memory cheap or expensive... -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Let's analyze your suggestion of using a FIR filter at f = 0.5/512 = 0.0009765625 for an interpolation filter for 512x oversampling. Here's the frequency response of a FIR filter of length 1000: http://morpheus.spectralhead.com/img/fir512_1000.png Closeup of the frequency range between 0-0.01 (cutoff marked with black line): http://morpheus.spectralhead.com/img/fir512_1000_closeup.png Apparently that's a pretty crappy anti-alias filter, the transition band is very wide. So let's try a FIR filter of length 5000: http://morpheus.spectralhead.com/img/fir512_5000_closeup.png Better, but still quite a lot of aliasing above the cutoff freq. FIR filter of length 20,000: http://morpheus.spectralhead.com/img/fir512_2_closeup.png Now this starts to look like a proper-ish anti-alias filter. The problem is - its length is 20,000 samples, so assuming 32-bit float representation, the coefficients alone need about 80 KB of memory... meaning that there's a high chance that it won't even fit into the L1 cache, causing a lot of cache misses, so this filter will be extra slow, since your CPU will be constantly waiting for the coefficients to arrive from the L2 cache and/or RAM. Also consider how much CPU power you need to do convolution with a 20,000 sample long kernel at 512x oversampled rate... I bet you're not trying to do this in realtime, are you? So, that's not exactly the brightest way to do 512x oversampling, unless you prefer to waste a lot of resources and spend a week on upsampling. In that case, it is ideal. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
In the starting post, it was not specified that resampling was also used - the question was: Is it possible to use a filter to compensate for high frequency signal loss due to interpolation? For example linear or hermite interpolation. Without specifying that variable rate playback is involved, that could be understood in various ways - for example, at first I thought the interpolation was for the purpose of a (static or modulated) fractional delay line. A third possible situation is using linear/hermite interpolation as an upsampling filter in a 2^N oversampler. It was only specified 18 posts later, that the interpolation is used for variable pitch playback. These three situations all are different, and different formulas apply... And the combination oversampling and linear/hermite interpolation can also be meant in multiple ways. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Here's a graph of performance in mflops of varying length FFT transforms from the fftw.org benchmark page, for Intel Pentium 4: http://morpheus.spectralhead.com/img/fftw_benchmark_pentium4.png Afaik Pentium 4 has 16 KB of L1 data cache. If you check the graph, around 8-16k the performance starts to drop drastically. I believe the main reason for this is that the data doesn't fit into the L1 data cache any more, which is 16 KB. You'll see similar graphs for most other CPU types as well, there's a dropoff near the L1 cache size. So using more memory is only free(ish) until a certain point - if your data doesn't fit into the L1 cache any more, it will cause cache misses and give you a performance penalty, because the CPU needs to fetch the data from the L2 cache, which is several times slower. In the graph, you can see ~3-4x performance difference between transforms that fit into the L1 cache, and transforms that don't. For this reason, very large filters have a notable performance penalty. The coefficients for a FIR filter of length 20,000 will certainly not fit into a 16 KB L1 data cache. Here's the memory topology of AMD Bulldozer server microarchitecture: https://upload.wikimedia.org/wikipedia/commons/9/95/Hwloc.png Each core has a 16 KB L1 data cache. The further away you go from the CPU core, the slower the memory access gets. L2 cache is 2 MB, and there's a shared 8 MB L3 cache across cores. There's a 64 KB instruction cache per two cores. Similar cache architectures are common among computer processors (sometimes without L3 cache). There's a document that discusses this in depth: Ulrich Drepper, What Every Programmer Should Know About Memory http://morpheus.spectralhead.com/pdf/cpumemory.pdf This document gives the following memory access times for Intel Pentium M: Register: = 1 cycle L1 data cache: ~3 cycles L2 cache: ~14 cycles RAM: ~240 cycles So this means, on a Pentium M, accessing data in the L1 cache is ~3x slower, accessing data in the the L2 cache is ~14x slower, and accessing data in the RAM is ~240x slower than accessing data in a register. (Earlier I wrongly said RAM is about 10x slower, rather that's about the L2 cache speed.) So if the data doesn't fit into the L1 cache and needs to be fetched from the L2 cache, that's nearly 5x slower on the Pentium M. A notable part of the L2 cache penalty is caused by the physical limits of the universe - data travels in wires at the speed of light, which is about 1 foot (30 cm) per nanosecond. The larger the cache, the longer the wires, hence the longer the data access delay. For further details and detailed performance analysis, see the above paper. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Let me just add, that in case of having a non-oversampled linearly
interpolated fractional delay line with exactly 0.5 sample delay (most
high-frequency roll-off position), the frequency response formula is
not sinc^2, but rather, sin(2*PI*f)/(2*sin(PI*f)), as I discussed
earlier.
In that case, the results are slightly different:
# Tcl code -
set pi 3.141592653589793238
set freqs {1/4. 1/8. 1/16. 1/32. 1/64. 1/128. 1/256. 1/512. 1/1024.}
set amt 2
foreach freq $freqs {
set amp [expr sin(2*$pi*$freq)/(2*sin($pi*$freq))]
set db [expr 20.0 * log($amp)/log(10)]
puts [format %-8s ${amt}X:][format %f $db] dB
set amt [expr $amt*2]
}
# End of code --
Results:
2X: -3.01 dB
4X: -0.688 dB
8X: -0.169 dB
16X:-0.0419 dB
32X:-0.0105 dB
64X:-0.00262 dB
128X: -0.00065 dB
256X: -0.000164 dB
512X: -0.41 dB
___
music-dsp mailing list
music-dsp@music.columbia.edu
https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 20/08/2015, Ethan Duni ethan.d...@gmail.com wrote: But I'm on the fence about whether it's the tightest use of resources (for whatever constraints). Then try and measure it yourself - you don't believe my words anyways. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Comparison of the two formulas from previous post: (1) in blue, sinc^2 (2) in red: http://morpheus.spectralhead.com/img/sinc.png sin(pi*x*2) -(1) 2*sin(pi*x) (Formula from Steven W. Smith, absolute value taken on graph) sin(pi*x) --- ^2 = sinc^2(x)(2) pi*x (Formula from JOS, Nigel R.) (1) and (2) (blue and red curve) are quite different. Let's test how equation (1) compares against measured frequency response of a LTI filter with coeffs [0.5, 0.5]: http://morpheus.spectralhead.com/img/halfsample_delay_response.png The maximum error between formula (1) and the measured frequency response of the filter (a0=0.5, a1=0.5) is 3.3307e-16, or -310 dB, which about equals the limits of the floating point precision at 64 bits. The frequency response was measured using Octave's freqz() function, using 512 points. Conclusion: Steven W. Smith's formula (1) seems correct. Frequency response of the same filter in decibel scale: http://morpheus.spectralhead.com/img/halfsample_delay_response2.png (this graph is normalized to 0..1 rad, not 0..0.5) The pole-zero plot was shown earlier, having a zero at z=-1, meaning -Inf gain at Nyquist. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 19/08/2015, Ethan Duni ethan.d...@gmail.com wrote: But why would you constrain yourself to use first-order linear interpolation? Because it's computationally very cheap? The oversampler itself is going to be a much higher order linear interpolator. So it seems strange to pour resources into that Linear interpolation needs very little computation, compared to most other types of interpolation. So I do not consider the idea of using linear interpolation for higher stages of oversampling strange at all. The higher the oversampling, the more optimal it is to use linear in the higher stages. So heavy oversampling seems strange, unless there's some hard constraint forcing you to use a first-order interpolator. The hard constraint is CPU usage, which is higher in all other types of interpolators. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
3.2 Multistage 3.2.1 Can I interpolate in multiple stages? Yes, so long as the interpolation ratio, L, is not a prime number. For example, to interpolate by a factor of 15, you could interpolate by 3 then interpolate by 5. The more factors L has, the more choices you have. For example you could interpolate by 16 in: - one stage: 16 - two stages: 4 and 4 - three stages: 2, 2, and 4 - four stages: 2, 2, 2, and 2 3.2.2 Cool. But why bother with all that? Just as with decimation, the computational and memory requirements of interpolation filtering can often be reduced by using multiple stages. 3.2.3 OK, so how do I figure out the optimum number of stages, and the interpolation ratio at each stage? There isn't a simple answer to this one: the answer varies depending on many things. However, here are a couple of rules of thumb: - Using two or three stages is usually optimal or near-optimal. - Interpolate in order of the smallest to largest factors. For example, when interpolating by a factor of 60 in three stages, interpolate by 3, then by 4, then by 5. (Use the largest ratio on the highest rate.) http://dspguru.com/dsp/faqs/multirate/interpolation ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 19/08/2015, Ethan Duni ethan.d...@gmail.com wrote: Obviously it will depend on the details of the application, it just seems kind of unbalanced on its face to use heavy oversampling and then the lightest possible fractional interpolator. It should also be noted that the linear interpolation can be used for the upsampling itself as well, reducing the cost of your oversampling, not just as your fractional delay. A potential method to do fractional delay is to upsample by a large factor, then delay by an integer number of samples, and then downsample, without the use of an actual fractional delay. Say, if the fraction is 0.37, then you may upsamle by 512x, then delay the upsampled signal by round(512*0.37) = 189 samples, then downsample back. So you did a fractional delay without using actual fractional interpolation for the delay - you delayed by an integer number of samples. You'll also have a little error - your delay is 0.369140625 instead of the desired 0.37, since it's quantized to 512 steps, so the error is -0.000859375. I'm not saying this is ideal, I'm just saying this is one possible way of doing a fractional delay. This is discussed by JOS[1]: In discrete time processing, the operation Eq.(4.5) can be approximated arbitrarily closely by digital upsampling by a large integer factor M, delaying by L samples (an integer), then finally downsampling by M, as depicted in Fig.4.7 [96]. The integers L and M are chosen so that eta ~= L/M, where eta the desired fractional delay. [1] Julius O. Smith, Physical Audio Signal Processing https://ccrma.stanford.edu/~jos/pasp/Convolution_Interpretation.html Ref. [96] is: R. Crochiere, L. Rabiner, and R. Shively, ``A novel implementation of digital phase shifters,'' Bell System Technical Journal, vol. 65, pp. 1497-1502, Oct. 1975. Abstract: A novel technique is presented for implementing a variable digital phase shifter which is capable of realizing noninteger delays. The theory behind the technique is based on the idea of first interpolating the signal to a high sampling rate, then using an integer delay, and finally decimating the signal back to the original sampling rate. Efficient methods for performing these processes are discussed in this paper. In particular, it is shown that the digital phase shifter can be implemented by means of a simple convolution at the sampling rate of the original signal. In short, there are a zillion ways of implementing both oversampling and fractional delays, and they can be combined arbitrarily. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 20/08/2015, Ethan Duni ethan.d...@gmail.com wrote: Ugh, I suppose this is what I get for attempting to engage with Peter S again. Not sure what I was thinking... Well, you asked, why use linear interpolation at all? We told you the advantages - fast computation, no coefficient table needed, and (nearly) optimal for high oversampling ratios, and you were given some literature. If you don't believe it - well, not my problem... it's still true. #notmyloss ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 19/08/2015, Peter S peter.schoffhau...@gmail.com wrote: Another way to show that half-sample delay has -Inf gain at Nyquist: see the pole-zero plot of the equivalent LTI filter a0=0.5, a1=0.5. It will have a zero at z=-1. A zero on the unit circle means -Inf gain, and z=-1 means Nyquist frequency. Therefore, a half-sample delay has -Inf gain at Nyquist frequency. It looks like this: http://morpheus.spectralhead.com/img/halfsample_delay_zplane.png -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
Another way to show that half-sample delay has -Inf gain at Nyquist: see the pole-zero plot of the equivalent LTI filter a0=0.5, a1=0.5. It will have a zero at z=-1. A zero on the unit circle means -Inf gain, and z=-1 means Nyquist frequency. Therefore, a half-sample delay has -Inf gain at Nyquist frequency. It would be ill-advised to dismiss Nyquist frequency because it may alias to DC signal when sampling. The zero on the unit circle is at Nyquist (z=-1), not at DC (z=1). Frequency response graphs of linear interpolation, according to JOS: https://ccrma.stanford.edu/~jos/Interpolation/Frequency_Responses_Linear_Interpolation.html ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, Nigel Redmon earle...@earlevel.com wrote: well, if it's linear interpolation and your fractional delay slowly sweeps from 0 to 1/2 sample, i think you may very well hear a LPF start to kick in. something like -7.8 dB at Nyquist. no, that's not right. it's -inf dB at Nyquist. pretty serious LPF to just slide into. Right the first time, -7.8 dB at the Nyquist frequency, -inf at the sampling frequency. No? -Inf at Nyquist when you're halfway between two samples. Assume you have a Nyquist frequency square wave: 1, -1, 1, -1, 1, -1, 1, -1... After interpolating with fraction=0.5, it becomes a constant signal 0,0,0,0,0,0,0... (because (-1+1)/2 = 0) ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, Ethan Duni ethan.d...@gmail.com wrote: In order to reconstruct that sinusoid, you'll need a filter with an infinitely steep transition band. No, even an ideal reconstruction filter won't do it. You've got your +Nyquist component sitting right on top of your -Nyquist component. Hence the aliasing. The information has been lost in the sampling, there's no way to reconstruct without some additional side information. You cannot calculate 1/x when x=0, can you? Since that's division by zero. Yet you'll know when x tends to zero from right towards left, then 1/x will tend to +infinity. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, Ethan Duni ethan.d...@gmail.com wrote: But the example of the weird things that can happen when you try to sample a sine wave right at the nyquist rate and then process it is orthogonal to that point. That's not weird, and that's *exactly* what you have in the highest bin of an FFT. The signal 1, -1, 1, -1, 1, -1 ... is the highest frequency basis function of the DFT: http://www.dspguide.com/graphics/F_8_5.gif If you think that's weird, then I guess you think that the Fourier transformation is weird. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, Tom Duffy tdu...@tascam.com wrote: In order to reconstruct that sinusoid, you'll need a filter with an infinitely steep transition band. I can use an arbitrarily long sinc kernel to reconstruct / interpolate it. Therefore, for any desired precision, you can find an appropriate sinc kernel length. Where's the problem? I can also oversample the signal arbitrarily, using an arbitrarily long sinc kernel. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, Ethan Duni ethan.d...@gmail.com wrote: Assume you have a Nyquist frequency square wave: 1, -1, 1, -1, 1, -1, 1, -1... The sampling theorem requires that all frequencies be *below* the Nyquist frequency. Sampling signals at exactly the Nyquist frequency is an edge case that sort-of works in some limited special cases, but there is no expectation that digital processing of such a signal is going to work properly in general. Not necessarily, at least in theory. In practice, an anti-alias filter will filter out a signal exactly at Nyquist freq, both when sampling it (A/D conversion), and both when reconstructing it (D/A conversion). But that doesn't mean that a half-sample delay doesn't have -Inf dB gain at Nyquist frequency. It's another thing that the anti-alias filter of a converter will typically filter it out anyways when reconstructing - but we weren't talking about reconstruction, so that is irrelevant here. A Nyquist frequency signal (1, -1, 1, -1, ...) is a perfectly valid bandlimited signal. But even given that, the interpolator outputting the zero signal in that case is exactly correct. That's what you would have gotten if you'd sampled the same sine wave (*not* square wave - that would imply frequencies above Nyquist) with a half-sample offset from the 1, -1, 1, -1, ... case. More precisely: a bandlimited Nyquist frequency square wave *equals* a Nyquist frequency sine wave. Or any other harmonic waveform for that matter (triangle, saw, etc.) In all cases, only the fundamental partial is there (1, -1, 1, -1, ... = Nyquist frequency sine), all the other partials are filtered out from the bandlimiting. So the signal 1, -1, 1, -1, *is* a Nyquist frequency bandlimited square wave, and also a sine-wave as well. They're identical. It *is* a bandlimited square wave - that's what you get when you take a Nyquist frequency square wave, and bandlimit it by removing all partials above Nyquist freq (say, via DFT). You may call it a square, a sine, saw, doesn't matter - when bandlimited, they're identical. The incorrect behavior arises when you try to go in the other direction (i.e., apply a second half-sample delay), and you still get only DC. What would be incorrect about it? I'm not sure what is your assumption. Of course if you apply any kind of filtering to a zero DC signal, you'll still have a zero DC signal. -Inf + -Inf = -Inf... Not sure what you're trying to achieve by applying a second half-sample delay... That also has -Inf dB gain at Nyquist, so you'll still have a zero DC signal after that. Since a half-sample delay has -Inf gain at Nyquist, you cannot undo it by applying another half-sample delay... But, again, that doesn't really say anything about interpolation.It just says that you sampled the signal improperly in the first place, and so digital processing can't be relied upon to work appropriately. That's false. 1, -1, 1, -1, 1, -1 ... is a proper bandlimited signal, and contains no aliasing. That's the maximal allowed frequency without any aliasing. It is a bandlimited Nyquist frequency square wave (which is equivalent to a Nyquist frequency sine wave). From that, you can reconstruct a perfect alias-free sinusoid of frequency SR/2. What's causing you to be unable to reconstruct the waveform? -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, Ethan Duni ethan.d...@gmail.com wrote: You cannot calculate 1/x when x=0, can you? Since that's division by zero. Yet you'll know when x tends to zero from right towards left, then 1/x will tend to +infinity. Not sure what that is supposed to have to do with the present subject. You cannot calculate 1/x when x=0, because that's division by zero, yet you can calculate the limit of 1/x as x tends towards zero. Meaning that you can approach zero arbitrarily, and 1/x will approach +infinity arbitrarily. Similarly, even if frequency f=0.5 may be considered ill-specified (because it's critical frequency), you can still approach it to arbitrary precision, and the gain will approach -infinity. So f=0.4 f=0.49 f=0.499 f=0.4999 f=0.49 f=0.499 etc. The more you approach f=0.5, the more the gain will approach -infinity. Even if f=0.5 is a critical frequency. f=0.4 isn't, and it's quite close to f=0.5. That's what I mean. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, robert bristow-johnson r...@audioimagination.com wrote: *my* point is that as the delay slowly slides from a integer number of samples [...] to the integer + 1/2 sample (with gain above), this linear but time-variant system is going to sound like there is a LPF getting segued in. Exactly. As the fractional delay varies between 0..1, it will sound like a fluttering LP filter that closes and opens as the delay varies, having the most 'muffled' (LPF'ed) sound when fraction = 1/2. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, robert bristow-johnson r...@audioimagination.com wrote: On 8/18/15 4:28 PM, Peter S wrote: 1, -1, 1, -1, 1, -1 ... is a proper bandlimited signal, and contains no aliasing. That's the maximal allowed frequency without any aliasing. well Peter, here again is where you overreach. assuming, without loss of generality that the sampling period is 1, the continuous-time signals x(t) = 1/cos(theta) * cos(pi*t + theta) are all aliases for the signal described above (and incorrectly as contain[ing] no aliasing). Well, strictly speaking, that is true. But I assumed the signal to be bandlimited to 0..SR/2. In that case, you can perfectly reconstruct it, as you have no other alias between 0..SR/2. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 18/08/2015, Ethan Duni ethan.d...@gmail.com wrote: That class of signals is band limited to SR/2. The aliasing is in the amplitude/phase offset, not the frequency. Okay, I get what you mean. But that doesn't change the frequency response of a half-sample delay, or doesn't mean that a half-sample delay doesn't have a specific gain at Nyquist. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Compensate for interpolation high frequency signal loss
On 17/08/2015, STEFFAN DIEDRICHSEN sdiedrich...@me.com wrote: I could write a few lines over the topic as well, since I made such a compensation filter about 17 years ago. So, there are people, that do care about that topic, but there are only some, that do find time to write up something. I also made a compensation filter for linear interpolation. Definitely doable. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 13/08/2015, Peter S peter.schoffhau...@gmail.com wrote: Bonus experiment: try to see if you can hear the difference between sine_fadeout16_noise.wav and sine_fadeout8_noise.wav in a blind ABX test. If not, then having extra bits of noise make zero sense. I did a blind ABX test between them. Despite both signals have a -36 dB noise floor, I could discern the difference, the quantized signal sounding slightly more noisy. I examined the noise (quantization error) in detail. Here is how it sounds like: http://morpheus.spectralhead.com/wav/quantization_noise.wav This is 1-bit of white noise, which is the quantization error. Its spectrum looks entirely flat: http://morpheus.spectralhead.com/img/quantization_noise_spectrum.png The bump at 0 Hz is caused by a DC offset, since it was truncated and not rounded, so the error is not symmetrical between -0.5 .. 0.5 bit, but is rather between -1 .. 0 bit. The waveform - when normalized to 0 dB, looks entirely like white noise: http://morpheus.spectralhead.com/img/quantization_noise_waveform.png Its spectrogram also looks like white noise - there are absolutely no harmonics or signal-related artifacts, it looks like fully uncorrelated, pure white noise: http://morpheus.spectralhead.com/img/quantization_noise_spectrogram.png The amplitude of the error is 1 LSB, which - in the case of 8-bit quantization - means -42 dB amplitude. Since 42 dB is a small SNR, I could discern the difference in blind ABX tests, though the difference is quite minor (do an ABX test yourself to hear yourself). If the quantization were to 24 bits instead of 8 bits, the amplitude of the quantization error in a similar situation would be -140 dB (= 1 LSB at 24 bits precision). That is absolutely beyond both every soundcards' dynamic range, and the human hearing's dynamic range. Since discerning the difference in a signal quantized to 8-bits was already not trivial and needed careful listening, I am absolutely 100% convinced that the difference in a 24-bit signal would be absolutely imperceptible in a blind ABX listening test under ideal listening conditions with the most high-end gear available. Therefore, based on two different tests, I still hold my original opinion that having extra 8 bits of noise in a properly dithered converter beyond the 24th bit is absolutely useless, and makes absolutely zero practical difference. There is not even a single soundcard on the planet that could even properly reproduce signals with -140 dB amplitude (most high-end ones have an SNR of only around 115-127 dB), therefore this couldn't even be properly tested. And even if there were an equipment capable of such precision, it is already beyond the dynamic range of the human hearing, which is around 120 decibels (excluding signal levels beyond the pain threshold), beyond which, the signal is perceived as noise-free. If it's already noise-free, there is no way of getting any better than that... -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 13/08/2015, Risto Holopainen ebel...@ristoid.net wrote: Spectral entropy is commonly used as a feature extractor in the MIR community. Pure sinusoids have the lowest spectral entropy whereas white noise has maximal entropy, so it's useful for distinguishing pitched and noisy signals. Thanks for the info. According to [1], steps for calculating spectral entropy are as follows: 1. Calculate the power spectrum of the signal using FFT command. 2. Calculate the Power Spectral Density using the power spectrum or using any other technique. 3. Normalize the Power Spectral Density between [0-1], so that it can be treated as a probability density function p_i. 4. Calculate the Entropy H(s) = SUM p_i * log2(p_i) Sounds quite trivial. [1] Spectral Entropy Calculation in MATLAB https://dsp.stackexchange.com/questions/10137/spectral-entropy-calculation-in-matlab (Includes Python source code for spectral entropy calculation.) Proven again, that I am not the first person in the history of mankind to try to calculate the entropy of an arbitrary waveform. On 13/08/2015, Laszlo Toth to...@inf.u-szeged.hu wrote: It's great when you quote, because it suddenly starts to make sense. Shannon is not talking about waveforms, but waveforms TOGETHER WITH A PROBABILITY MEASURE. Ah, here is the probability distribution that many people were missing! Apparently, spectral entropy just assumes the Fourier spectrum to be the probability distribution. Since every waveform has a spectrum, therefore every waveform has a specified 'spectral entropy'. Apparently, Norbert Wiener also wanted to estimate the entropy of arbitrary waveforms: Wiener entropy is a measure of the width and uniformity of the power spectrum. Noise is typically broadband with sound energy smeared rather smoothly within the noise range, whereas animal sounds, even when multi-harmonic, are less uniform in their frequency structure. Wiener entropy is a pure number, that is, it does not have units. On a scale of 0-1, white noise has an entropy value of 1 and complete order, and a pure tone has an entropy value of 0.[1] This measure is used in the MPEG-7 standard (and is also called 'spectral flatness' and 'tonality coefficient')[2]. [1] http://soundanalysispro.com/manual-1/chapter-4-the-song-features-of-sap2/wiener-entropy [2] https://en.wikipedia.org/wiki/Wiener_entropy -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
This paper discusses Shannon entropy, Wiener entropy, Rényi entropy, spectral entropy, and some related measures for the purpose of speech processing: ON THE GENERALIZATION OF SHANNON ENTROPY FOR SPEECH RECOGNITION http://architexte.ircam.fr/textes/Obin12e/index.pdf Quote: 2. SPECTRAL ENTROPY With an appropriate normalization, the power spectrum of an audio signal can be interpreted as a probability density. According to this interpretation, some techniques belonging to the domains of probability and information theory can be applied to sound representation and recognition: in particular, the concept of entropy can be extended to provide a concentration measure of a time-frequency density - which can be interpreted as a measure of the degree of voicing (alternatively, noisiness) of an audio signal. The representation adopted in this study (see [9] for the original formulation) is based on the use of Rényi entropies, as a generalization of the Shannon entropy [10]. In this section, the Rényi entropy is introduced with regard to information theory, and some relevant properties for the representation of audio signals are presented. In particular, the Rényi entropy presents the advantage over the Shannon entropy of focusing on the noise or the harmonic content. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 12/08/2015, Cesare Ferrari ces...@loftsoft.co.uk wrote: As far as I can see, you are arguing about the definition of these terms, and then answering questions about your new definitions. I would imagine that people who you disagree with probably are using a different definition, and hence are reaching a different conclusion? Exactly, that's what I pointed out to, and this is why I repeated my definitions. Since I cannot process infinite amount of data (obviously), I may possibly reach a different conclusion, as I explained already in detail. I just hate having to repeat what I already told. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 12/08/2015, Theo Verelst theo...@theover.org wrote: If we presume perfect messages of a certain bit size, e.g. 2 bits, a perfect encoded message will use them perfectly such that if we send N messages for N~large, we have perfect white noise, and dissociated messages when all messages occur just as often. So we can use the 2 bit words optimal to for instance transfer a large file (if desired even repeatedly for different large files), when the correlation between the messages is zero? If you have two transmissions (large files) with two sets of probability distributions P and Q, then if you use code optimized for probability distribution Q to encode messages with probability distribution P, then you'll need extra bits, because your encoding is not optimal for probabiliy distribution P. This difference is called the Kullback–Leibler divergence, or relative entropy: D(p || q) = SUM[i] p_i * log2(p_i / q_i) https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence Now, where is the assumption I mean, and what's wrong with all those perverted derivations and phrasings here I tried to correct ? What's wrong with all those perverted derivations and phrasings? ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 13/08/2015, Laszlo Toth to...@inf.u-szeged.hu wrote: It's great when you quote, because it suddenly starts to make sense. Shannon is not talking about waveforms, but waveforms TOGETHER WITH A PROBABILITY MEASURE. Ah, here is the probability distribution that many people were missing! Exactly. I talked about histograms and probability distributions earlier quite much, so if someone missed that, it's not my fault. As I see it, it doesn't matter how you break down a signal - whatever method you use to break it down into constituents, you can estimate probabilities, create a histogram, from that you can create a probability distribution, and from that you can estimate Shannon-entropy. Depending on how you break down a signal into constituents, you may get better or worse estimates. So far I haven't tried Fourier-domain decomposition to estimate entropy, but sounds like a fun project. (Sidenote: algorithmic entropy is not a function of a probability distribution, though it's different from Shannon entropy.) ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 13/08/2015, robert bristow-johnson r...@audioimagination.com wrote: Are you *sure* truncating adds extra quantization noise? [sigma-delta modulator][decimation filter][quantizer]--- adds extra quantization noise to: [sigma-delta modulator][decimation filter] Okay, this is the VERY LAST time I'll show this demonstration. If you want to understand this, then pay attention VERY closely: Here is a fading out 440 Hz sine wave at 16 bits precision: http://morpheus.spectralhead.com/wav/sine_fadeout16.wav Here is its spectrogram - a single sinusoid at 440 Hz: http://morpheus.spectralhead.com/img/sine_fadeout16_spectrogram.png If you simply truncate it to 8 bits, here is what you get: http://morpheus.spectralhead.com/wav/sine_fadeout8.wav As expected, the spectrogram shows harmonic distortion: http://morpheus.spectralhead.com/img/sine_fadeout8_spectrogram.png This is because the sinewave gets distorted, and starts looking like a square wave, having harmonic spectral artifact, as expected. ...HOWEVER *IF* the signal is already noisy, say, from the thermal noise of the converter, *THEN* this will NOT happen!!! To demonstrate this, here is a -36 dB white noise floor: http://morpheus.spectralhead.com/wav/noise_36db.wav If we mix this with the original sine wave, we got this: http://morpheus.spectralhead.com/wav/sine_fadeout16_noise.wav It is a fading out 440 Hz sine wave, buried in a -36 dB white noise floor. If we quantize this to 8 bit, this is what we get: http://morpheus.spectralhead.com/wav/sine_fadeout8_noise.wav Do you hear harmonic distortion in this signal? ABSOLUTELY NOT!!! To prove, here is the spectrogram of this 8-bit noisy signal: http://morpheus.spectralhead.com/img/sine_fadeout8_noise_spectro.png Do you see harmonic distortion on the spectrogram? ABSOLUTELY NOT!!! A/B spectrogram comparison between quantization with no noise floor, and quantization with -36 dB white noise floor: http://morpheus.spectralhead.com/img/sine_fadeout_8_vs_16.png The original, non-noisy sinusoid, quantized to 8 bit, then converted back to 16 bit, subtracted from the original signal, normalized to -16 dB (= the harmonic noise from the quantization, amplified, without the original signal): http://morpheus.spectralhead.com/wav/sine_fadeout_quant_noise.wav The noisy sinusoid, quantized to 8 bit, then converted back to 16 bit, subtracted from the original signal, normalized to -16 dB (= the error from the quantization, amplified, without the original signal): http://morpheus.spectralhead.com/wav/sine_fadeout_noise_quant.wav Conclusion: when there is a loud enough noise floor present in the signal, then there's no harmonic distortion from the quantization!!! Nope, nil, nada, zilch, zero, null!! The white noise floor eliminated it completely! That's the whole purpose of dithering... The error in the case of a quantized noisy sinusoid is almost like white noise. It would be good if - after multiple demonstrations, and giving you 30 external references - you could finally understand that noise from either dithering or a thermal noise floor eliminates harmonic distortion from the quantization step. Bonus experiment: try to see if you can hear the difference between sine_fadeout16_noise.wav and sine_fadeout8_noise.wav in a blind ABX test. If not, then having extra bits of noise make zero sense. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [ot] about entropy encoding
On 12/08/2015, Sampo Syreeni de...@iki.fi wrote: Of course you need to: both of the words could have probability one, so that their occurrence pairwise would also have probability one for certain, and so the joint surprisal could be a flat zero. And even in this case, the entropy (surprisal) may be nonzero. After all, how do you know how to group the symbols into pairs? With two symbols, there are 2 possible ways of organizing symbols into groups. Is the pair ab, or is it ba ? Is the signal abababab or is it babababa ? These are equivalent to two 1-bit square waves with a 180 deg phase difference (= inverted phase), hence they're two different signals. Without your transmitter sending the phase information, how does your receiver know, if the signal to be reconstructed is abababab... or babababa... ? In the case the two signals are equally probable, that's precisely 1 shannon (= 1 bit) of entropy there, even if two symbols have a probability of 1 when grouped as pairs - since there are 2 ways of grouping two symbols into pairs. Without sending a nonzero amount of information, your receiver won't know how to reconstruct the signal with 100% certainity. (And that is just the probabilistic Shannon entropy, we're not even speaking about codebook length or algorithmic entropy here.) Why do you think all data compression competitions include the size of the decompressor program? Because you could just say: Look Mom! No entropy! I just hard-coded the data into the decoder program! If you do that, all that is going to happen, is you're going to get disqualified from the competition, for cheating. Having zero Shannon entropy means doing data compression by cheating - hiding the data somewhere, and pretending it's not there, which gets you disqualified from any data compression competition. One has to be a special kind of retarded to actually believe that you can reconstruct something from nothing. (The Burrows-Wheeler Transform will not help you, nor a document you wrote many years ago.) The probabilistic Shannon entropy is a highly simplified view of data compression. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [ot] about entropy encoding
If you want to prove that a fully deterministic signal has zero entropy, here is a challenge for you: I have a fully deterministic signal here (fully periodic, zero surprisal over the signal). Since you think it has zero entropy, I just sent you nothing (precisely zero bits). Now try to reconstruct my fully deterministic signal from the nothing that you received from me! Good luck! Until you succeed recovering my signal from the nothing you received, do not tell me that a fully deterministic signal has zero entropy. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
So, since Theo Verelst still doesn't understand, let's discuss it again the Nth time: Shannon defines entropy as the minimum average length of a message in bits when you want to transmit messages of a probabilistic ensemble. He dismisses the codebook size and the size of the decoder/receiver program, assuming them to be zero. Since without the codebook (either transmitted or pre-agreed), the messages cannot be decoded, and your transmission is entirely useless. Therefore, I consider the codebook the integral part of the transmission, hence its size is always nonzero (either transmitted or pre-agreed), therefore the actual entropy is always nonzero, even in the case of a single message with 100% probability. (I gave some formulas earlier about this.) The algorithmic information theory folks (Kolmogorov, Solomonoff, Chaitin, etc.) define entropy in relation to some sort of machine, typically an abstract Turing machine. Once an eval function is defined for that machine, the precise algorithmic entropy can be defined as the minimum length binary program that outputs that string of bits. This is also always nonzero, and is not a function of probability. (See Kolmogorov complexity / algorithmic information theory.) Entropy _rate_ is the measure of average information per symbol as the number of symbols asymptotically reaches infinity. There's a problem with this - when was the last time you made an infinite amount of observations? Since your lifetime is finite, you can never process an infinite amount of data, so in practice, dividing the total entropy by the number of symbols will never actually reach zero. Since the actual entropy of a signal is always nonzero, the only way the measured entropy _rate_ of a signal can be zero is by making an infinite amount of observations, since finite/infinite = zero. So that can happen only in theory, but never in practice. And since all this was already discussed, what I still don't understand: Why didn't you guys paid attention the first time? Why do I need to repeat everything 3-4-5 times? -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [ot] about entropy encoding
On 12/08/2015, Sampo Syreeni de...@iki.fi wrote: And mind you, I've been doing this since something like age 17-19a in data compression theory. I even feature in the oldskool comp.compression FAQ wrt Burrows-Wheeler Transformation, as the first guy who explained its compression half to the wider net audience. P.S. Burrows-Wheeler Transform has pretty much nothing to do with entropy estimation. So that's both irrelevant and off-topic. Just as your age, which is also entirely irrelevant. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [ot] about entropy encoding
On 12/08/2015, Sampo Syreeni de...@iki.fi wrote: Suppose I gave you a signal derived from a cryptographic pseudorandom number generator with a period in the order of 2^256? Its output would by design pass even the most stringent statistical tests you could throw at it. What would your entropy estimator say about it given the output signal? I cannot algorithmically distinguish a deterministic random number generator from a true random number generator, and I never in my life claimed I can. So your point is? Because its entropy rate would again formally be precisely zero. And entropy rate != entropy. You guys keep mixing up these two very different notions, for some weird reason. If you want to transmit the parameters for your CSPRNG, then if you have multiple sets of paramateres you want to transmit, then the *entropy* (length of your message) will be *nonzero*, period. Why confuse entropy with entropy rate? They're very different. (And algorithmic entropy is always nonzero, even when the Shannon entropy is zero. To understand that, you need to understand algorithmic entropy.) Oh I do know Kolmogorov's theory. Now can you prove the initial packet of code is always within a logarithmically bounded constant of any other? I can't, because it takes extra conditions to prove that. Which precise extra assumptions are needed there? Why would I need to prove that? I only said, the program that generates your (whatever) signal is always nonzero length, unless your signal is a constant zero signal. You do not need to specify the probability space 'fully' to know that when you have a probabilistic ensemble of at least two different 'words' (as you call them) from the symbol space with probability different from one, then the Shannon entropy of a message (average information content per message in bits) will be nonzero, without exception. Of course you need to: both of the words could have probability one, so that their occurrence pairwise would also have probability one for certain, and so the joint surprisal could be a flat zero. Your argument is pretty stupid. First, because I said at least two different 'words' from the symbol space with probability different from one, notice the meaning of the words different from one. So why do you deliberately dismiss what I wrote? (= strawman argument) Second, if you have two different events (words, messages), let's call them p and q, then the total probability equals 1. So p+q = 1. Therefore, q = 1 - p, therefore, if p = 1, then q = 0. So both of the words could have probability one is impossible, because if p=1 and q=1, then p+q = 2, which is a contradiction, since p+q = 1. Therefore, your argument is invalid. I just did, to yet another edge case. Which is an impossible case, that can never happen. And mind you, I've been doing this since something like age 17-19a in data compression theory. I even feature in the oldskool comp.compression FAQ wrt Burrows-Wheeler Transformation, as the first guy who explained its compression half to the wider net audience. Remind me, when were you born again? ;) You're either trolling, or a special kind of stupid. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [ot] about entropy encoding
On 12/08/2015, Sampo Syreeni de...@iki.fi wrote: Of course you need to: both of the words could have probability one, so that their occurrence pairwise would also have probability one for certain, and so the joint surprisal could be a flat zero. Let's discuss this further. By this, probably you mean that you have a sequence ababababab, where p(a) = 0.5, p(b) = 0.5, and taken as pairs, p(ab) = 1. In this case, by observing N symbols, how do you know that the signal is fully deterministic? Unless you do infinite amount of observations, you can't know with 100% certainity. If I send you a signal that has 100,000 ab pairs, followed by an uniform distribution random sequence of a and bs, then the signal is not deterministic. If you only analyze the first 100,000 pairs, you might think that it is deterministic, but it isn't. How do you know? You can't know. Secondly, my later entropy estimation algorithms can deal with this kind of situation very well, as they do not analyze only single symbols (like in the simple case of Shannon entropy), but also symbol pairs, triplets, and various other patterns. This is exactly the reason, why it gives a low entropy score for periodic patterns, like abababababab, and other square waves (aabbaabb..., aaabbbaaabbb... etc.) or periodic patterns as well. Just as I've shown here already: http://morpheus.spectralhead.com/entropy/estimate3/ Note the graph titled Square wave entropy estimate as function of frequency. Note on the bottom of the graph, that for f = 0.5 (which equals your sequence abababababab, which equals a square wave of normalized frequency 0.5), it gives an entropy estimate of near zero, as you can see on the graph. So *yes*, my algorithm deals with this particular corner case, and lots of other corner cases as well. Note that this analysis was done on a *short* sequence (either 1000 or 10,000 samples), yet it still realizes that the expected entropy is near zero, without having to do a large amount of observations. So yes, my algorithm is smart enough to figure out über-simple patterns like ababababab..., and more complex ones as well. (And it could use this information to do data compression as well.) Since this has been discussed a few times already, I'm not sure if you're just trolling, or a special kind of stupid. Why don't you guys pay attention? -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] List settings after the switch
On 10/08/2015, Vadim Zavalishin vadim.zavalis...@native-instruments.de wrote: - the reply-to field in the list mails is configured to the sender. So hitting reply no longer sends the answer to the list. I experience the same recently, and a few times I accidentally sent the reply to the sender instead of the list. -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 10/08/2015, Sampo Syreeni de...@iki.fi wrote: Correct. But notice that the main dither talked about in the papers is always done within the quantization loop, so that its power spectrum becomes dependent on the loop's high order highpass response. Its spectrum can and will be driven off the utility audio band, and so effectively only used to linearize the loop itself. Never fully - some percentage of the noise still lies in the audio band, even with a lot of oversampling. Sometimes an extra TPDF dither is added in front of the loop as a traditional dither, but as somebody already mentioned, in high end converters that comes for free from thermal noise, so that no additional noise is necessary. Yes, I mentioned it. Yet that doesn't mean there's no noise floor, and then the noise floor's spectrum is not shaped. 4) dithering eliminates signal-dependent quantization artifacts, hence the noise floor becomes entirely independent of the input No it does not. And now you are the one who doesn't get dither, I think. ;) I showed how to encode a -60 dB sinewave in a 8-bit signal (with a theoretical dynamic range of 48 dB) without any harmonic distortion whatsoever, using dithering. So yes, it definitely does eliminate harmonic distortion. If you think it doesn't, you need more to prove that, and define your model of error. So it does, and you just forgot to make the distinction between dither introduced within the coding loop and in its input. In both case, you have a noise floor. Now, what is *really* interesting and germane to your point about noise floors and converter depths is that you didn't make a proper distinction between input thermal noise floor and the noise induced by the delta-sigma coding loop. This is crucial, because only one of them is really noise: the input floor. False. Dithering is literally 'added noise', and noise from the dithering is also a 'noise floor'. Even with high amount of oversampling and noise shaping, there's a definite noise floor left in the audio band. If you're in doubt, try to record the input of any sound card without any input (or with a closed loopback from its output), and see what you get. You'll get a constant noise floor in virtually all 16-bit sound cards. It *is* noise, from the dithering. Afaik there's no modern sound card where the lowest bits aren't going to be noise, they all have a noise floor (from thermal or dithering noise). What the (undithered) loop does is completely different: it introduces all kinds of wild modulation phenomena which are only *modelled* as noise of certain power. If you do not suppress those over the utility band, you can get the funniest whines, or even free running limit cycle oscillations with enough power and harmonic content to be heard from -30dB below the noise floor. Especially you can get harmonics production from stuff similarly far down below the input noise floor. Now, this is also false, and I showed this using a sine wave buried in rectangular PDF white noise, which was free of any harmonic distortion after quantization. If your signal already has enough noise (from thermal noise or from whatever added noise), then you won't have quantization artifacts and limit cycle oscillations you describe. If you're in doubt - try it yourself, and see what you get. I've shown it already, I don't want to repeat myself all over. (This is assuming you have enough thermal noise present in the input.) So, there are several reasons to engineer for below input noise floor purity. Don't neglect them. I think no one argued that. Do notice that we're in the business of producing audio systems and software. Not all of them are meant for pure human consumption. For example, in audio forensics work, you'd like your signal chain to be pure enough to pick up line hum at -60dB or even -80dB below the already low noise floor, given enough integration time. I doubt you'll ever pick up any signal at -80 dB below the noise floor. That's 10^(-80/20) = 0.0001 times the amplitude of the noise, meaning the noise is 10,000x louder than the signal you're trying to pick up. Good luck trying to do that. you shouldn't be too swift at dismissing beyond-human audio processing either. I merely said, dismissing supersonic noise by lowpass filtering also changes the shape of the waveform. Beyond that, Peter, I actually agree with you. 24 bits is beyond what we can dissolve. In fact I'd go with the Acoustic Renaissance for Audio paper which argues that a properly executed, noise shaped, 55kHz @~ 18bpS ought to be enough. https://www.meridian-audio.com/meridian-uploads/ara/araconta.htm#precision Since the human hearing's dynamic range is practically around 120 dB (or 20 bits), there's not much point in ever going beyond 24 bits (unless you're trying to record some extra quiet signal). According to a research at Ampex labs in 1981, a dithered signal with a dynamic range of 118 dB is enough for a subjectively
Re: [music-dsp] Non-linearity or filtering
On 10/08/2015, robert bristow-johnson r...@audioimagination.com wrote: the thing that i *think* Peter is missing in this is the same as some of the early manufacturers when they truncated the 30-bit words (or whatever they had in the decimation filters) to 18 meaningful bits. that simply adds one more source of quantization noise. and, especially when our DSPs were 24 bits or more, we could make use of those bits. i'd much rather have a 24-bit word, even if the dynamic range divided by 6.02 is only 18 or 20. Are you *sure* truncating adds extra quantization noise? When the lowest bits are already noise, I highly doubt that truncation would add harmonic distortion or limit cycle oscillations. I tested the difference, and if the noise floor is high enough, in blind ABX tests, I personally hear no difference between the non-truncated and the truncated signal (whatever bit depth). Have you measured the difference? Do you actually hear the difference? Can your ear distinguish between the two in blind listening tests? Mine doesn't, and my analysis shows no difference in harmonic distortion, so I believe that those lowest, discarded bits of noise practically add nothing to the signal, and they can be discarded without any loss of quality or any introduced artifacts. That's my belief, and I base this on analysis and blind ABX listening tests between the two. there is *still* a part of your signal in those bottom 6 bits. why ditch them? Because adding further bits of noise add no perceptional difference. In blind ABX tests, I cannot distinguish the signal if I discard the lowest bits of noise, and it doesn't introduce any harmonic distortion either. (And since the human hearing's dynamic range is around 20 bits (120 dB), the bits below the 20th bit wouldn't add much useful anyways, so I can understand the choice of a manufacturer to truncate to ~18-20 bits.) -P ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
What I don't understand, is why don't you guys pay attention. I showed all these already using various demonstrations and tests, so having to repeat the same over and over again feels quite pointless... ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] [ot] about entropy encoding
On 09/08/2015, Sampo Syreeni de...@iki.fi wrote: In limine you could have *any* signal *at all*, but always sent with 100% certainty: when you saw it, its surprisal would be precisely zero. That's the trivial case of the underlying probability space for the entire signal being composed of a single message with probability 1, *whatever* that one signal might be. Exactly. And in *all* other cases except that this very single corner case, surprisal is nonzero, hence entropy is nonzero (whatever the probabilities). Why do you guys keep repeating this single corner case of 1 signal with 100% probability having a Shannonian probabilistic entropy of zero? Yes, as discussed ad nauseum, in that case, (Shannon) entropy is zero, as it is a special corner case. Has been discussed several times. And in virtually *all* other cases, surprisal is nonzero, hence Shannon entropy is *nonzero*, irregardless of the actual probabilities of the ensemble. It would be entirely pointless to repeat it again, I showed this in various ways. (And algorithmic entropy is always nonzero, even when the Shannon entropy is zero. To understand that, you need to understand algorithmic entropy.) Peter's problem then seems to be that he doesn't specify that underlying probability space nor state his assumptions fully. You do not need to specify the probability space 'fully' to know that when you have a probabilistic ensemble of at least two different 'words' (as you call them) from the symbol space with probability different from one, then the Shannon entropy of a message (average information content per message in bits) will be nonzero, without exception. It comes straight from Shannon's entropy formula. Since I also showed this in detail with formulas, there's no point in repeating it like the 3rd time. If you don't understand it, read Shannon's paper. If you cannot apply logic to a simple mathematical formula, that is not my problem. But between the line he assumes that signals coming from an entirely different kind of, far more correlated-between-his-chosen-partition were an equal option. I never assumed that. Your reading sounds flawed. Maybe first re-read what I wrote. Best regards, Peter S ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 27/07/2015, Steven Cook stevenpaulc...@tiscali.co.uk wrote: I *think* I've got it now, but just in case, could you explain the whole thing again from the start because I wasn't listening? Depends, how much will you pay? ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
Said in other words, by adding extra bits of noise, you gain pretty much nothing. You could create a 48-bit, 64-bit, 128-bit etc. converter, and while technically you have a 48/64/128-bit converter, in reality, all you do is just add extra bits of useless noise. Unless you use cryogenics and superconductive materials, the thermal noise will be the same. Hence, in your 128-bit converter, the lowest 106-108 bits will be noise, most of which will be entirely useless. For any bit depth, only the highest 20-22 bits are usable, below that, you go into the noise floor, and your signal soon gets lost in noise. And no matter what bit depth converter you make, it won't produce a signal that couldn't be produced virtually identically by a 24-bit converter, since -127 dB noise floor means that using 24 bits, the lowest 3 bits are already noise, which also dithers the signal, extending the dynamic range to around 150-160 dB. By adding further, extra bits of noise, you basically gain nothing - the quality or SNR or dynamic range won't increase, and a dithered signal already has a (theoretically) infinite dynamic range (until the noise masks the signal). Hence, a 24 bit converter is practically enough, there's no further gain to go to higher bit depths (at least until such new materials are invented that would allow a SNR above 140 dB, which I don't expect to happen tomorrow). ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 27/07/2015, Peter S peter.schoffhau...@gmail.com wrote: http://morpheus.spectralhead.com/wav/noise120db_vs_sine160db_norm.wav Sorry, this link gave a 404 error. Fixed now. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 27/07/2015, Peter S peter.schoffhau...@gmail.com wrote: Mixture of -24 dB noise floor and -48 dB sine wave, at 16 bits precision: http://morpheus.spectralhead.com/wav/noise24db_vs_sine48db.wav Same thing converted to 8 bit: http://morpheus.spectralhead.com/wav/noise24db_vs_sine48db_8bit.wav To me, they sound exactly identical, I cannot hear any difference whatsoever. Their spectrum also looks exactly identical Here is an interesting observation. If you compare their waveforms, the 16 bit one looks smooth, and the 8 bit one looks quantized and having the typical stairstep pattern: http://morpheus.spectralhead.com/img/dithered_sine_16_vs_8bit.png Despite this, they both sound identical, and their spectrum also looks identical. Despite having a quantized, stairstep waveform, the quantized signal represents the smooth, curved sinewave without any distortion at all - there are no harmonics visible on the spectrum whatsoever, it's a single sinusodial peak at 1 kHz. This is the advantage of dithering - by adding noise, the quiet, smooth sinusoidal signal blends with the noise, eliminating any distortion that would otherwise result from the quantization. Since I simply used white noise, the above excercise is equivalent to using 4 bits of dithering with rectangular PDF. Adding more bits of noise to make the noise smoother does not improve the representation or the perception of the signal - the above signal quantized to 8 bits sounds exactly identical to the 16 bit signal. My ear cannot hear the slightest difference, despite one signal is a quantized stairstep pattern, and the other one is seemingly smooth. I tried to do a blind ABX test between them, to see whether I can differentiate between them - here are the test results: File A: noisy_sine_8bit.wav File B: noisy_sine_16bit.wav 17:14:15 : Test started. 17:16:12 : 01/01 17:16:28 : 01/02 17:16:53 : 01/03 17:17:21 : 02/04 17:17:43 : 03/05 17:17:58 : 04/06 17:18:21 : 04/07 17:18:38 : 04/08 17:19:23 : 04/09 17:19:51 : 04/10 17:20:28 : 04/11 17:20:53 : 05/12 17:21:08 : 05/13 17:22:16 : 05/14 17:22:57 : 05/15 17:23:31 : 05/16 17:23:31 : Test finished. -- Total: 5/16 Probability that you were guessing: 96.2% Out of 16 blind ABX listening trials, I could correctly tell which is which only 5 times. In other words, my ear doesn't hear the difference between the 8-bit quantized and the 16-bit version of this noisy sine wave signal. (Feel free to repeat this test yourself.) In theory, a 2-bit triangular PDF dither is enough to represent the signal without any distortion at all, hence more bits of dithering (= noise) is unnecessary, and there is not much you can gain further from having more bits of noise. All you get, is a smoother looking noise, and nothing else. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 25/07/2015, Tom Duffy tdu...@tascam.com wrote: You didn't change the bandwidth. I did. Probably you mean the 'sampling rate'. If the target signal is max 30Hz and you have a 192kHz sampler, you low pass at 2x your max frequency (60Hz, but lets say 100Hz for convenience) using a brick wall digital filter (processed at 192kHz). Then you do a downsampling of the signal from 192kHz to 100Hz. Then you end up with lower noise on the 30Hz signal, but you need to play it back with a 100Hz DAC. Yes, and then you end up with a very imperfect 30 Hz saw wave. Imagine how a 30 Hz saw wave sounds after you brickwall lowpass filter it at 100 Hz. It will only have 3 partials, hence it won't be a saw wave, or anything remotely similar. This will sound like this: http://morpheus.spectralhead.com/wav/saw30_100hz.wav Its spectrum will look like this: http://morpheus.spectralhead.com/img/saw30_100hz_spectrum.png Therefore, this will work only for sub-bass signals, but not for general broadband signals. Also, note that RBJ said measure a 30 Hz waveform, not measure a sub-bass waveform bandlimited to 30 Hz, which are two markedly different class of signals. A 30 Hz waveform could be a 30 Hz non-bandlimited saw wave, hence this won't work for an arbitrary 30 Hz waveform, which was claimed. Also, to be fully precise, for a 100 Hz DAC you'd need to brickwall lowpass filter it at 50 Hz not 100 Hz, since a signal of frequency F needs a sampling rate of 2*F. By brickwall filtering at 100 Hz and using a 100Hz rate DAC, the 60 Hz partial would alias to 40 Hz, and the 90 Hz partial would alias back to 10 Hz. If I actually resample the 100 Hz brickwall-filtered saw wave to 100 Hz, the resulting spectrum looks like this: http://morpheus.spectralhead.com/img/saw30_100hz_resampled.png Alias frequenices from the improper filtering are visible at 40 and 10 Hz (from partials at 60 and 90 Hz), hence, you'd need an 50 Hz lowpass filter to eliminate aliasing. So the 30 Hz saw wave, when properly resampled to a 100 Hz rate, would become a single sine wave at 30 Hz, because the second partial at 60 Hz is already filtered out. Not a very good way to perfectly reconstruct a saw wave. ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 23/07/2015, Peter S peter.schoffhau...@gmail.com wrote: Robert, what you say, simply makes no sense. You speak about imaginary converters with imaginary components that do not exist in the real world. If your imaginary converters were real, then we could simply have 138 dB dynamic range with dithering in any 24-bit sound card, no noise shaping would even be needed at *all*. Yet that doesn't happen There's no converter on the planet that has 138 dB dynamic range Hence, your imaginary converters with perfect components do not exist. Hence, your assumption that the major source of noise in a converter is numerical error, is flawed. The major sources of noise in any converter: 1) analog noise from the components 2) dithering noise Here's how Nigel Redmon expressed the same: Yes, Mark, it’s true that we have 24-bit converters now. But not really. That is, each additional bit gives us half of what the previous bit gave us. By the time you get to the 24th bit, its contribution is so small, that it’s now below the thermal noise of any real circuit we can put it in. For instance, if one bit gives us one volt, the 24th bit would give us one ten-millionth of a volt. Considering that the background noise of your circuit needs to be quieter than that in order for it to be effective, you’ll need to resort to cryogenics. But that’s not to say that we might as well use 20-bit converters. Converters are typically spec’d based on the accuracy of their least-significant bit. So, the 20th bit of a 24-bit converter is likely to be more accurate (maybe 16 times more accurate, at the same spec) than the 20th bit of a 20-bit converter. So we might as well use 24-bit converters, whether we think we can hear the last couple of bits or not. Other than that, we’re at the limits of physics here—the thermal noise of atoms isn’t going to change any time soon. So if someone tries to sell you a 32-bit converter 5 years from now, claiming a breakthrough in knowledge, just have a good chuckle about it. At 24 bits—144 dB dynamic range—distortion in the lowest bit will be buried under the noise floor of any electronics that you will play it through.[2] The good news: Pretty much anything you record will come with its own dither signal for free, because electronics of any kind generates more thermal noise than the 24-bit floor.[3] You can’t hear the error in a signal truncated to 24 bits because it’s buried in the thermal noise of the 24-bit converters and subsequent electronics.[3] Consider the words: thermal noise, real circuit, one ten-millionth of a volt, background noise, cryogenics, limits of physics, thermal noise of atoms, noise floor of electronics. If you disagree, please discuss it further with Nigel Redmon. You will find him on this mailing list, and hopefully he can teach you a few things about converters. I don't have much patience with slow learners. Source: What is dither? http://www.earlevel.com/main/1996/10/20/what-is-dither/ [1] comment at February 11, 2011 at 5:02 pm [2] comment at April 8, 2012 at 2:09 pm [3] comment at February 20, 2013 at 11:29 am - Peter ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 25/07/2015, robert bristow-johnson r...@audioimagination.com wrote: and that's not counting noise-shaping. Since noise shaping merely changes the shape of the noise floor, that's pretty much irrelevant. Noise shaping is equivalent to applying a filter to the noise floor, for example a first order noise shaping will simply change the noise floor to a -6 dB/oct slope, like applying a first order highpass filter to white noise (like I demonstrated earlier). The overall difference will be minor (but feel free to repeat the experiment using a different noise floor, and present your results). (Also note that upsampling does not involve noise shaping at all, unless it also involves quantization.) Seems the conceptual misunderstanding here is that dithering does not mean that the error is described as noise. oh dear. that's the whole point of it. do you understand what the difference between rectangular pdf dither and triangular pdf dither? I could ask the same from you. what it does to the quantization signal? the whole purpose is to take whatever nasty non-linearity (which a staircase function is) and turn it into something that is white (or bandlimited white), and with it's first two moments decoupled from the value of the input signal. Yes, by adding noise (using triangular, rectangular, or whatever probability distribution dither function). Sure, trivially you can describe noise as noise, since it is noise (what else), but it comes from _added_ noise from a noise generator, _not_ the noise from the quantization (step) function. Rather, dithering means that we deliberately *add* noise to the signal (because that has advantages, see the literature). sigh You cannot dither a signal if you only have a step function. That won't work. You *must* need a noise generator, hence, you're adding external noise to the signal. sigh is a really convincing technical argument, you almost convinced me. But let's see some sources: Quote from https://en.wikipedia.org/wiki/Dither: Dither is an intentionally applied form of noise used to randomize quantization error [...] Lipshitz and Vanderkooy pointed out that different noise types, with different probability density functions (PDFs) behave differently when used as dither signals, and suggested optimal levels of dither signal for audio.[10][11] Gaussian noise requires a higher level for full elimination of distortion than rectangular PDF or triangular PDF noise. Triangular PDF noise has the advantage of requiring a lower level of added noise to eliminate distortion and also minimizing 'noise modulation'. RPDF stands for Rectangular Probability Density Function, equivalent to a roll of a dice. Any number has the same random probability of surfacing. TPDF stands for Triangular Probability Density Function, equivalent to a roll of two dice (the sum of two independent samples of RPDF). Gaussian PDF is equivalent to a roll of a large number of dice. The relationship of probabilities of results follows a bell-shaped, or Gaussian curve, typical of dither generated by analog sources such as microphone preamplifiers. If the bit depth of a recording is sufficiently great, that preamp noise will be sufficient to dither the recording. Consider the meaning of the the words: intentionally applied form of noise, randomize, noise types when used as dither signals, triangular PDF noise, added noise, random probability, analog sources, preamp noise. Let's try what happens when I dither a 16-bit constant zero signal to a 8-bit dithered signal. 16-bit constant zero signal (all samples are zero): http://morpheus.spectralhead.com/wav/zero16bit.wav Same thing dithered to 8-bit, using triangular dither (no noise shaping): http://morpheus.spectralhead.com/wav/zero8bit_dithered.wav Its spectrum looks like uniform distribution white noise: http://morpheus.spectralhead.com/img/zero8bit_dithered_spectrum.png By applying logic, consider how is it possible that the original signal was a constant zero signal (all samples zero), and the resulting dithered signal looks and sounds exactly like white noise. Some further sources to confirm: Dithering is described in Section 5.1.2.5 as the addition of low-amplitude random noise to an audio signal as it is being quantized. The purpose of dithering is to prevent neighboring sample values from quantizing all to the same level, which can cause breaks or choppiness in the sound. Noise shaping can be performed in conjunction with dithering to raise the noise to a higher frequency where it is not noticed as much.[1] The concept of dither is to add some random noise to the waveform in order to break up the statistical determineability of the stair stepped waves. We do this by literally adding noise to the signal.[2] To dither means to add noise to our audio signal. Yes, we add noise on purpose, and it is a good thing.[3] Consider the meaning of the words: addition of low-amplitude random noise, add some random noise,
Re: [music-dsp] Non-linearity or filtering
Okay, a few more thoughts: On 23/07/2015, robert bristow-johnson r...@audioimagination.com wrote: okay, since there is no processing, just passing the signal from A/D to D/A converter, there is only one quantization operation, at the A/D. That's only true *if* it's a non-dithered converter (read: a converter from the previous century). if it's properly dithered, the error will be well described as noise that's white with DC=0 and AC power decoupled from the input value. Seems the conceptual misunderstanding here is that dithering does not mean that the error is described as noise. Rather, dithering means that we deliberately *add* noise to the signal (because that has advantages, see the literature). Failure to understand dithering will result in failure to understand noise shaping, since noise shaping is almost exclusively used together with dithering (= adding noise to the signal). This will result in failure to understand converters, and failure to understand that all audio converters made in this century have a constant noise floor that is independent of the signal. depends on what we have available for sample rates. essentially we are only limited by the laws in Information Theory. if i have a 192 kHz system and i only need to measure a 30 Hz waveform, there is a lot i can do to mitigate noise. Let's consider this question further. Let's consider that the 30 Hz waveform is a perfect saw wave, exactly as the one suggested in the post that started this thread (linearly up to a fixed point, virtually zero time back to a given voltage, and then very linearly up again), and the goal is to reconstruct that waveform perfectly. By saying there is a lot i can do to mitigate noise, I assume you probably mean to do some lowpass filtering to filter out supersonic frequencies of the signal. If you meant that, here is why it will worsen your reconstruction and your results: A perfect analog saw wave is (in theory) not bandlimited, meaning it will have signal energy up to infinte frequency. Therefore, if you apply any kind of lowpass filtering to a theoretically perfect saw wave, you will 1) remove some energy from the signal 2) make it bandlimited 3) create ripples in the waveform (see Gibbs effect) Therefore, if you sample a saw wave at 192 kHz and filter it at say, 20 kHz to remove components above 20 kHz, then what you essentially do, is you remove partials above 20 kHz, causing ripples, therefore adding error to the reconstruction. If you compare it to your original, perfect analog saw, then your measurement will not be better, but rather worse, so you didn't improve the reconstruction, but rather, worsened it. (This is assuming the goal was to reconstruct a perfect analog saw wave as perfectly as possible, if I understand right the starting post.) But in reality, this is not so simple, as you also have a noise floor, using whatever representation. Using 16 bit, the theoretical noise floor is at -96 dB, using 24 bits, it is at -144 dB. And in practice, the noise floor is typically above that, in reality, around -120 dB for a high quality sound card, corresponding to about 20 bits precision. Knowing that a saw waveform has partials with amplitudes decreasing at 6 dB/oct, it is expected that at some point, the ampltiude of a partial will fall below the noise floor. Let's calculate the frequency at which this happens, assuming the saw wave is a full-scale (0 dBFS) saw wave at 30 Hz, and the sound card is has 24 bits, having 20 bits dynamic range (-120 dB noise floor)! In that case, the amplitude of the partials will be: 30 Hz: 0 dB 60 Hz: -6 dB 120 Hz: -12 dB 240 Hz: -18 dB 480 Hz: -24 dB 960 Hz: -30 dB ... 3,932,160 Hz: -102 dB 7,864,320 Hz: -108 dB 15,728,640 Hz: -114 dB 31,457,280 Hz: -120 dB 62,914,560 Hz: -126 dB ... So, it means, that - assuming a noise floor of -120 dB - in order to be able to actually _improve_ the reconstruction of a 30 Hz full scale 0 dBFS perfect saw waveform by lowpass filtering, you would need to use a sampling rate of at least 63 MHz, that is, _mega_hertz, not kilohertz. Above that rate, the amplitude of the noise floor is higher than the amplitude of the partials, so if you filter out that, you'll actually improve your reconstruction (because you removed more noise than signal energy). By using any less sampling rate, by doing lowpass filtering you're removing partials from the signal that are louder than the noise, so you're worsening your reconstruction not improving it, and you'll have a worse reading on your measurement. (That is, assuming your measurement equipment is not bandlimited, and has a theoretical bandwidth of infinity, or at least up to several dozen MHz. Also, by reconstructing the signal, you're adding another -120 dB noise floor from the DAC, so it is questionable if this filtering would actually result in an actual improvement in your readings, unless your DAC is
Re: [music-dsp] Non-linearity or filtering
On 25/07/2015, Peter S peter.schoffhau...@gmail.com wrote: On 23/07/2015, robert bristow-johnson r...@audioimagination.com wrote: depends on what we have available for sample rates. essentially we are only limited by the laws in Information Theory. if i have a 192 kHz system and i only need to measure a 30 Hz waveform, there is a lot i can do to mitigate noise. By saying there is a lot i can do to mitigate noise, I assume you probably mean to do some lowpass filtering to filter out supersonic frequencies of the signal. If you meant that, here is why it will worsen your reconstruction and your results: [...] Since the numbers do not lie, let's test this experimentally. First, here's a 192 kHz saw waveform (for simplicity, only 16 bits): http://morpheus.spectralhead.com/wav/saw30.wav (It's a trivial, non-bandlimited saw wave, but that doesn't change the results.) Now, let's simulate the A/D converter's noise floor by adding some noise. To actually hear the noise, I'll add -48 dB white noise (because if I added -120 dB noise, you wouldn't hear it). For simplicity, I'll use uniform distribution noise. Here's the noise I am adding to the signal: http://morpheus.spectralhead.com/wav/noise.wav After mixing the 30 Hz saw wave with this noise, here's what I get (noisy saw): http://morpheus.spectralhead.com/wav/saw30_noise.wav So, we have a noisy saw wave. Let's assume that we want to eliminate some of that noise. So do what RBJ suggested, and let's do some lowpass filtering at 20 kHz to eliminate supersonic noise! (well, he didn't actually suggest this, but I think that is what he meant, so let's test it). After applying a 8th order lowpass Butterworth filter tuned to 20 kHz, here's what we have: http://morpheus.spectralhead.com/wav/saw30_filtered.wav If you listen to this, it will sound pretty much the exact same as the original noisy signal, since we only removed supersonic frequencies that the human ear doesn't hear. So they should identical (if they don't, then it means that there's some nonlinearity in the system you use to reproduce it). Hence, for the purpose of human perception, we actually haven't removed any of the noise, since we only removed noise that we didn't hear anyways. (Technically we removed about 3/4 of the noise, specifically the noise in the supersonic bands.) Let's see how the waveforms look before and after filtering: http://morpheus.spectralhead.com/img/saw30_waveform.png http://morpheus.spectralhead.com/img/saw30_filtered_waveform.png Apparently, the only notable difference is that we added a ripple at the transition. This is the error or artifact of this noise removal (which by the way, didn't remove any audible noise), hence we added this error to the signal. To see the error (difference) between the original and the filtered, let's subtract the original signal from the filtered, to see the error that this operation caused. If we check the resulting waveform, it looks like this: http://morpheus.spectralhead.com/img/saw30_diff_waveform.png This is the ripple from the filtering we applied, causing this artifact to appear. If we zoom in, here's how it looks from a close-up view: http://morpheus.spectralhead.com/img/saw30_diff_closeup.png If you want to listen how this error sounds, here's the difference between the original noisy saw and the filtered saw wave: http://morpheus.spectralhead.com/wav/saw30_filtered_diff.wav That's the error we introduced to the signal by doing the filtering, so this is the artifact of this noise removal. Since the original noise amplitude was -48 dBFS, and the amplitude of the error of the filtering is 0 dBFS, it means that the error we introduced is about 250 times louder than the original error from the noise. That's the cost of removing ~3/4 of the original noise present in the supersonic bands using this method - we instead added a large ripple to the signal. Conclusion: FAIL As expected, this method does not improve the reconstruction, because it introduces significantly more error than it removes, by causing large ripples at the transitions that are lot higher in amplitude than the amplitude of the noise it removes. So what we end up with, is worse than what we started with, as it has several hundred times more overall error, because we removed a lot more signal energy, than the amount of noise we removed. Best regards, Peter ___ music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
Since there's this contant behavioral pattern on this mailing list, namely that irregardless of the topic, there's always some person who basically has no clue, but tries to prove me wrong and teach me a lesson, it's getting extremely boring. Yes, those comments were excellent quality, if you discard that they only apply to soundcards that are several decades old and belong to the museum, hence off by about 20 years. Since this behavioral pattern happens all the time, and is so extremely boring, I decided to take a break from participating in this mailing list. I've yet to decide whether I premanently unsubscribe, or just take a break. Have a nice discussion. Best regards, Peter Schoffhauzer -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
After checking at least the first half dozen papers I linked, it should be apparent that 1) a dithered sigma-delta converter is typically better quality than one without dithering 2) dithering means adding noise to the signal (usually white noise, or a modified spectrum via noise shaping) 3) dithering adds a constant noise floor to the signal 4) dithering eliminates signal-dependent quantization artifacts, hence the noise floor becomes entirely independent of the input (*) 5) dithering is usually placed inside the feedback loop of the noise shaping (**) (*) Such dithering, with the optimal triangular probability density function (TPDF) dither, in principle completely eliminates all distortion, noise modulation, and other signal-dependent artifacts, leaving a storage system with a constant, signal-independent, and hence benign noise floor.[1] (**) One criticism of the 1 bit converter [...] is that because only 1 bit is used in both the signal and the feedback loop, adequate amounts of dither cannot be used in the feedback loop and distortion can be heard under some conditions.[1][2] Most A/D converters made since 2000 use multi-bit or multi-level delta sigma modulators that yield more than 1 bit output so that proper dither can be added in the feedback loop. For traditional PCM sampling the signal is then decimated to 44.1 ks/s or other appropriate sample rates.[3] So, I would expect the overwhelming majority of sigma-delta based soundcards to be dithered, since those are better quality. Hence, RBJ's assumptions about sigma-delta converters only apply to a small subset of sigma-delta converters, namely the low quality, non-dithered converters, that are not used in modern sound cards (maybe only in some old and obsolete, pre-2000 sound cards). Therefore, my assumptions about the signal-independent, constant noise floor of converters was correct, since the overwhelming majority of sigma-delta converters are expected to use dithering. Best regards, Peter References: [1] Why 1-Bit Sigma-Delta Conversion is Unsuitable for High-Quality Applications http://sjeng.org/ftp/SACD.pdf [2] Why Professional 1-Bit Sigma-Delta Conversion is a Bad Idea http://peufeu.free.fr/audio/extremist_dac/files/1-Bit-Is-Bad.pdf [3] https://en.wikipedia.org/wiki/Noise_shaping -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 23/07/2015, Peter S peter.schoffhau...@gmail.com wrote: sorry to point out, but also having invalid assumptions - tell me, have you ever tried to record something with no input using a 24-bit converter? Have you ever looked what you have at the lowest bits? Do you understand what the term noise floor means?). In case you never did that in your life, have never heard about noise floor, and you don't have an actual soundcard to test this, here's what happens when you record input without connecting any actual input. I tested this using a 16-bit sound card, by selecting Line In as input, without connecting anything. After normalizing what I recorderd to -3 dBFS, where's what I got: http://morpheus.spectralhead.com/img/linein.png Here's what it's averaged spectrum looks like (after normalization): http://morpheus.spectralhead.com/img/linein_spectrum.png Here is its spectrogram (Y axis = 20..22050 Hz): http://morpheus.spectralhead.com/img/linein_spectrogram.png And here is how it sounds like: morpheus.spectralhead.com/wav/linein.wav To me, this pretty much looks and sounds like, near uniform distribution white noise, mixed with some small amplitude constant hum (which is a particular feature of this low quality 16-bit converter). Other than that hum, it's pretty much just noise. This is *definitely* not a function of the input, since I connected no input at all, yet what I got, is very similar to white noise. This is practically unpredictable, so no way to fix this for a broadband signal. This is what we call noise floor, and literally every sound card has this. In any 24-bit sound card, typically the lowest 3-4 bits will be noise. I wonder you never heard about this. Example: in this Sound Blaser Live! 24-bit analysis test, the noise floor looks like near uniform distribution noise at around -133 dB (second graph titled Noise level): http://www.gmaptool.eu/audio/rmaa/SBL2-24-48.htm Since -133 dB is defnitely above -144 dB, it implies that the lowest bits will contain this near uniform distribution noise (and if you actually look at the bits of a recording of no input, you'll basically see random bits with near uniform distribution). If you have an imaginary converter that uses imaginary Area 51 alien technology that has superconductive electrical components cooled by liquid oxygen, then you may get a 0 bit noise floor (-144 dB at 24 bit precision). If you leave the imaginary land, and get back to the real world, then you pretty much *always* have a noise floor irregardless of input, using whatever real-world technology, and that typically looks quite similar to uniform distribution noise. well, then you're missing something. that's the main noise. if it ain't, then the hardware design is dirty. The hardware design is always dirty, unless it's some imaginary alien technology that doesn't exist on Earth and haven't yet been invented. while delay might be something to worry about, Yes, error from the mismatched delay is another interesting error source. semiconductor noise is *not* what i would worry about. Maybe you don't worry about it, yet it is *always* there. if i had to worry about it, there is something else deeply wrong in the hardware circuit. Or it just means it is a real-world circuit, and not an imaginary circuit. you mean an open circuit on the other end of the cable? Yes. well, there's hum and there's noise. not the same thing. The noise is always there, and it cannot be predicted. and this noise has a numerical root. Yes, but not *all* noises in the signal have numerical root. better look up sigma-delta converters. Better look up real-world circuits. the error due to non-linearity of the A/D is a function of the input. Yes. And the noise _floor_ of the converter is *not* a function of the input. the noise floor, due to an open circuit, might include hum, because your open circuit cable might be a little antenna. Exactly. That doesn't mean it doesn't have a noise component, which is unpredictable, thus unfixable. one reason is that we cannot make parts with the tolerances necessary. Hence, you'll *always* have noise floor from the components. I'm glad you finally admit. no, i just know something about oversampling. And surely sound card manufacturers never heard about oversampling... to what degree you reduce noise depends on the bandwidth of interest of your signal and the sampling rate of the A/D and D/A. We were speaking about broadband signals, as far as I know. do you understand the sources of that noise floor? Do you? exactly what audio codecs are you using? if they're sigma-delta (which are, now-a-daze, nearly ubiquitous in audio), the major component of the noise floor is actually from the operation of the converter and has a numerical root. Major component != only component that is louder than the noise from the analog front end Quantization noise being louder doesn't mean that the noise from the analog front-end
Re: [music-dsp] Non-linearity or filtering
On 23/07/2015, robert bristow-johnson r...@audioimagination.com wrote: Peter, do you know how a sigma-delta (or delta-sigma, people don't always agree on the semantics) converter works? like how a sigma-delta modulator works? oversampling? possible dithering? noise-shaping? decimation (involving low-pass filtering and downsampling)? do you have any idea what i am talking about (or writing about)? Yes I do. Do you have any idea that 'dithering' means adding noise to the signal? Do you realize that -110 dB is a *lot* louder than the theoretical noise floor of an 1-bit dithered 24-bit signal without noise shaping, which would be at -138 dB? If - according to you - they could have a -138 dB noise floor, why do they have a -110 dB noise floor instead? Ever wondered that? That makes zero sense. Do you realize that noise shaping *increases* the amplitude of the noise? also, even in the least-significant bits, there is signal embedded (or buried) in the noise. it's not as if they appended 4 noisy bits to the right of a 20-bit word. they didn't do that. Despite that, it's still *noise*, hence it is unpredictable, hence it is unfixable. If you have an 1-bit amplitude signal, and embed it in 4 bits of noise, then you won't hear that 1-bit amplitude signal because the 4 bits of noise will mask it entirely. Hence, your lowest bits are unusable, because the noise entirely masks it. -P -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
If noise shaping with oversampling actually worked to eliminate the noise floor, then 24-bit sound cards would have 160 bit dynamic range. Yet for *some* reason, they have 110-115 dB dynamic range instead, not even approaching the theoretical 144 dB range of a 24 bit signal. Think about that. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
Robert, what you say, simply makes no sense. You speak about imaginary conveters with imaginary components that do not exist in the real world. If your imaginary converters were real, then we could simply have 138 dB dynamic range with dithering in any 24-bit sound card, no noise shaping would even be needed at *all*. Yet that doesn't happen There's no converter on the planet that has 138 dB dynamic range Hence, your imaginary converters with perfect components do not exist. Hence, your assumption that the major source of noise in a converter is numerical error, is flawed. The major sources of noise in any converter: 1) analog noise from the components 2) dithering noise Do you realize that 'dithering' basically means: adding noise? to what end? why do you think they would they be adding that noise? how would that help? By asking that question, it seems that you do not understand the purpose of dithering, at all. Specifically it is to eliminate the quantization noise (and turn it into a noise floor instead), and to increase the dynamic range. Quote from Noise shaping page from Wikipedia: Noise shaping must also always involve an appropriate amount of dither within the process itself so as to prevent determinable and correlated errors to the signal itself. If dither is not used then noise shaping effectively functions merely as distortion shaping — pushing the distortion energy around to different frequency bands, but it is still distortion. If dither is added to the process as y[n] = x[n] + A_1*e[n-1] + dither then the quantization error truly becomes noise, and the process indeed yields noise shaping. Source: https://en.wikipedia.org/wiki/Noise_shaping So probably you should at least understand the basics of noise shaping and dithering first. -Peter -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 23/07/2015, Theo Verelst theo...@theover.org wrote: I'm not answering much to treacherous psychopaths (from the use of words and the content of the communication here, anyway) Okay, I think this was the last case when I tried to teach you anything about digital signal processing or waste my valuable time to try to demonstrate some DSP processes to those small handful of readers who read these threads. Good luck figuring out the mysteries of DSP, learning what dithering and noise shaping is, etc. You clearly don't need the advice of psychopaths, you can learn all that without me as well. Best regards, Peter -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 23/07/2015, Johannes Taelman johannes.tael...@gmail.com wrote: RBJ's comments are of excellent quality, and are not about obsolete or imaginary technologies. Also, 16-bit converters are pretty much obsolete (if you want high-quality audio). Also, 24-bit converters with near zero noise floor are currently nonexistent (=imaginary). Hence, these are either obsolete or not yet existing technologies. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
Noise shaping is a filtering process that shapes the spectral energy of quantization error, typically to either de-emphasise frequencies to which the ear is most sensitive or separate the signal and noise bands completely. If dither is used, its final spectrum depends on whether it is added inside or outside the feedback loop of the noise shaper: if inside, the dither is treated as part of the error signal and shaped along with actual quantization error; if outside, the dither is treated as part of the original signal and linearises quantization without being shaped itself. In this case, the final noise floor is the sum of the flat dither spectrum and the shaped quantization noise. While real-world noise shaping usually includes in-loop dithering, it is also possible to use it without adding dither at all, in which case the usual harmonic-distortion effects still appear at low signal levels. Source: https://en.wikipedia.org/wiki/Dither#Digital_audio I think it is clear from the above that without dithering, you get your regular quantization errors that accompany low amplitude signals. Hence, noise shaping is usually used in combination with dithering, in that case, you add noise to the signal to increase dynamic range and eliminate the quantization noise, and get a fixed noise floor instead. Noise shaping basically just modifies the spectrum of the noise, at the expense of increasing the amplitude of the noise. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
Here is a somewhat random selection of 24-bit sound cards with SNR data included: http://sound-cards-review.toptenreviews.com/ Output SNR: a) 124 dB b) 124 dB c) 109 dB d) 112 dB e) 117 dB f) 109 dB g) 100 dB h) 113 dB Input SNR: a) 118 dB b) 118 dB d) 98 dB e) 115 dB g) 100 dB h) 113 dB Let's calculate how many bits are noise in their input / output when using 24 bit precision! To do that, simply subtract these numbers from 144 dB (the theoretical maximum SNR for a 24-bit card, not counting dithering), then divide the difference by 6.02 to get the amount of noise in bits. Here is what you get: Output: a) 3.3 bits b) 3.3 bits c) 5.8 bits d) 5.3 bits e) 4.5 bits f) 5.8 bits g) 7.3 bits h) 5.1 bits Input: a) 4.3 bits b) 4.3 bits d) 7.6 bits e) 4.8 bits g) 7.3 bits h) 5.1 bits So, according to the specification of these sound cards, the lowest 3-7 bits are expected to be noise. In theory, if you merely quantize a signal to the nearest bit without dithering and noise shaping, the peak-to-peak amplitude of that noise will be 1 bit, since the error will be between -0.5..0.5. Trivially, 3-7 bits of noise is a *lot* higher than the theoretical quantization noise of 1 bit. However, dithering and noise shaping may change quantization noise amplitude. More specifically, in spite of what Robert claims, when applying dithering with noise shaping, then the amplitude of the noise will actually *increase*, and not decrease. The reason why it may be perceived as more silent, is that the noise is typically pushed into bands where the human hearing is less sensitive. This does *not* mean that the amplitude decreases, rather on the contrary, it actually increases - the higher order the noise shaping, the more the amplitude increases. To demonstrate this, here are some samples of various noise shaping algorithms applied to a 8-bit signal: [WARNING: these samples may contain loud high frequencies, so set your playback volume to low.] Without dithering, the SNR would be -48 dB. 1-bit triangular dithering, no noise shaping: http://morpheus.spectralhead.com/wav/shaping_off.wav Noise floor: -42 dB 1-bit triangular dithering, noise shaping A: http://morpheus.spectralhead.com/wav/shaping_a.wav Noise floor: -42 dB 1-bit triangular dithering, noise shaping B: http://morpheus.spectralhead.com/wav/shaping_b.wav Noise floor: -36 dB 1-bit triangular dithering, noise shaping C: http://morpheus.spectralhead.com/wav/shaping_c.wav Noise floor: -30 dB 1-bit triangular dithering, noise shaping D: http://morpheus.spectralhead.com/wav/shaping_d.wav Noise floor: -30 dB 1-bit triangular dithering, noise shaping E: [WARNING: loud high frequency noise] http://morpheus.spectralhead.com/wav/shaping_e.wav Noise floor: -12 dB As you see, the very high order noise shaping introduced a -12 dB (25% amplitude) noise floor, meaning that 6 out of 8 bits in the signal (75% of the bits) are basically noise, more specifically, dithering noise. So much about noise shaping eliminating noise. In fact, dithering and noise shaping is the best way to introduce noise and *increase* the noise floor in the signal. the *major* component of audible noise is coming from the numerical processes inside the codec Seriously, where do you get that from? Do you realize that 'dithering' basically means: adding noise? Best regards, Peter -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 23/07/2015, robert bristow-johnson r...@audioimagination.com wrote: exactly what audio codecs are you using? if they're sigma-delta (which are, now-a-daze, nearly ubiquitous in audio), the major component of the noise floor is actually from the operation of the converter and has a numerical root. that is louder than the noise from the analog front end, unless your analog front end is particularly noisy. I'd argue that's not necessarily true. If it is a 24-bit converter where the lowest 4 bits are noise (due to analog noise), and we assume the peak-to-peak amplitude of the quantization noise to be 1-bit (since the maximum error between the original and the quantized signal is maximum 1 bit), then the former is certainly louder than the latter (precisely, 4x as loud). Hence, the major component of the noise floor is actually from the operation of the converter and has a numerical root is not necessarily true. The analog noise can be a lot louder than 1-bit (typically, up to 3-4 bits or more in a 24-bit converter). If you only worked with 8 or 16-bit converters in your life, then that statement may be true. Yet it's typically not true in case of 24-bit converters, which usually have several bits of noise floor. -P -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
Okay, I'll quit this discussion. It's quite pointless anyways - there's not very much you can do to improve the reconstruction of your soundcard, at least nothing that would be as good as buying a better sound card. Also if you fail to notice that the current year is 2015, and the rules you learned 20 years ago for 8-bit and 16-bit converters do not necessarily apply for today's typical 24-bit converters (that usually have several bits of noise in the lowest bits), then there's not much point in the discussion, because you're speaking about imaginary converters, and I'm speaking about real-world converters, so we will never agree. Good luck discussing this topic further. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 22/07/2015, Theo Verelst theo...@theover.org wrote: distortion figures indicate. Which should be a lot better than .1 dB which would imply an error of over 1 percent, which wouldn't be very good for a 50s HiFi system. But the specified harmonic distortion of a lot of well known DACs sure isn't such that 24 bits accuracy is achieved. A typical high-end card is a lot better than that. Example: http://www.ixbt.com/proaudio/lynxstudio/lynxaurora8%28+4dbu%29-2444.shtml THD: 0.0005% IMD + Noise: 0.0007% The largest peak at the 1k THD graph is around -110 dB. That's far below the threshold of human hearing. 115 dB dynamic range would imply about 19 bits precision. Frequency response is total flat up to 20k (0.02 dB error). That would mean 0.22% amplitude error at 20 kHz. Yours for only $2595. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
You have your signal S. When you digitize that signal, you add the noise floor of the ADC (among other noises), let's call it N1. When you reconstruct the signal, you add the noise floor of the DAC (among other noises), let's call that N2. So you have S + N1 + N2 Then you subtract the original signal (let's disregard the delay) and measure what you have: S + N1 + N2 - S Since S - S = 0, what you're measuring is: N1 + N2 In other words, you're measuring the combined noise floor from your ADC and DAC (among other noises). And you're trying to fix that noise. Here's why it won't work - the noise floor of the ADC and DAC are pretty much close to uniform distribution noise, without any pattern. The lowest bits are just random. In order to be able to fix that noise, you'd need to be able to predict that noise. But since you cannot predict noise, there's pretty much nothing you can do to fix that noise... Once you reach the noise threshold of your ADC/DAC, there's nothing you can do to improve your reconstruction (other than, buying another soundcard that costs a few thousand dollars more, and has a lower noise floor). If your soundcard is not so high-end, and has a slight roll-off at high frequencies, then the best you may do is correct that with a filter, and improve your readings slightly. About that's all you can do, you cannot fix noise. If noise were fixable, then sound card manufactures would already be doing it. How much you can fix noise, correlates with how many thousand dollars you want to spend on your sound card, and even that works only up to some limit, above which, there's nothing you can do - you have noise, whatever you do. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
On 23/07/2015, robert bristow-johnson r...@audioimagination.com wrote: okay, since there is no processing, just passing the signal from A/D to D/A converter, there is only one quantization operation, at the A/D. if it's an old-fashioned conventional A/D, the quantization operation is, essentially, a non-linear staircase operation and the error is a measurable and then predictable function of the input. if it's properly dithered, the error will be well described as noise that's white with DC=0 and AC power decoupled from the input value. with noise-shaping, the noise wouldn't have to be white, but the AC power level would increase. sigma-delta (Σ-Δ or Δ-Σ) converters might have a similar signal+noise model. that's N1. Let me point out to some very important detail that you've just missed - I was not talking about quantization noise at *all*. I implied that in the clause among other noises, without giving any detail about quantization noise _whatsover_. I was talking about the noise _floor_, which is a remarkably different type of noise. If you have any converter and plug in some cable, without any actual input/output, what you'll measure is *noise* (both on the ADC and the DAC). I was talking about _that_ (noise floor), not the noise from the quantization. Yes, there's *also* quantization noise, which I was not even mentioning at all, so it's not in the formulas that I wrote, which would be: S + N1 + Q1 + N2 + Q2 where Q1 and Q2 are the _quantization_ noise from the ADC and DAC, and N1/N2 are the noise from the noise _floor_ of the ADC and DAC. there *is* error at the D/A, due to non-linearity, but i wouldn't call that noise. There is _definitely_ a noise floor. Which is, pretty much just noise. If you record the input of your 24 bit sound card without plugging in *anything*, you pretty much get just noise (whatever exact distribution, possibly depending on the noise shaping of the converter and other factors). and it's a function of the input. The noise floor is definitely NOT a function of the input. Even if you have no input at all (= you plug in _nothing_), you *will* get noise in the lowest bits of a 24-bit converter. *Irregardless* of the input. Even using the most high-end sound cards. The lowest 3-4 bits will *always* be noise. Feel free to test this experimentally, if you doubt my words If there's *no* noise floor, then why don't we have 32-bit converters with near 192 dB dynamic range? According to what you say, just apply some noise shaping, and voila! ~192 dB dynamic range! Yet that doesn't happen. In order to be able to fix that noise, you'd need to be able to predict that noise. or make use of the statistics of that noise (known in advance). You cannot use the statistics of the noise floor to fix (= eliminate) that noise. No matter how many statistics papers you read, that won't work. If it was fixable, then please tell me, why do even the best, most expensive, $3000 sound cards will have just noise in the lowest bits? Tell me. If that were possible, then those zillions of dollars spent on converter research, should have been enough to figure it out, don't you think? Or you know something that _all_ sound card manufacturers don't know? but if the noise is colored (and you know about the spectrum) and if the noise has a p.d.f. that is not uniform, you can make guesses about where the next sample is that are better than wild-assed guesses. Yet you cannot fix it (=eliminate it). Yes you may improve it _slightly_ with noise shaping (so you push it into bands where psychoacoustically it may sound like less noise to the human ear), but that won't eliminate it. Sorry that's impossible. No matter how many degrees you have and how many thousand hours you spend in the library, that just won't work. And even if that gives you psychoacoustically better noise profile, that won't fool a measurement equipment (which is unaffected by psychoacoustics, unless it's deliberately built in via some weighting curve.) And there's no magic alien technology that will eliminate the noise _floor_ of a converter, and magically give you 32-bit 192 dB dynamic range. No such converter exists. Why don't 24-bit converters have 144 dB dynamic range? Answer: noise floor. Peter, if you can take or sit-in some grad courses, i might recommend a course in Statistical Communications or at least a course in Random or Stochastic Processes. there is ostensibly some stuff you're missing here. Robert, your constant consdecending is starting to get really boring. Earlier you told me to not become patronizing, yet now you do this condescending behaviour (even without understanding what I say, and sorry to point out, but also having invalid assumptions - tell me, have you ever tried to record something with no input using a 24-bit converter? Have you ever looked what you have at the lowest bits? Do you understand what the term noise floor means?). And by the way, yes, I know about noise
Re: [music-dsp] Non-linearity or filtering
So... if you sell your car and buy the most expensive sound card you can get, your readings may improve by 10-20 decibels. By doing some digital voodoo, you may increase your match between the original and the resonstructed signal by a few decibels... But beyond that, there's no other way than to shell out a few thousand dollars for better converters. Typically, 24 bit converters have a dynamic range of only about 18-20 bits. The lowest bits are generally just noise, even in the most high-end cards - the maximum dynamic range is around 110-115 dB, rarely above that. Beyond that, it's just noise territory. Since noise is basically entropy (= unpredictability), you cannot predict it, hence you cannot fix it... That's as best as you can get. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] Non-linearity or filtering
So in short why this won't work well: Trying to correct a highpass filter, you need an inverse filter that has a gain of infinity at DC (since the highpass filter has gain of -infinity at DC). Problem is, -infinity + infinity != zero, so you likely end up with a signal that has increasingly growing DC offset, until you get into clipping. To fix that, you'd need to put another DC filter on the signal... Eventually you may end up with something that's somewhat better than the original, but certainly not as good as throwing away your soundcard and buying a better DC-coupled sound card that has no capacitor on the input at all, obviating the need for correcting it. That's the best you can do. Also, when you want to compare the output of the corrected signal, then - unless you use a soundcard that has a DC-coupled output - the DAC will also introduce distortions, changing its shape by eliminating the frequencies below ~20 Hz. So your measurement will be off, and the shape of the output will differ from the shape of your digital wave. So unless you use a DC-coupled soundcard, this won't work well, and if you use a DC-coupled soundcard, then this whole 'correction' is unnecessary. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 20/07/2015, Peter S peter.schoffhau...@gmail.com wrote: If we extend this idea further and include not just the minimum size of the data of an ideal data compressor, but also the size of the decoder program as well and forget about the probabilities, then we get the algorithmic entropy (Kolmogorov complexity) C(c). Kolmogorov complexity is a modern notion of randomness dealing with the quantity of information in individual objects; that is, pointwise randomness rather than average randomness as produced by a random source. It was proposed by A.N. Kolmogorov in 1965 to quantify the randomness of individual objects in an objective and absolute manner. This is impossible by classical probability theory (a branch of measure theory satisfying the so-called Kolmogorov axioms formulated in 1933). Kolmogorov complexity is known variously as `algorithmic information', `algorithmic entropy', `Kolmogorov-Chaitin complexity', `descriptional complexity', `shortest program length', `algorithmic randomness', and others. The Kolmogorov complexity of an object is a form of absolute information of the individual object. This is not possible to do by C.E. Shannon's information theory. Unlike Kolmogorov complexity, information theory is only concerned with the average information of a random source. `Kolmogorov' complexity was earlier proposed by Ray Solomonoff in a Zator Technical Report in 1960 and in a long paper in Information and Control in 1964. Also Gregory Chaitin proposed this idea in a paper in J. ACM in 1969. This paper continues a paper by G. Chaitin in J.ACM in 1966, which however was concerned with a complexity measure based on Turing machine state-symbol product following ideas of C.E. Shannon. Unlike Kolmogorov complexity, those complexity measures are not objective and absolute (recursively invariant). The incompressibility method and Kolmogorov complexity is truly a versatile mathematical tool. It is a sharper relative of classical information theory (absolute information of individual object rather than average information over a random ensemble) and yet satisfies many of the laws of classical information theory---although with a slight error term. Source: http://homepages.cwi.nl/~paulv/kolmogorov.html Seems I'm not the first in the history of mankind to consider the amount of information in an arbitrary piece of data that is not a function of a probability distribution - others already did the same about a half century ago. -P -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 19/07/2015, robert bristow-johnson r...@audioimagination.com wrote: On 7/18/15 10:10 PM, Peter S wrote: It follows from the above, that Shannon's entropy model is a simplified, idealized model of information, that pretends that algorithms have zero length and thus no entropy, and you can magically share a codebook telepathically without actually transmitting it. Shannon makes no issue of that. if you need the codebook at the other end (because you don't already have it), you need to send it. It just implies to have a (nonzero length) codebook, that may need to send, without explicitly building it into the formulas. Shannon makes no issue of that, yet it's a real issue. it's not so big and i have seen that in some compressed files More precisely: its size depends on the entropy (unpredictability) of the distribution of symbol set in the symbol space. If there symbol space is bound and not too large, and there is an efficient representation of the symbols in that space (meaning they're highly predictable), *then* the codebook may be small. If the symbols (messages) to be sent are 1,2,3,4,5, then that's easy to transmit in a compact form, since they're consecutive, small numbers. If that's not the case, then the codebook may be actually large, in the worst case, as large as your symbol set. Example: consider the size of the codebook when the messages to be sent are the following: 163, 717, 7916, 4697, 50475, 469455, 53507017, 44617430, 4753678431, 3454009715, 924306674532, 364168088575 How do you represent these numbers as a compact codebook? Anwer: you can't. Since I sampled these numbers from uniform distribution noise, they're highly unpredictable (hence maximal entropy). So you basically have no way to represent them in a sequence that is smaller than their original size, so you pretty much need to send each of these numbers in the codebook 'as is', to be able to decode them later. White noise is uncompressable, hence there is no way to represent it shorter than its size. Hence, in the corner case when the messages to be sent are sampled from white noise, then the minimum size (entropy) of the codebook will equal the size of the symbol set to be sent. There is _absolutely_ no way to compress that further, no matter what you do. If you say but if that's the only symbol set to be sent with 100% probability, then I can just 'build it into the receiver' and so it has zero entropy. By doing that, you're effectively 'cheating' - all you did, is you just put those numbers into your program and turned it into algorithmic entropy, and pretend that there's no entropy. In fact, it's still there, except the numbers are not in a codebook that is sent, but rather, inside your program and you're pretending that your program has zero length. Without your program containing the above numbers, it cannot reconstruct the codebook, and hence, cannot reconstruct the encoded signal to be sent, whatever probability distribution. You cannot say but the entropy of the probability distribution of the symbol set is small, then I can just encode that probability distribution as a short number. Yes, you can encode the probability distribution of the codebook, but then to transmit the original codebook, you would first need to transmit *another* codebook, containing the symbol set of the first codebook, and in case of uniform distribution, you're nowhere further... That's like, trying to recursively apply data compression - no matter how many times you try to recursively compress white noise, it will never get smaller, whatever you do. What you effectively do, is you just keep adding extra bits and boilerplate to the original data with the extra codebooks you may send, but it *never* gets shorter. And by 'building it into your receiver', you merely put those bits into your program instead, and pretend it has zero size. Note, that the size of the codebook depends on the entropy (unpredictability) of the distribution of the symbol _set_ in the symbol space (which is not the same as the actual distribution of _symbols_ to be sent, but rather, the distribution of the _set_ of symbols, as that's what the codebook contains). Since that is unpredictable, it will also have entropy, which will depend on how small you can represent that distribution. If you want to recursively represent the codebook by encoding it further, then you'll first need to send *another* codebook to represent the *first* codebook... And then there's a limit of how much recursive compression you can apply to something, at some point you'll reach its (theoretical) entropy, and you cannot compress it further. Hence, its _actual_ entropy (minimal length to represent it) will *only* depend on the Kolmogorov complexity (algorithmic entropy) of the symbol set, which - in the corner case of uniform distribution noise - is the same as its length, hence no way to represent it shorter. By applying probabilistic Shannon theory and saying that the entropy
Re: [music-dsp] about entropy encoding
If we assume the Shannon entropy of a probabilistic ensemble p to be H(p) = -K * SUM p_i * log2(p_i) where p_1, p_2, ... p_k is the probability of the ith message, and K is merely a normalization constant, then if the number of messages sent is N, it can be exressed as H(p) = -K * SUM c_i/N * log2(c_i/N) where c_i denotes the frequency (count) of the ith message out of N messages in an ensemble of messages. If assume that the codebook for the ensemble must be sent before the transmission for the decoder to be able to understand it, and the (theoretical) entropy of the codebook is denoted H(c), then the actual entropy H' of a message in the ensemble of N messages combined with the entropy of the codebook will be H(c) H' = H + N because the average overhead per messages equals the minimum length of the codebook - in other words, the entropy of the codebook H(c) - divided by the total number of messages in the ensemble, N. (Note this is not fully precise as it contains the theoretical, not actual entropy of the codebook, more on this later.) Since several consecutive transmissions can be possible with different message sets and associated codebooks, it can be also thought of as a probabilistic ensemble, denoting the probability of that particular set of message out of all possible sets of messages. Example: the following sets may denote possible sets of messages over a series of transmissions: (1,2,3,4,5) (23,111,259,3636,15523) (a,b,c,d,e,f) (1,10,111,0101,1001110,01110111010) The actual representation is irrelevant, a set of messages is effectively always a set of numbers that represent a set of messages (whether they mean characters, strings, or something else). Therefore, each possible set of messages (numbers) may be associated with a probability, meaning the probability of the set of messages sent in a transmission being that particular set of messages. From that it follows, that it has an associated (probabilistic) entropy, which can be calculated. Let 's_j' denote the jth set of messages in a series of transmissions, and let 'q' denote a probabilistic ensemble meaning the probabilties associated with each possible set of messages s_1, s_2, ... s_l. In that case, the entropy of the probabilistic ensemble of the messages sets is H(q) = -K * SUM q_i * log2(q_i) where q_i means the probability of the ith message set being the particular set of messages in transmissions among a series of transmissions. We can estimate the probability of message set 's_j' from the frequency of that message set occurring across several transmissions of ensembles of messages. Let M denote the total number of transmissions of message ensembles over a period of time. In that case, the entropy of the probabilistic ensemble of the message sets will be H(q) = -K * SUM d_i/M * log2(d_i/M) where d_i denotes the frequency count of the ith message set occurring over a series of transmissions, and M is the total number of transmissions. Let 'c_j' denote the jth codebook in a series of transmissions. Let 'r' denote the probability distribution of the probability distributions (p1, p2, ... pM, each denoting the probability distribution of the messages in the jth transmission) over the set of M transmissions. Since the codebook contains the set of messages and their associated probabilities needed to decode the jth transmission, the theoretical entropy of the codebooks H(c) equals H(c) = H(q) + H(r) where q is the probability distribution of sets of messages, and r is the probability distribution of probability distributions. This is because both the set of messages, and the probabilities need to be transmitted in the codebook to uniquely reconstruct the code. Let H(c) mean the theoretical (probabilistic) entropy of the codebook, and H'(c) mean the actual entropy of the codebook, meaning the minimum side of each message needed to encode the codebook, and the minimum size of the second codebook that encodes the original codebooks (using a recursive definition), since that must also need to be sent to reconstruct the codebook. The codebook can be further encoded (compressed) as well (unless it has maximal entropy), so H'(c2) H'(c) = H(c) + -- M where H'(c2) denotes the actual entropy of the second codebook, encoding the first codebooks to be used in the series of transmissions, and M is the number of transmissions. This is a recursive definition, H'(c2) could be broken down further as the sum of H(c2) (meaning the theoretical entropy of codebook #2) and H'(c3) (meaning actual entropy of codebook #3 that encodes codebook #2) divided by M, and so on, so H(c4) + ... H(c3) + --- M H(c2) + -
Re: [music-dsp] about entropy encoding
On 19/07/2015, robert bristow-johnson r...@audioimagination.com wrote: even so, Shannon information theory is sorta static. it does not deal with the kind of redundancy of a repeated symbol or string. just with the probability of occurrence (which is measured from the relative frequency of occurrence) of messages. run-length coding isn't in there. maybe there is something in Shannon information theory about information measure with conditional probability (that might obviate what LPC can do to compress data). I think that's a bit of an oversimplification, there is a lot of detail in the small print. Consider: A more complicated structure is obtained if successive symbols are not chosen independently but their probabilities depend on preceding letters. In the simplest case of this type a choice depends only on the preceding letter and not on ones before that. The statistical structure can then be described by a set of transition probabilities p_i(j), the probability that letter i is followed by letter j. The next increase in complexity would involve trigram frequencies but no more. The choice of a letter would depend on the preceding two letters but not on the message before that point. A set of trigram frequencies p(i,j,k) or equivalently a set of transition probabilities p_ij(k) would be required. Continuing in this way one obtains successively more complicated stochastic processes. In the general n-gram case a set of n-gram probabilities p(i1,i2,...,i_n) or of transition probabilities p_i1,i2,...,i_n-1(i_n) is required to specify the statistical structure. To me this sounds like a (nonlinear) predictor that tries to predict a symbol from the preceding symbols. -P -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
I had an idea on how to improve the entropy estimate, especially for some types of signals. Here are the results: http://morpheus.spectralhead.com/entropy/estimate2/ -P -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 17/07/2015, Ethan Duni ethan.d...@gmail.com wrote: Said another way: if that's the only signal I want to transmit, then I'll just build those parameters into the receiver and not bother transmitting anything. The entropy will be zero. The receiver will simply output the desired waveform without any help from the other end. I argue that's not a transmitter, but rather, a signal generator. In that case, you do not transmit anything, you just have a signal generator that constantly outputs a square wave with constant parameters. Once you build an _actual_ transmitter, that actually transmits something, then the entropy of a message is 100% certain to be bound to be nonzero, since you need to transmit at least a single bit to transmit anything. You cannot transmit nothing ... That's not a transmitter. [Sidenote: assuming your signal generator is a digital one, it will contain bits that represent 1) its code 2) its parameters, both of which have nonzero entropy. The just build those parameters into the receiver step is exactly where that entropy comes from. Consider algorithmic entropy[1] (also called Kolmogorov complexity) - any program that has an output of nonzero length, is bound to have nonzero size, hence nonzero entropy. Therefore, your constant signal generator will have nonzero (algorithmic) entropy. If you want to transmit your constant signal generator program in a message, then the length of the message is bound to be nonzero.] It's only if there's some larger class of signals containing this one that I want to handle, that the question of entropy comes up. Larger class of signals precisely means, at least two signals, since 2 1. Once you have at least two signals that you possibly want to transmit (which is virtuall all cases when you have an actual transmitter and not a constant signal generator), then each message will always have nonzero entropy. Hence, when you have an actual transmitter, the entropy of a message is bound to be nonzero in all cases. [And if you only have a single signal with 100% probability, then your constant signal generator will have nonzero algorithmic entropy, which means the length of the smallest program that produces that output. You cannot produce something from nothing - by hard-coding the parameters into your program, you just hid that entropy inside the program's algorithm or data. If you want to transmit the program or its parameters, the length of the message will be nonzero. Quote from [1]: In Shannon's theory, entropy is a property of a statistical ensemble [...]. Kolmogorov and Chaitin showed independently that entropy is in fact an *intrinsic* property of an object, which can be computed (in theory at least) without reference to anything outside that object. The field that deals with this is called algorithmic information theory, developed around 1965, and these crackpots believe that entropy (information content) is not necessarily a function of a probability distribution. This theory is a half century old, I'm not saying anything new.] -P Reference: [1] http://www.maths.tcd.ie/pub/Maths/Courseware/AlgorithmicEntropy/Entropy.pdf -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 18/07/2015, Peter S peter.schoffhau...@gmail.com wrote: [Sidenote: assuming your signal generator is a digital one, it will contain bits that represent 1) its code 2) its parameters, both of which have nonzero entropy. The just build those parameters into the receiver step is exactly where that entropy comes from. Example: if your 100% probability squarewave that you want to generate is characterized by the set of parameters amplitude=100, phase=30, frequency=1000, and duty cycle=75%, represented by the binary sequence 1 1100100 0 101000 1001011, then if you just build those parameters into the receiver means, that you put those bits (or some other represenations of) into your program. Without your program containing these numbers (in whatever form or encoding), how will it know that it needs to output a square wave with amplitude of 100, phase of 30, frequency of 1000 and duty cycle of 75% ? If any of those parameters are missing, it won't know, so it won't output that exact square wave. Therefore, your program *must* contain these numbers. So all you did, is you put that entropy into your program, and you pretend that there's no entropy. It's still there, except it is now in your program, not in a message. You merely turned it into algorithmic entropy - you cannot make something out of nothing. -P -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
Arbitrary waveform entropy estimate #3 http://morpheus.spectralhead.com/entropy/estimate3/ Results: - For white noise, gives ~1. - For simple periodic waveforms, gives ~0.1. - For mixture of two simple nonharmonic periodic waveforms, gives 0.5. - For increasing length periodic noise, gives increasing estimate. At this point I'm convinced that it is possible to create entropy estimators that work well for arbitrary simple periodic waveforms. For nonharmonic mixtures of periodic waveforms, not as good, but still distinguishes them from noise. -P -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
Consider what happens when you have an actual transmitter/receiver pair, and not just a constant signal generator. There are two possibilities: 1) They use a pre-agreed codebook. 2) The transmitter first sends the codebook to be used to decode the messages. If either of these is not true, then the receiver will have absolutely no clue what to output. It will just receive nessages '0', '001', '101', '1100', '1110', '10', ... If receiver doesn't have a codebook (either pre-agreed or received) to look up what each of these codes mean, then the receiver will think: What to do? I have no idea what these mean, so I don't know what to output. In case 1) when they have a pre-agreed codebook, you simply turned its entropy into algorithmic entropy (you hid that entropy inside your program's data), and the receiver can only reconstruct a certain set of hard-coded parameters (a certain set of hard-coded waveforms). In case 2), transmitter first builds a codebook (for example, based on the probabilities of the messages it is going to send using Huffman coding), and transmits it to the receiver. For example, let's say that there are going to be only 2 different square waves: A) Amplitude=100, phase=30, frequency=1000, duty cycle=75%, represented by binary sequence '1 1100100 0 101000 1001011'. B) Amplitude=50, phase=0, frequency=333, duty cycle=50%, represented by binary sequence '1 110010 0 101001101 110010'. Transmitter decides to send square wave A as bit '0', and send square wave B as bit '1'. Now, in order for them to be able to communicate, transmitter *must* first send this mapping (codebook) to the receiver for the receiver to be able to understand and decode the messages! So the *very* first message will be this: Hey receiver! When I send you '0', that means 'square wave, amplitude=100, phase=30, frequency=1000, duty cycle=75%'. And when I send you '1', that means 'square wave, amplitude=50, phase=0, frequency=333, duty cycle=50%'. End of codebook. Expressed as binary, it may look something like this (encoded as either prefix code, fixed length code or some other encoding): 0 1 1100100 0 101000 1001011 1 1 110010 0 101001101 110010 The encoding of the codebook may be different, but it is *certain* to contain all the parameters for all the possible square waves that will be sent, and this _must_ be the very first message, otherwise the receiver won't know what the received code means! Note, there's no way to encode this further using another encoding, because then for that other encoding, it would need to send another codebook first Hence, the codebook must once be sent 'as-is' or using some prefix encoding that doesn't use a custom code mapping. So, before sending *any* encoded messages, unless the codebook is pre-agreed - the transmitter *must* first send *all* sets of parameters *once*, for the receiver to be able to understand the messages. Irregardless of _whatever_ probability distribution they have. If transmitter wants to send some *very* complex waveform, even if that will be encoded as a short, single '0' bit later on, it must first send the waveform *once* for the receiver to be able to reconstruct it! Otherwise, receiver will not know what to do when receiving the message '0'! So, assuming that the messages are sent by encoding via a codebook, using however short code something may be encoded, unless it is pre-agreed (and thus hard-coded as algorithmic entropy in the program), it *must* be sent at least *once* for the receiver to reconstruct it. Without that, receiver is unable to decode the message. So, if codebook has length of 100 (using whatever encoding), and there are going to be 100 messages, then the *actual* entropy (length) of each message will be increased by 1 bit, since on average, there is 1 extra bit in the code book for each message that is sent, which is absolutely necessary for decoding the messages. Said otherwise, if the average length of a message is H bits and you send N messages, and the codebook has a length of L bits, then the *actual* effective entropy H' of each message will be H' = H + L/N bits, not H bits, because transmitter must also send the codebook first, increasing the overall average bits per message (hence, the actual effective entropy) by L/N bits. Since the length of the codebook L is nonzero unless it's pre-agreed, L/N will be nonzero unless the number of messages to be sent are infinite (so, always). Hence, the encoded parameter set in the cookbook (from whatever probability distribution) will *always* boost the average entropy per message by a *nonzero* amount, its exact amount only determined by the length of the codebook (the size of the parameter set), and the number of messages. (And that is just in addition to the entropy of the messages coming from the probability distribution, which is already nonzero once you have at least two possible messages of nonzero probability. So we have
Re: [music-dsp] about entropy encoding
It follows from the above, that Shannon's entropy model is a simplified, idealized model of information, that pretends that algorithms have zero length and thus no entropy, and you can magically share a codebook telepathically without actually transmitting it. If you dismiss all these details and just pretend they do not exist, then in a highly simplistic, idealized model, you may actually have an entropy of zero in a *single*, special corner case. Even in Shannon's highly simplified model, in virtually all other cases, entropy is nonzero. Later models of information, like the one of Kolmogorov, base the amount of information not on statistical probability, but rather, the minimal length of the algorithm needed to encode that information, which turn out to be near equivalent definitions. Quote from [1]: Firstly, we have to point out the relation between Kolmogorov complexity and Shannon's entropy. Briefly, classic information theory says a random variable X distributed according to P(X=x) has entropy or complexity of a statistical form, where the interpretation is that H(X) bits are on the average sufficient to describe an outcome x. Computational or algorithmic complexity says that an object x has complexity C(x)=the minimum length of a binary program for x. It is very interesting to remark that these two notions turn out to be much the same. Thus, the intended interpretation of complexity C(x) as a measure of the information content of an individual object x is supported by a tight quantitative relationship to Shannon's probabilistic notion. In particular, the entropy H=-SUM P(x)*logP(x) of the distribution p is asymptotically equal to the expected complexity SUM P(x)*C(x). According to [2]: Algorithmic entropy is closely related to statistically defined entropy, the statistical entropy of an ensemble being, for any concisely describable ensemble, very nearly equal to the ensemble average of the algorithmic entropy of its members. According to [3]: Nevertheless, once allowance is made for the units used, the expectation value of the algorithmic entropy for a set of strings belonging to an ensemble is virtually the same as the Shannon entropy, or the Boltzmann or Gibbs entropies derived for a set of states. According to these sources, the algorithmic entropy asymptotically equals, very nearly equals, or virtually the same as the probabilistic entropy. Therefore, any signal that has nonzero length, needs a nonzero-length program to encode, therefore, has nonzero entropy. [1] On Shannon, Fisher, and Algorithmic Entropy in Cognitive Systems http://uni-obuda.hu/conferences/saci04/Crisan.pdf [2] 2003, Niels Henrik Gregersen, From Complexity to Life https://en.wiktionary.org/wiki/algorithmic_entropy [3] 2009, Sean Devine, The Insights of Algorithmic Entropy http://www.mdpi.com/1099-4300/11/1/85/pdf -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
Dear Ethan, You suggested me to be short and concise. My kind recommendation to you: 1) Read A Mathematical Theory of Communication. 2) Try to understand Theorem 2. 3) Try to see, when p_i != 1, then H != 0. I hope this excercise will help you grasp this topic. Best regards, Peter On 17/07/2015, Ethan Duni ethan.d...@gmail.com wrote: Peter S, your combative attitude is unwelcome. It seems that you are less interested in grasping these topics than you are in hectoring myself and other list members. Given that and the dubious topicality of this thread, this will be my last response to you. I hope that you find a healthy way to address the source of your hostility, and also that you gain more insight into Information Theory. My apologies to the list for encouraging this unfortunate tangent. E -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
On 17/07/2015, robert bristow-johnson r...@audioimagination.com wrote: On 7/17/15 1:26 AM, Peter S wrote: On 17/07/2015, robert bristow-johnsonr...@audioimagination.com wrote: in your model, is one sample (from the DSP semantic) the same as a message (from the Information Theory semantic)? A message can be anything - it can be a sample, a bit, a combination of samples or bits, a set of parameters representing a square wave, whatever. doesn't answer my question. It does, it just depends on the model. In the 1-bit square wave duty cycle estimator, a message is a bit. In the English text compression experiment, a message is a character. In the white noise entropy estimation experiment, a message is a byte. In the binary waveform entropy experiment, a message is a string of bits. In the bitflip counter, a message is an event that two consecutive bits differ In the parametric squarewave thought experiment, message is a set of parameteres describing a square wave. Whatever arbitrarily chosen thing I send to you over a channel, becomes a message. There's no universal definition of what a message is, depending on the particular model, it can be literally anything. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] about entropy encoding
A linear predictor[1] tries to predict the next sample as the linear combination of previous samples as x'[n] = SUM [i=1..k] a_i * x[n-i] where x'[n] is the predicted sample, and a_1, a_2 ... a_k are the prediction coefficients (weights). This is often called linear predictive coding (LPC), and one typical application is to encode the formants in telephone telecommunications using source-filter model of speech[2]. Another typical application is waveform encoding in lossless wave compression codecs[3] like Shorten, FLAC, WavPack, MPEG-4 ALS, etc. The simplest possible form of a linear predictor is when a_1 = 1, and all other coefficients a_2, a_3, ... a_n = 0. So it simply predicts the next sample as x'[n] = x[n-1] In other words, it predicts the next sample to be the same as the previous one. This special case of linear prediction is called delta encoding[4], where the next sample is predicted to be the same, and only the difference between two consecutive samples are encoded. One compressed audio file format that uses this type of encoding is the Amiga IFF 8SVX file format[5]. If we simplify this delta encoding further and apply it to the smallest finite field GF(2), then we get: b'[n] = b[n-1] where b[n] is the Nth bit, and b'[n] is the prediction for the Nth bit. In this model, if we try to estimate the prediction error, then - since GF(2) only has two elements: 0 and 1 - the difference (delta) between two consecutive bits can be either 0 or 1. Since the addition operation in GF(2) is XOR, the difference (delta) between two consecutive bits equals A XOR B, that gives 1 for 0-1 or 1-0 transition, and gives zero for two identical bits. To calculate the total prediction error, we simply need to sum the total number of binary transitions in the binary string: E = SUM [i=2..N] ( b[i] XOR b[i-1] ) where N is the total number of bits, and E is the total error between the prediction and the actual samples. So an algorithm that counts bitflips in a binary string, can be thought of as the sum of the errors of the simplest possible linear predictor over GF(2) with a_1 = 1 and a_2, a_3, ... a_n = 0, that simply predicts that the next bit will be the same as the previous bit. By counting the binary transitions, we get the sum of 'deltas', or the sum of 'prediction error' of binary delta coding. If we think of entropy as 'unpredictability' and we expect it to (possibly nonlinearly) correlate with the sum of errors between a prediction and actual samples, then this simplistic linear predictor can be thought of as a very simplistic approximation to entropy. Of course, this is an extremely simplistic and dumb predictor - it is the simplest possible linear predictor that can possibly exist, as GF(2) is the smallest finite field, and it only has a single coefficient a_1 = 1. It cannot account for periodicity, previous bits, statistical probabilities, or other correlations. It can be thought of as an excercise in trying to create the simplest possible predictor. Despite this, and however simple this predictor may seem, by extending this idea further I could successfully create a lossless PCM wave compression algorithm with compression ratios comparable to that of codecs using more complex linear predictors, like FLAC or Shorten. (More on this later.) Best regards, Peter References: [1] https://en.wikipedia.org/wiki/Linear_prediction [2] https://en.wikipedia.org/wiki/Source%E2%80%93filter_model_of_speech_production [3] https://en.wikipedia.org/wiki/Linear_predictive_coding#Applications [4] https://en.wikipedia.org/wiki/Delta_encoding [5] https://en.wikipedia.org/wiki/8SVX [6] https://en.wikipedia.org/wiki/GF%282%29 -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
