On 7/17/15 2:28 AM, Peter S wrote:
Dear Ethan,
You suggested me to be short and concise.
My kind recommendation to you:
1) Read "A Mathematical Theory of Communication".
2) Try to understand Theorem 2.
3) Try to see, when p_i != 1, then H != 0.
I hope this excercise will help you grasp this topic.
listen, one thing i have to remind myself here (because if i don't, i'm
gonna get embarrassed) is to not underestimate either the level of
scholarship nor the level of practical experience doing things related
to music (or at least audio) and DSP. just a friendly caution to
consider the same.
On 7/17/15 3:27 PM, Peter S wrote:
A linear predictor[1] tries to "predict" the next sample as the linear
combination of previous samples as
x'[n] = SUM [i=1..k] a_i * x[n-i]
where x'[n] is the predicted sample, and a_1, a_2 ... a_k are the
prediction coefficients (weights). This is often called linear
predictive coding (LPC),
listen, i understand LPC. i also understand (i remember having multiple
email exchanges with Claude Cellier about the topic in the '90s) how to
use LPC to reduce the word size (if the LPC is effective, the LPC guess
for the following sample is, statistically-speaking, usually close to
the actual sample, so the word size is smaller and the probability on
each of those reduced-width words is higher) followed by Huffman coding
on those often used delta values. if you have a non-flat spectrum, LPC
will make it flat, and if you have a non-uniform p.d.f., Huffman coding
will similarly take advantage of that. you can combine the two and
tweek it to the point that the losslessly compressed file can't get much
more compressed without adding more to the model (like periodicity, so
that the LPC doesn't just use the most current N samples to predict the
future sample, but also uses a bunch of samples that are P samples
prior, where P is the best-fit period for a quasi-periodic input). i
know how to do that (and done MATLAB implementations, but never anything
that saw the commercial light-of-day).
i also know how i might do it for music that is the combination of a few
sources, if i get good pitch-detection data from a source separation
alg, which is another big, difficult problem.
i know how to do this stuff. i know how to teach this stuff and in a
previous lifetime have done it. (i like to use a recurring character of
George Carlin: Al Sleet, the Hippy-Dippy Weatherman, to illustrate
Shannon's use of message probability, or frequency-of-occurance as a
natural measure of the true information content a message has.)
so be careful, lest you lose people's interest, to restrain any
patronizing tone. we're not as dumb as you think.
On 7/17/15 1:38 PM, Peter S wrote:
On 17/07/2015, robert bristow-johnson<r...@audioimagination.com> wrote:
On 7/17/15 1:26 AM, Peter S wrote:
On 17/07/2015, robert bristow-johnson<r...@audioimagination.com> wrote:
in your model, is one sample (from the DSP semantic) the same as a
"message" (from the Information Theory semantic)?
A "message" can be anything - it can be a sample, a bit, a combination
of samples or bits, a set of parameters representing a square wave,
whatever.
doesn't answer my question.
It does, it just depends on the model.
In the 1-bit square wave duty cycle estimator, a message is a "bit".
In the English text compression experiment, a message is "a character".
In the white noise entropy estimation experiment, a message is a "byte".
In the binary waveform entropy experiment, a message is "a string of bits".
In the bitflip counter, a message is an event that "two consecutive bits differ"
In the parametric squarewave thought experiment, message is "a set of
parameteres describing a square wave".
Whatever arbitrarily chosen thing I send to you over a channel,
becomes a "message". There's no universal definition of what a
"message" is, depending on the particular model, it can be literally
anything.
so here's my question(s) ... please read carefully because i'm the guy
who gets to decide if my question really got answered, and i am still
pretty much clueless about it:
what is the atom of information that you are counting to create a histogram?
maybe it's sophisticated, first you count every 1-bit message and create
a two-column histogram.
then rewind the soundfile and count every 2-bit message, grouped
adjacently and add that to the histogram.
then rewind the soundfile again and count each 3-bit message, and add
that to the histogram.
continue to some defined word-width limit.
i can sorta see a generic measure of entropy doing that, but it assumes
messages that are adjacent bits. you can add to that messages that are
grouped together in any imagined fashion, not just adjacent bits.
there are virtually uncountable ways of grouping together the bits of a
binary file to measure entropy. i simply cannot believe you did it
every-which-way.
so, because i am too lazy to look around at previous posts, what is the
*specific* way or ways you grouped together the bits into messages so
that you could count their frequency of occurrence and get a measure of
mean bits of information per message?
that question you certainly did not answer to my satisfaction.
--
r b-j r...@audioimagination.com
"Imagination is more important than knowledge."
--
dupswapdrop -- the music-dsp mailing list and website:
subscription info, FAQ, source code archive, list archive, book reviews, dsp
links
http://music.columbia.edu/cmc/music-dsp
http://music.columbia.edu/mailman/listinfo/music-dsp
