In a thread early last month I was doing my thing of "stirring the pot"
by making noise about the equivalence of 'information' and 'uncertainty'
- and I was quoting Shannon to back me up.
We all know that the two concepts are ultimately semantically opposed -
if for no other reason than uncertainty adds to confusion and
information can help to clear it up. So, understandably, Owen - and I
think also Frank - objected somewhat to my equating them. But I was able
to overwhelm the thread with more Shannon quotes, so the thread kinda
tapered off.
What we all were looking for, I believe, is for Information Theory to
back up our common usage and support the notion that information and
uncertainty are, indeed, semantically opposite; while at the same time
they are both measured by the same function: Shannon's version of
entropy (which is also Gibbs' formula with some constants established).
Of course, Shannon /does/ equate them - at least mathematically so, if
not semantically so. Within the span of three sentences in his famous
1948 paper, he uses the words "information", "uncertainty" and "choice"
to describe what his concept of entropy measures. But he never does get
into any semantic distinctions among the three - only that all three
measured by the same formula.
Even contemporary information theorists like Vlatko Vedral, Professor of
Quantum Information Science at Oxford, appear to be of no help with any
distinction between 'information' and 'uncertainty'. In his 2010 book
_Decoding Reality: the universe as quantum information_, he traces the
notion of /information/ back to the ancient Greeks.
"The ancient Greeks laid the foundation for its (information)
development when they suggested that the information content of an
event somehow depends only on how probable this event really is.
Philosophers like Aristotle reasoned that the more surprised we are
by an event the more information the event carries....
Following this logic, we conclude that information has to be
inversely proportional to probability, i. e. events with smaller
probability carry more information...."
But it was the Russian probability theorist A. I. Khinchin who provides
us the satisfaction we seek. Seeing that the Shannon paper (bless his
soul) lacked both mathematical rigor and satisfying semantic
justifications, he set about to set the situation right with his slim
but essential little volume entitled _The Mathematical Foundations of
Information Theory_ (1957). He manages to make the pertinent distinction
between 'information' and 'uncertainty' most cleanly in this single
paragraph. (By "scheme" Khinchin means "probability distribution".)
"Thus we can say that the information given us by carrying out some
experiment consists of removing the uncertainty which existed before
the experiment. The larger this uncertainty, the larger we consider
to be the amount of information obtained by removing it. Since we
agreed to measure the uncertainty of a finite scheme A by its
entropy, H(A), it is natural to express the amount of information
given by removing this uncertainty by an increasing function of the
quantity H(A)....
Thus, in all that follows, we can consider the amount of information
given by the realization of a finite scheme to be equal to the
entropy of the scheme."
On 6/6/11 8:17 AM, Owen Densmore wrote:
Nick: Next you are in town, lets read the original Shannon paper together.
Alas, it is a bit long, but I'm told its a Good Thing To Do.
-- Owen
On Jun 6, 2011, at 7:44 AM, Nicholas Thompson wrote:
Grant,
This seems backwards to me, but I got properly thrashed for my last few
postings so I am putting my hat over the wall very carefully here.
I thought……i thought …. the information in a message was the number of bits by
which the arrival of the message decreased the uncertainty of the receiver.
So, let’s say you are sitting awaiting the result of a coin toss, and I am on
the other end of the line flipping the coin. Before I say “heads” you have 1
bit of uncertainty; afterwards, you have none.
The reason I am particularly nervous about saying this is that it, of course,
holds out the possibility of negative information. Some forms of
communication, appeasement gestures in animals, for instance, have the effect
of increasing the range of behaviors likely to occur in the receiver. This
would seem to correspond to a negative value for the information calculation.
Nick
From: friam-boun...@redfish.com [mailto:friam-boun...@redfish.com] On Behalf Of
Grant Holland
Sent: Sunday, June 05, 2011 11:07 PM
To: The Friday Morning Applied Complexity Coffee Group; Steve Smith
Subject: Re: [FRIAM] Quote of the week
Interesting note on "information" and "uncertainty"...
Information is Uncertainty. The two words are synonyms.
Shannon called it "uncertainty", contemporary Information theory calls it
"information".
It is often thought that the more information there is, the less uncertainty.
The opposite is the case.
In Information Theory (aka the mathematical theory of communications) , the
degree of information I(E) - or uncertainty U(E) - of an event is measurable as
an inverse function of its probability, as follows:
U(E) = I(E) = log( 1/Pr(E) ) = log(1) - log( Pr(E) ) = -log( Pr(E) ).
Considering I(E) as a random variable, Shannon's entropy is, in fact, the first
moment (or expectation) of I(E). Shannon entropy = exp( I(E) ).
Grant
On 6/5/2011 2:20 PM, Steve Smith wrote:
"Philosophy is to physics as pornography is to sex. It's cheaper, it's easier and
some people seem to prefer it."
Modern Physics is contained in Realism which is contained in Metaphysics which
I contained in all of Philosophy.
I'd be tempted to counter:
"Physics is to Philosophy as the Missionary Position is to the Kama Sutra"
Physics also appeals to Phenomenology and Logic (the branch of Philosophy were
Mathematics is rooted) and what we can know scientifically is constrained by
Epistemology (the nature of knowledge) and phenomenology (the nature of
conscious experience).
It might be fair to say that many (including many of us here) who hold Physics
up in some exalted position simply dismiss or choose to ignore all the messy
questions considered by *the rest of* philosophy. Even if we think we have
clear/simple answers to the questions, I do not accept that the questions are
not worthy of the asking.
The underlying point of the referenced podcast is, in fact, that Physics, or
Science in general might be rather myopic and limited by it's own viewpoint by
definition.
"The more we know, the less we understand."
Philosophy is about understanding, physics is about knowledge first and
understanding only insomuch as it is a part of natural philosophy.
Or at least this is how my understanding is structured around these matters.
- Steve
On Sun, Jun 5, 2011 at 1:15 PM, Robert Holmes<rob...@holmesacosta.com> wrote:
> From the BBC's science podcast "The Infinite Monkey Cage":
"Philosophy is to physics as pornography is to sex. It's cheaper, it's easier and
some people seem to prefer it."
Not to be pedantic, but I suspect that s/he has conflated "philosophy" with "new
age", as much of science owes itself to philosophy.
marcos
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org