Dear Riz, I will leave the math out, and use a very simplistic example.
Think of the information of a signal as how much data you have to transmit to describe the signal perfectly to someone. A 100x100 black and white image contains 10000 bits, but how much information is in it? If the image is heavily structured (e.g. a checker board) how much information does it contain? I could transmit 10000 bits (i.e. 01010101...). Or I could transmit the English statement. "It is a checkerboard pattern starting with a 0", which is much shorter, but describes the image just as perfectly. Thus I don't need 10000 bits to describe that image. The amount of information that it contains is very small. However I wouldn't be able to describe perfectly a random image other than by describing individual bits, thus there is more information in that one. - ----- Of course in practice you wouldn't use English as a form of data compression, and we haven't defined precisely which 100x100 binary images are possible, or how likely each is. The example above is mathematically informal on purpose but I hope it helps to build the right intuition. Jorge Jorge Moraleda Stanford University > On Mon, 28 Oct 2002, Rizwan Choudrey wrote: > > > Dear all, > > > > I wondered if anyone could help with a paradox at the heart of my > > understanding of entropy, information and pattern recognition. > > > > I understand an informative signals as one which contains patterns, as > > opposed to radomly distributed numbers e.g. noise. Therefore, I > > equate information with structure in the signals distribution. However, > > Shannon equates information with entropy, which is maximimum when each > > symbol in the signal is equally as likely as the next i.e. a distribution > > with no `structure'. These views are contradictory. > > > > What am I misisng in my understanding? > > > > Many thanks in advance, > > Riz > > > > > > Rizwan Choudrey > > Robotics Group > > Department of Engineering Science > > University of Oxford > > 07956 455380 > > > > > > Bert Kappen SNN University of Nijmegen > tel: +31 24 3614241 fax: +31 24 3541435 > URL: www.snn.kun.nl/~bert >
