Richard Loosemore <[EMAIL PROTECTED]> wrote:
>"Understanding" 10^9 bits of information is not the same as storing 10^9 
>bits of information.

That is true.  "Understanding" n bits is the same as compressing some larger 
training set that has an algorithmic complexity of n bits.  Once you have done 
this, you can use your probability model to make predictions about unseen data 
generated by the same (unknown) Turing machine as the training data.  The 
closer to n you can compress, the better your predictions will be.

I am not sure what it means to "understand" a painting, but let's say that you 
understand art if you can identify the artists of paintings you haven't seen 
before with better accuracy than random guessing.  The relevant quantity of 
information is not the number of pixels and resolution, which depend on the 
limits of the eye, but the (much smaller) number of features that the high 
level perceptual centers of the brain are capable of distinguishing and storing 
in memory.  (Experiments by Standing and Landauer suggest it is a few bits per 
second for long term memory, the same rate as language).  Then you guess the 
shortest program that generates a list of feature-artist pairs consistent with 
your knowledge of art and use it to predict artists given new features.

My estimate of 10^9 bits for a language model is based on 4 lines of evidence, 
one of which is the amount of language you process in a lifetime.  This is a 
rough estimate of course.  I estimate 1 GB (8 x 10^9 bits) compressed to 1 bpc 
(Shannon) and assume you remember a significant fraction of that.

Landauer, Tom (1986), “How much do people
remember?  Some estimates of the quantity
of learned information in long term memory”, Cognitive Science (10) pp. 477-493

Shannon, Cluade E. (1950), “Prediction and
Entropy of Printed English”, Bell Sys. Tech. J (3) p. 50-64.  

Standing, L. (1973), “Learning 10,000 Pictures”,
Quarterly Journal of Experimental Psychology (25) pp. 207-222.



-- Matt Mahoney, [EMAIL PROTECTED]

----- Original Message ----
From: Richard Loosemore <[EMAIL PROTECTED]>
To: agi@v2.listbox.com
Sent: Wednesday, November 15, 2006 9:33:04 AM
Subject: Re: [agi] A question on the symbol-system hypothesis

Matt Mahoney wrote:
> I will try to answer several posts here. I said that the knowledge
> base of an AGI must be opaque because it has 10^9 bits of information,
> which is more than a person can comprehend. By opaque, I mean that you
> can't do any better by examining or modifying the internal
> representation than you could by examining or modifying the training
> data. For a text based AI with natural language ability, the 10^9 bits
> of training data would be about a gigabyte of text, about 1000 books. Of
> course you can sample it, add to it, edit it, search it, run various
> tests on it, and so on. What you can't do is read, write, or know all of
> it. There is no internal representation that you could convert it to
> that would allow you to do these things, because you still have 10^9
> bits of information. It is a limitation of the human brain that it can't
> store more information than this.

"Understanding" 10^9 bits of information is not the same as storing 10^9 
bits of information.

A typical painting in the Louvre might be 1 meter on a side.  At roughly 
16 pixels per millimeter, and a perceivable color depth of about 20 bits 
that would be about 10^8 bits.  If an art specialist knew all about, 
say, 1000 paintings in the Louvre, that specialist would "understand" a 
total of about 10^11 bits.

You might be inclined to say that not all of those bits count, that many 
are redundant to "understanding".

Exactly.

People can easily comprehend 10^9 bits.  It makes no sense to argue 
about degree of comprehension by quoting numbers of bits.


Richard Loosemore

-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?list_id=303



-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?list_id=303

Reply via email to