On Fri, Nov 28, 2014 at 11:54 PM, Logan Streondj <[email protected]> wrote: > On Thu, Nov 27, 2014 at 10:35:03PM -0500, Matt Mahoney wrote: >> >> > You >> >> > are able to understand my words because you can predict a large >> >> > fraction of them and only need to remember the differences. >> > >> > that just sounds like a nonsensical statement. >> >> If you disagree, then try taking some of my words and scrambling them >> in random order and see which sentence is easier to remember. Then try >> scrambling the letters in random order. Do you see that the task >> becomes progressively harder because you are not able to predict the >> next word or next letter? > > er if words are in random order, they aren't grammatical, > if letters are in random order, they aren't vocabulary. > without grammar and vocabulary, there is no langauge.
Again we are disagreeing on the semantics of the word "predict". Semantics, grammar, and vocabulary are sets of rules that we use to assign probabilities to strings of text such as sentences. If I scramble the words or letters, then they violate these rules and the probability is reduced. To understand how this makes prediction harder, you can use the chain rule, which states that for any string xy (x concatenated with y): P(xy) = P(x) P(y|x) and you can likewise split x and y into smaller strings and repeat, which gives you: P(x_1..n) = product over i in 1..n of P(x_i | x_1..i-1). Thus, the probability of any sentence can be expressed as the product of the conditional probabilities of each word, letter, or bit, given the parts of the sentence you have already seen. Since we usually input the words or letters sequentially over time, then recognizing a sentence as correct is the same as assigning it a high probability, which is the same as predicting it with a high rate of accuracy. There is one more point: prediction implies understanding. If I want to test if you understand something I say, I would ask you to repeat it back to me. Obviously you can't do that if you can't remember it. Your short term memory capacity is about 100 bits, or enough to repeat back a list of about 7 random words. But you can remember sentences longer than 7 words if they obey language rules because you don't have to remember the whole thing. You only have to remember the difference between what I said and what you predicted I would say. Typical written English has an entropy of about 1 bit per character relative to your language model. This allows you to repeat back well formed sentences up to about 20 words. -- -- Matt Mahoney, [email protected] ------------------------------------------- AGI Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/21088071-f452e424 Modify Your Subscription: https://www.listbox.com/member/?member_id=21088071&id_secret=21088071-58d57657 Powered by Listbox: http://www.listbox.com
