RE: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[John G. Rose]

Note:

 

Theorem 1.7.1 There eRectively exists a universal computer.

 

If you copy and paste this declaration the "ff" gets replaced with a circle
cap R :)

 

Not sure how this shows up...

 

John

 

From: Ben Goertzel [mailto:b...@goertzel.org] 
Sent: Friday, July 09, 2010 8:50 AM
To: agi
Subject: Re: [agi] Solomonoff Induction is Not "Universal" and Probability
is not "Prediction"

 


To make this discussion more concrete, please look at

http://www.vetta.org/documents/disSol.pdf 

Section 2.5 gives a simple version of the proof that Solomonoff induction is
a powerful learning algorithm in principle, and Section 2.6 explains why it
is not practically useful.

What part of that paper do you think is wrong?

thx
ben



On Fri, Jul 9, 2010 at 9:54 AM, Jim Bromer  wrote:

On Fri, Jul 9, 2010 at 7:56 AM, Ben Goertzel  wrote:

If you're going to argue against a mathematical theorem, your argument must
be mathematical not verbal.  Please explain one of

1) which step in the proof about Solomonoff induction's effectiveness you
believe is in error

2) which of the assumptions of this proof you think is inapplicable to real
intelligence [apart from the assumption of infinite or massive compute
resources]



 

Solomonoff Induction is not a provable Theorem, it is therefore a
conjecture.  It cannot be computed, it cannot be verified.  There are many
mathematical theorems that require the use of limits to "prove" them for
example, and I accept those proofs.  (Some people might not.)  But there is
no evidence that Solmonoff Induction would tend toward some limits.  Now
maybe the conjectured abstraction can be verified through some other means,
but I have yet to see an adequate explanation of that in any terms.  The
idea that I have to answer your challenges using only the terms you specify
is noise.

 

Look at 2.  What does that say about your "Theorem".

 

I am working on 1 but I just said: "I haven't yet been able to find a way
that could be used to prove that Solomonoff Induction does not do what Matt
claims it does."

  Z

What is not clear is that no one has objected to my characterization of the
conjecture as I have been able to work it out for myself.  It requires an
infinite set of infinitely computed probabilities of each infinite "string".
If this characterization is correct, then Matt has been using the term
"string" ambiguously.  As a primary sample space: A particular string.  And
as a compound sample space: All the possible individual cases of the
substring compounded into one.  No one has yet to tell of his "mathematical"
experiments of using a Turing simulator to see what a finite iteration of
all possible programs of a given length would actually look like.

 

I will finish this later.

 

 

On Fri, Jul 9, 2010 at 7:49 AM, Jim Bromer  wrote:

Abram,

Solomoff Induction would produce poor "predictions" if it could be used to
compute them.  


Solomonoff induction is a mathematical, not verbal, construct.  Based on the
most obvious mapping from the verbal terms you've used above into
mathematical definitions in terms of which Solomonoff induction is
constructed, the above statement of yours is FALSE.

If you're going to argue against a mathematical theorem, your argument must
be mathematical not verbal.  Please explain one of

1) which step in the proof about Solomonoff induction's effectiveness you
believe is in error

2) which of the assumptions of this proof you think is inapplicable to real
intelligence [apart from the assumption of infinite or massive compute
resources]

Otherwise, your statement is in the same category as the statement by the
protagonist of Dostoesvky's "Notes from the Underground" --

"I admit that two times two makes four is an excellent thing, but if we are
to give everything its due, two times two makes five is sometimes a very
charming thing too."

;-)

 

Secondly, since it cannot be computed it is useless.  Third, it is not the
sort of thing that is useful for AGI in the first place.


I agree with these two statements

-- ben G 

 


agi |  <https://www.listbox.com/member/archive/303/=now> Archives
<https://www.listbox.com/member/archive/rss/303/> | Modify Your Subscription


 <https://www.listbox.com/member/archive/rss/303/> 

 <https://www.listbox.com/member/archive/rss/303/>  


 <https://www.listbox.com/member/archive/rss/303/> agi | Archives | Modify
Your Subscription

 <https://www.listbox.com/member/archive/rss/303/> 

 <https://www.listbox.com/member/archive/rss/303/> 


-- 
Ben Goertzel, PhD
CEO, Novamente LLC and Biomind LLC
CTO, Genescient Corp
Vice Chairman, Humanity+
Advisor, Singularity University and Singularity Institute
External Research Professor, Xiamen University, China
b...@goertzel.org

" 
"When nothin

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

 I guess the Godel Theorem is called a theorem, so Solomonoff Induction
would be called a theorem.  I believe that Solomonoff Induction is
computable, but the claims that are made for it are not provable because
there is no way you could prove that it approaches a stable limit (stable
limits).  You can't prove that it does just because the sense of "all
possible programs" is so ill-defined that there is not enough to go
on.  Whether my outline of a disproof could actually be used to find an
adequate disproof, I don't know.  My attempt to disprove it may just be an
unprovable theorem (or even wrong).

Jim Bromer



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

Solomonoff Induction is not a mathematical conjecture.  We can talk about a
function which is based on "all mathematical functions," but since we cannot
define that as a mathematical function it is not a realizable function.



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

On Fri, Jul 9, 2010 at 1:12 PM, Jim Bromer  wrote:
The proof is based on the diagonal argument of Cantor, but it might be
considered as variation of Cantor's diagonal argument.  There can be no one
to one *mapping of the computation to an usage* as the computation
approaches infinity to make the values approach some limit of precision. For
any computed values there is always a *possibility* (this is different than
Cantor) that there are an infinite number of more precise values (of the
probability of a string (primary sample space or compound sample space))
within any two iterations of the computational program (formula).

Ok, I didn't get that right, but there is enough there to get the idea.
For any computed values there is always a *possibility* (I think this is
different than Cantor) that there are an infinite number of more precise
values (of the probability of a string (primary sample space or compound
sample space)) that may fall outside the limits that could be derived from
any finite sequence of iterations of the computational program (formula).

On Fri, Jul 9, 2010 at 1:12 PM, Jim Bromer  wrote:

>  On Fri, Jul 9, 2010 at 11:37 AM, Ben Goertzel  wrote:
>
>>
>> I don't think Solomonoff induction is a particularly useful direction for
>> AI, I was just taking issue with the statement made that it is not capable
>> of correct prediction given adequate resources...
>
>
> Pi is not computable.  It would take infinite resources to compute it.
> However, because Pi approaches a limit, the theory of limits can be used to
> show that it can be refined to any limit that is possible and since it
> consistently approaches a limit it can be used in general theorems that can
> be proven through induction.  You can use *computed values* of pi in a
> general theorem as long as you can show that the usage is valid by using the
> theory of limits.
>
> I think I figured out a way, given infinite resources, to write a program
> that could "compute" Solomonoff Induction.  However, since it cannot be
> shown (or at least I don't know anyone who has ever shown) that the
> probabilities approaches some value (or values) as a limit (or limits), this
> program (or a variation on this kind of program) could not be used to show
> that it can be:
> 1. computed to any specified degree of precision within some finite number
> of steps.
> 2. proven through the use of mathematical induction.
>
> The proof is based on the diagonal argument of Cantor, but it might be
> considered as variation of Cantor's diagonal argument.  There can be no one
> to one *mapping of the computation to an usage* as the computation
> approaches infinity to make the values approach some limit of precision. For
> any computed values there is always a *possibility* (this is different
> than Cantor) that there are an infinite number of more precise values (of
> the probability of a string (primary sample space or compound sample space))
> within any two iterations of the computational program (formula).
>
> So even though I cannot disprove what Solomonoff Induction might be given
> infinite resources, if this superficial analysis is right, without a way to
> compute the values so that they tend toward a limit for each of the
> probabilities needed, it is not a usable mathematical theorem.
>
> What uncomputable means is that any statement (most statements) drawn from
> it are matters of mathematical conjecture or opinion.  It's like opinioning
> that the Godel sentence, given infinite resources, is decidable.
>
> I don't think the question of whether it is valid for infinite resources or
> not can be answered mathematically for the time being.  And conclusions
> drawn from uncomputable results have to be considered dubious.
>
> However, it certainly leads to other questions which I think are more
> interesting and more useful.
>
> What is needed to promote greater insight about the problem of conditional
> probabilities in complicated situations where the probability emitters
> and the elementary sample space may be obscured by the use of complicated
> interactions and a preliminary focus on compound sample spaces?  Are there
> theories, which like asking questions about the givens in a problem, that
> could lead toward a greater detection of the relation between the givens and
> the primary probability emitters and the primary sample space?
>
> Can a mathematical theory be based solely on abstract principles even
> though it is impossible to evaluate the use of those abstractions with
> examples from the particulars (of the abstractions)?  How could those
> abstract principles be reliably defined so that they aren't too simplistic?
>
> Jim Bromer
>



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.co

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

On Fri, Jul 9, 2010 at 11:37 AM, Ben Goertzel  wrote:

>
> I don't think Solomonoff induction is a particularly useful direction for
> AI, I was just taking issue with the statement made that it is not capable
> of correct prediction given adequate resources...


Pi is not computable.  It would take infinite resources to compute it.
However, because Pi approaches a limit, the theory of limits can be used to
show that it can be refined to any limit that is possible and since it
consistently approaches a limit it can be used in general theorems that can
be proven through induction.  You can use *computed values* of pi in a
general theorem as long as you can show that the usage is valid by using the
theory of limits.

I think I figured out a way, given infinite resources, to write a program
that could "compute" Solomonoff Induction.  However, since it cannot be
shown (or at least I don't know anyone who has ever shown) that the
probabilities approaches some value (or values) as a limit (or limits), this
program (or a variation on this kind of program) could not be used to show
that it can be:
1. computed to any specified degree of precision within some finite number
of steps.
2. proven through the use of mathematical induction.

The proof is based on the diagonal argument of Cantor, but it might be
considered as variation of Cantor's diagonal argument.  There can be no one
to one *mapping of the computation to an usage* as the computation
approaches infinity to make the values approach some limit of precision. For
any computed values there is always a *possibility* (this is different than
Cantor) that there are an infinite number of more precise values (of the
probability of a string (primary sample space or compound sample space))
within any two iterations of the computational program (formula).

So even though I cannot disprove what Solomonoff Induction might be given
infinite resources, if this superficial analysis is right, without a way to
compute the values so that they tend toward a limit for each of the
probabilities needed, it is not a usable mathematical theorem.

What uncomputable means is that any statement (most statements) drawn from
it are matters of mathematical conjecture or opinion.  It's like opinioning
that the Godel sentence, given infinite resources, is decidable.

I don't think the question of whether it is valid for infinite resources or
not can be answered mathematically for the time being.  And conclusions
drawn from uncomputable results have to be considered dubious.

However, it certainly leads to other questions which I think are more
interesting and more useful.

What is needed to promote greater insight about the problem of conditional
probabilities in complicated situations where the probability emitters
and the elementary sample space may be obscured by the use of complicated
interactions and a preliminary focus on compound sample spaces?  Are there
theories, which like asking questions about the givens in a problem, that
could lead toward a greater detection of the relation between the givens and
the primary probability emitters and the primary sample space?

Can a mathematical theory be based solely on abstract principles even though
it is impossible to evaluate the use of those abstractions with examples
from the particulars (of the abstractions)?  How could those abstract
principles be reliably defined so that they aren't too simplistic?

Jim Bromer



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Ben Goertzel]

I don't think Solomonoff induction is a particularly useful direction for
AI, I was just taking issue with the statement made that it is not capable
of correct prediction given adequate resources...

On Fri, Jul 9, 2010 at 11:35 AM, David Jones  wrote:

> Although I haven't studied Solomonoff induction yet, although I plan to
> read up on it, I've realized that people seem to be making the same mistake
> I was. People are trying to find one silver bullet method of induction or
> learning that works for everything. I've begun to realize that its OK if
> something doesn't work for everything. As long as it works on a large enough
> subset of problems to be useful. If you can figure out how to construct
> justifiable methods of induction for enough problems that you need to solve,
> then that is sufficient for AGI.
>
> This is the same mistake I made and it was the point I was trying to make
> in the recent email I sent. I kept trying to come up with algorithms for
> doing things and I could always find a test case to break it. So, now I've
> begun to realize that it's ok if it breaks sometimes! The question is, can
> you define an algorithm that breaks gracefully and which can figure out what
> problems it can be applied to and what problems it should not be applied to.
> If you can do that, then you can solve the problems where it is applicable,
> and avoid the problems where it is not.
>
> This is perfectly OK! You don't have to find a silver bullet method of
> induction or inference that works for everything!
>
> Dave
>
>
>
> On Fri, Jul 9, 2010 at 10:49 AM, Ben Goertzel  wrote:
>
>>
>> To make this discussion more concrete, please look at
>>
>> http://www.vetta.org/documents/disSol.pdf
>>
>> Section 2.5 gives a simple version of the proof that Solomonoff induction
>> is a powerful learning algorithm in principle, and Section 2.6 explains why
>> it is not practically useful.
>>
>> What part of that paper do you think is wrong?
>>
>> thx
>> ben
>>
>>
>>
>> On Fri, Jul 9, 2010 at 9:54 AM, Jim Bromer  wrote:
>>
>>>  On Fri, Jul 9, 2010 at 7:56 AM, Ben Goertzel  wrote:
>>>
>>> If you're going to argue against a mathematical theorem, your argument
>>> must be mathematical not verbal.  Please explain one of
>>>
>>> 1) which step in the proof about Solomonoff induction's effectiveness you
>>> believe is in error
>>>
>>> 2) which of the assumptions of this proof you think is inapplicable to
>>> real intelligence [apart from the assumption of infinite or massive compute
>>> resources]
>>>  
>>>
>>> Solomonoff Induction is not a provable Theorem, it is therefore a
>>> conjecture.  It cannot be computed, it cannot be verified.  There are many
>>> mathematical theorems that require the use of limits to "prove" them for
>>> example, and I accept those proofs.  (Some people might not.)  But there is
>>> no evidence that Solmonoff Induction would tend toward some limits.  Now
>>> maybe the conjectured abstraction can be verified through some other means,
>>> but I have yet to see an adequate explanation of that in any terms.  The
>>> idea that I have to answer your challenges using only the terms you specify
>>> is noise.
>>>
>>> Look at 2.  What does that say about your "Theorem".
>>>
>>> I am working on 1 but I just said: "I haven't yet been able to find a way
>>> that could be used to prove that Solomonoff Induction does not do what Matt
>>> claims it does."
>>>   Z
>>> What is not clear is that no one has objected to my characterization of
>>> the conjecture as I have been able to work it out for myself.  It requires
>>> an infinite set of infinitely computed probabilities of each infinite
>>> "string".  If this characterization is correct, then Matt has been using the
>>> term "string" ambiguously.  As a primary sample space: A particular string.
>>> And as a compound sample space: All the possible individual cases of the
>>> substring compounded into one.  No one has yet to tell of his "mathematical"
>>> experiments of using a Turing simulator to see what a finite iteration of
>>> all possible programs of a given length would actually look like.
>>>
>>> I will finish this later.
>>>
>>>


  On Fri, Jul 9, 2010 at 7:49 AM, Jim Bromer wrote:

> Abram,
> Solomoff Induction would produce poor "predictions" if it could be used
> to compute them.
>

 Solomonoff induction is a mathematical, not verbal, construct.  Based on
 the most obvious mapping from the verbal terms you've used above into
 mathematical definitions in terms of which Solomonoff induction is
 constructed, the above statement of yours is FALSE.

 If you're going to argue against a mathematical theorem, your argument
 must be mathematical not verbal.  Please explain one of

 1) which step in the proof about Solomonoff induction's effectiveness
 you believe is in error

 2) which of the assumptions of this proof you think is inapplicable to
>

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[David Jones]

The same goes for inference. There is no silver bullet method that is
completely general and can infer anything. There is no general inference
method. Sometimes it works, sometimes it doesn't. That is the nature of the
complex world we live in. My current theory is that the more we try to find
a single silver bullet, the more we will just break against the fact that
none exists.



On Fri, Jul 9, 2010 at 11:35 AM, David Jones  wrote:

> Although I haven't studied Solomonoff induction yet, although I plan to
> read up on it, I've realized that people seem to be making the same mistake
> I was. People are trying to find one silver bullet method of induction or
> learning that works for everything. I've begun to realize that its OK if
> something doesn't work for everything. As long as it works on a large enough
> subset of problems to be useful. If you can figure out how to construct
> justifiable methods of induction for enough problems that you need to solve,
> then that is sufficient for AGI.
>
> This is the same mistake I made and it was the point I was trying to make
> in the recent email I sent. I kept trying to come up with algorithms for
> doing things and I could always find a test case to break it. So, now I've
> begun to realize that it's ok if it breaks sometimes! The question is, can
> you define an algorithm that breaks gracefully and which can figure out what
> problems it can be applied to and what problems it should not be applied to.
> If you can do that, then you can solve the problems where it is applicable,
> and avoid the problems where it is not.
>
> This is perfectly OK! You don't have to find a silver bullet method of
> induction or inference that works for everything!
>
> Dave
>
>
>
> On Fri, Jul 9, 2010 at 10:49 AM, Ben Goertzel  wrote:
>
>>
>> To make this discussion more concrete, please look at
>>
>> http://www.vetta.org/documents/disSol.pdf
>>
>> Section 2.5 gives a simple version of the proof that Solomonoff induction
>> is a powerful learning algorithm in principle, and Section 2.6 explains why
>> it is not practically useful.
>>
>> What part of that paper do you think is wrong?
>>
>> thx
>> ben
>>
>>
>>
>> On Fri, Jul 9, 2010 at 9:54 AM, Jim Bromer  wrote:
>>
>>>  On Fri, Jul 9, 2010 at 7:56 AM, Ben Goertzel  wrote:
>>>
>>> If you're going to argue against a mathematical theorem, your argument
>>> must be mathematical not verbal.  Please explain one of
>>>
>>> 1) which step in the proof about Solomonoff induction's effectiveness you
>>> believe is in error
>>>
>>> 2) which of the assumptions of this proof you think is inapplicable to
>>> real intelligence [apart from the assumption of infinite or massive compute
>>> resources]
>>> 
>>>
>>> Solomonoff Induction is not a provable Theorem, it is therefore a
>>> conjecture.  It cannot be computed, it cannot be verified.  There are many
>>> mathematical theorems that require the use of limits to "prove" them for
>>> example, and I accept those proofs.  (Some people might not.)  But there is
>>> no evidence that Solmonoff Induction would tend toward some limits.  Now
>>> maybe the conjectured abstraction can be verified through some other means,
>>> but I have yet to see an adequate explanation of that in any terms.  The
>>> idea that I have to answer your challenges using only the terms you specify
>>> is noise.
>>>
>>> Look at 2.  What does that say about your "Theorem".
>>>
>>> I am working on 1 but I just said: "I haven't yet been able to find a way
>>> that could be used to prove that Solomonoff Induction does not do what Matt
>>> claims it does."
>>>   Z
>>> What is not clear is that no one has objected to my characterization of
>>> the conjecture as I have been able to work it out for myself.  It requires
>>> an infinite set of infinitely computed probabilities of each infinite
>>> "string".  If this characterization is correct, then Matt has been using the
>>> term "string" ambiguously.  As a primary sample space: A particular string.
>>> And as a compound sample space: All the possible individual cases of the
>>> substring compounded into one.  No one has yet to tell of his "mathematical"
>>> experiments of using a Turing simulator to see what a finite iteration of
>>> all possible programs of a given length would actually look like.
>>>
>>> I will finish this later.
>>>
>>>


  On Fri, Jul 9, 2010 at 7:49 AM, Jim Bromer wrote:

> Abram,
> Solomoff Induction would produce poor "predictions" if it could be used
> to compute them.
>

 Solomonoff induction is a mathematical, not verbal, construct.  Based on
 the most obvious mapping from the verbal terms you've used above into
 mathematical definitions in terms of which Solomonoff induction is
 constructed, the above statement of yours is FALSE.

 If you're going to argue against a mathematical theorem, your argument
 must be mathematical not verbal.  Please explain one of

>>

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[David Jones]

Although I haven't studied Solomonoff induction yet, although I plan to read
up on it, I've realized that people seem to be making the same mistake I
was. People are trying to find one silver bullet method of induction or
learning that works for everything. I've begun to realize that its OK if
something doesn't work for everything. As long as it works on a large enough
subset of problems to be useful. If you can figure out how to construct
justifiable methods of induction for enough problems that you need to solve,
then that is sufficient for AGI.

This is the same mistake I made and it was the point I was trying to make in
the recent email I sent. I kept trying to come up with algorithms for doing
things and I could always find a test case to break it. So, now I've begun
to realize that it's ok if it breaks sometimes! The question is, can you
define an algorithm that breaks gracefully and which can figure out what
problems it can be applied to and what problems it should not be applied to.
If you can do that, then you can solve the problems where it is applicable,
and avoid the problems where it is not.

This is perfectly OK! You don't have to find a silver bullet method of
induction or inference that works for everything!

Dave



On Fri, Jul 9, 2010 at 10:49 AM, Ben Goertzel  wrote:

>
> To make this discussion more concrete, please look at
>
> http://www.vetta.org/documents/disSol.pdf
>
> Section 2.5 gives a simple version of the proof that Solomonoff induction
> is a powerful learning algorithm in principle, and Section 2.6 explains why
> it is not practically useful.
>
> What part of that paper do you think is wrong?
>
> thx
> ben
>
>
>
> On Fri, Jul 9, 2010 at 9:54 AM, Jim Bromer  wrote:
>
>>  On Fri, Jul 9, 2010 at 7:56 AM, Ben Goertzel  wrote:
>>
>> If you're going to argue against a mathematical theorem, your argument
>> must be mathematical not verbal.  Please explain one of
>>
>> 1) which step in the proof about Solomonoff induction's effectiveness you
>> believe is in error
>>
>> 2) which of the assumptions of this proof you think is inapplicable to
>> real intelligence [apart from the assumption of infinite or massive compute
>> resources]
>> 
>>
>> Solomonoff Induction is not a provable Theorem, it is therefore a
>> conjecture.  It cannot be computed, it cannot be verified.  There are many
>> mathematical theorems that require the use of limits to "prove" them for
>> example, and I accept those proofs.  (Some people might not.)  But there is
>> no evidence that Solmonoff Induction would tend toward some limits.  Now
>> maybe the conjectured abstraction can be verified through some other means,
>> but I have yet to see an adequate explanation of that in any terms.  The
>> idea that I have to answer your challenges using only the terms you specify
>> is noise.
>>
>> Look at 2.  What does that say about your "Theorem".
>>
>> I am working on 1 but I just said: "I haven't yet been able to find a way
>> that could be used to prove that Solomonoff Induction does not do what Matt
>> claims it does."
>>   Z
>> What is not clear is that no one has objected to my characterization of
>> the conjecture as I have been able to work it out for myself.  It requires
>> an infinite set of infinitely computed probabilities of each infinite
>> "string".  If this characterization is correct, then Matt has been using the
>> term "string" ambiguously.  As a primary sample space: A particular string.
>> And as a compound sample space: All the possible individual cases of the
>> substring compounded into one.  No one has yet to tell of his "mathematical"
>> experiments of using a Turing simulator to see what a finite iteration of
>> all possible programs of a given length would actually look like.
>>
>> I will finish this later.
>>
>>
>>>
>>>
>>>  On Fri, Jul 9, 2010 at 7:49 AM, Jim Bromer  wrote:
>>>
 Abram,
 Solomoff Induction would produce poor "predictions" if it could be used
 to compute them.

>>>
>>> Solomonoff induction is a mathematical, not verbal, construct.  Based on
>>> the most obvious mapping from the verbal terms you've used above into
>>> mathematical definitions in terms of which Solomonoff induction is
>>> constructed, the above statement of yours is FALSE.
>>>
>>> If you're going to argue against a mathematical theorem, your argument
>>> must be mathematical not verbal.  Please explain one of
>>>
>>> 1) which step in the proof about Solomonoff induction's effectiveness you
>>> believe is in error
>>>
>>> 2) which of the assumptions of this proof you think is inapplicable to
>>> real intelligence [apart from the assumption of infinite or massive compute
>>> resources]
>>>
>>> Otherwise, your statement is in the same category as the statement by the
>>> protagonist of Dostoesvky's "Notes from the Underground" --
>>>
>>> "I admit that two times two makes four is an excellent thing, but if we
>>> are to give everything its due, two times two makes five i

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Ben Goertzel]

To make this discussion more concrete, please look at

http://www.vetta.org/documents/disSol.pdf

Section 2.5 gives a simple version of the proof that Solomonoff induction is
a powerful learning algorithm in principle, and Section 2.6 explains why it
is not practically useful.

What part of that paper do you think is wrong?

thx
ben


On Fri, Jul 9, 2010 at 9:54 AM, Jim Bromer  wrote:

> On Fri, Jul 9, 2010 at 7:56 AM, Ben Goertzel  wrote:
>
> If you're going to argue against a mathematical theorem, your argument must
> be mathematical not verbal.  Please explain one of
>
> 1) which step in the proof about Solomonoff induction's effectiveness you
> believe is in error
>
> 2) which of the assumptions of this proof you think is inapplicable to real
> intelligence [apart from the assumption of infinite or massive compute
> resources]
> 
>
> Solomonoff Induction is not a provable Theorem, it is therefore a
> conjecture.  It cannot be computed, it cannot be verified.  There are many
> mathematical theorems that require the use of limits to "prove" them for
> example, and I accept those proofs.  (Some people might not.)  But there is
> no evidence that Solmonoff Induction would tend toward some limits.  Now
> maybe the conjectured abstraction can be verified through some other means,
> but I have yet to see an adequate explanation of that in any terms.  The
> idea that I have to answer your challenges using only the terms you specify
> is noise.
>
> Look at 2.  What does that say about your "Theorem".
>
> I am working on 1 but I just said: "I haven't yet been able to find a way
> that could be used to prove that Solomonoff Induction does not do what Matt
> claims it does."
>   Z
> What is not clear is that no one has objected to my characterization of
> the conjecture as I have been able to work it out for myself.  It requires
> an infinite set of infinitely computed probabilities of each infinite
> "string".  If this characterization is correct, then Matt has been using the
> term "string" ambiguously.  As a primary sample space: A particular string.
> And as a compound sample space: All the possible individual cases of the
> substring compounded into one.  No one has yet to tell of his "mathematical"
> experiments of using a Turing simulator to see what a finite iteration of
> all possible programs of a given length would actually look like.
>
> I will finish this later.
>
>
>>
>>
>>  On Fri, Jul 9, 2010 at 7:49 AM, Jim Bromer  wrote:
>>
>>> Abram,
>>> Solomoff Induction would produce poor "predictions" if it could be used
>>> to compute them.
>>>
>>
>> Solomonoff induction is a mathematical, not verbal, construct.  Based on
>> the most obvious mapping from the verbal terms you've used above into
>> mathematical definitions in terms of which Solomonoff induction is
>> constructed, the above statement of yours is FALSE.
>>
>> If you're going to argue against a mathematical theorem, your argument
>> must be mathematical not verbal.  Please explain one of
>>
>> 1) which step in the proof about Solomonoff induction's effectiveness you
>> believe is in error
>>
>> 2) which of the assumptions of this proof you think is inapplicable to
>> real intelligence [apart from the assumption of infinite or massive compute
>> resources]
>>
>> Otherwise, your statement is in the same category as the statement by the
>> protagonist of Dostoesvky's "Notes from the Underground" --
>>
>> "I admit that two times two makes four is an excellent thing, but if we
>> are to give everything its due, two times two makes five is sometimes a very
>> charming thing too."
>>
>> ;-)
>>
>>
>>
>>> Secondly, since it cannot be computed it is useless.  Third, it is not
>>> the sort of thing that is useful for AGI in the first place.
>>>
>>
>> I agree with these two statements
>>
>> -- ben G
>>
>>   *agi* | Archives 
>>  | 
>> ModifyYour Subscription
>> 
>>
>
>*agi* | Archives 
>  | 
> ModifyYour Subscription
> 
>



-- 
Ben Goertzel, PhD
CEO, Novamente LLC and Biomind LLC
CTO, Genescient Corp
Vice Chairman, Humanity+
Advisor, Singularity University and Singularity Institute
External Research Professor, Xiamen University, China
b...@goertzel.org

"
“When nothing seems to help, I go look at a stonecutter hammering away at
his rock, perhaps a hundred times without as much as a crack showing in it.
Yet at the hundred and first blow it will split in two, and I know it was
not that blow that did it, but all that had gone before.”



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Su

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

On Fri, Jul 9, 2010 at 7:56 AM, Ben Goertzel  wrote:

If you're going to argue against a mathematical theorem, your argument must
be mathematical not verbal.  Please explain one of

1) which step in the proof about Solomonoff induction's effectiveness you
believe is in error

2) which of the assumptions of this proof you think is inapplicable to real
intelligence [apart from the assumption of infinite or massive compute
resources]


Solomonoff Induction is not a provable Theorem, it is therefore a
conjecture.  It cannot be computed, it cannot be verified.  There are many
mathematical theorems that require the use of limits to "prove" them for
example, and I accept those proofs.  (Some people might not.)  But there is
no evidence that Solmonoff Induction would tend toward some limits.  Now
maybe the conjectured abstraction can be verified through some other means,
but I have yet to see an adequate explanation of that in any terms.  The
idea that I have to answer your challenges using only the terms you specify
is noise.

Look at 2.  What does that say about your "Theorem".

I am working on 1 but I just said: "I haven't yet been able to find a way
that could be used to prove that Solomonoff Induction does not do what Matt
claims it does."
  Z
What is not clear is that no one has objected to my characterization of
the conjecture as I have been able to work it out for myself.  It requires
an infinite set of infinitely computed probabilities of each infinite
"string".  If this characterization is correct, then Matt has been using the
term "string" ambiguously.  As a primary sample space: A particular string.
And as a compound sample space: All the possible individual cases of the
substring compounded into one.  No one has yet to tell of his "mathematical"
experiments of using a Turing simulator to see what a finite iteration of
all possible programs of a given length would actually look like.

I will finish this later.


>
>
>  On Fri, Jul 9, 2010 at 7:49 AM, Jim Bromer  wrote:
>
>> Abram,
>> Solomoff Induction would produce poor "predictions" if it could be used to
>> compute them.
>>
>
> Solomonoff induction is a mathematical, not verbal, construct.  Based on
> the most obvious mapping from the verbal terms you've used above into
> mathematical definitions in terms of which Solomonoff induction is
> constructed, the above statement of yours is FALSE.
>
> If you're going to argue against a mathematical theorem, your argument must
> be mathematical not verbal.  Please explain one of
>
> 1) which step in the proof about Solomonoff induction's effectiveness you
> believe is in error
>
> 2) which of the assumptions of this proof you think is inapplicable to real
> intelligence [apart from the assumption of infinite or massive compute
> resources]
>
> Otherwise, your statement is in the same category as the statement by the
> protagonist of Dostoesvky's "Notes from the Underground" --
>
> "I admit that two times two makes four is an excellent thing, but if we are
> to give everything its due, two times two makes five is sometimes a very
> charming thing too."
>
> ;-)
>
>
>
>> Secondly, since it cannot be computed it is useless.  Third, it is not the
>> sort of thing that is useful for AGI in the first place.
>>
>
> I agree with these two statements
>
> -- ben G
>
>   *agi* | Archives 
>  | 
> ModifyYour Subscription
> 
>



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Ben Goertzel]

On Fri, Jul 9, 2010 at 8:38 AM, Matt Mahoney  wrote:

> Ben Goertzel wrote:
>> > Secondly, since it cannot be computed it is useless.  Third, it is not
>> the sort of thing that is useful for AGI in the first place.
>>
>
> > I agree with these two statements
>
> The principle of Solomonoff induction can be applied to computable subsets
> of the (infinite) hypothesis space. For example, if you are using a neural
> network to make predictions, the principle says to use the smallest network
> that computes the past training data.
>


Yes, of course various versions of Occam's Razor are useful in practice, and
we use an Occam bias in MOSES inside OpenCog for example  But as you
know, these are not exactly the same as Solomonoff Induction, though they're
based on the same idea...

-- Ben



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Matt Mahoney]

Ben Goertzel wrote:
>> Secondly, since it cannot be computed it is useless.  Third, it is not the 
>> sort 
>>of thing that is useful for AGI in the first place.

> I agree with these two statements

The principle of Solomonoff induction can be applied to computable subsets of 
the (infinite) hypothesis space. For example, if you are using a neural network 
to make predictions, the principle says to use the smallest network that 
computes the past training data.
 -- Matt Mahoney, matmaho...@yahoo.com





From: Ben Goertzel 
To: agi 
Sent: Fri, July 9, 2010 7:56:53 AM
Subject: Re: [agi] Solomonoff Induction is Not "Universal" and Probability is 
not "Prediction"




On Fri, Jul 9, 2010 at 7:49 AM, Jim Bromer  wrote:

Abram,
>Solomoff Induction would produce poor "predictions" if it could be used to 
>compute them.  
>

Solomonoff induction is a mathematical, not verbal, construct.  Based on the 
most obvious mapping from the verbal terms you've used above into mathematical 
definitions in terms of which Solomonoff induction is constructed, the above 
statement of yours is FALSE.

If you're going to argue against a mathematical theorem, your argument must be 
mathematical not verbal.  Please explain one of

1) which step in the proof about Solomonoff induction's effectiveness you 
believe is in error

2) which of the assumptions of this proof you think is inapplicable to real 
intelligence [apart from the assumption of infinite or massive compute 
resources]

Otherwise, your statement is in the same category as the statement by the 
protagonist of Dostoesvky's "Notes from the Underground" --

"I admit that two times two makes four is an excellent thing, but if we are to 
give everything its due, two times two makes five is sometimes a very charming 
thing too."

;-)

 
Secondly, since it cannot be computed it is useless.  Third, it is not the sort 
of thing that is useful for AGI in the first place.

I agree with these two statements

-- ben G 


agi | Archives  | Modify Your Subscription  


---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Ben Goertzel]

On Fri, Jul 9, 2010 at 7:49 AM, Jim Bromer  wrote:

> Abram,
> Solomoff Induction would produce poor "predictions" if it could be used to
> compute them.
>

Solomonoff induction is a mathematical, not verbal, construct.  Based on the
most obvious mapping from the verbal terms you've used above into
mathematical definitions in terms of which Solomonoff induction is
constructed, the above statement of yours is FALSE.

If you're going to argue against a mathematical theorem, your argument must
be mathematical not verbal.  Please explain one of

1) which step in the proof about Solomonoff induction's effectiveness you
believe is in error

2) which of the assumptions of this proof you think is inapplicable to real
intelligence [apart from the assumption of infinite or massive compute
resources]

Otherwise, your statement is in the same category as the statement by the
protagonist of Dostoesvky's "Notes from the Underground" --

"I admit that two times two makes four is an excellent thing, but if we are
to give everything its due, two times two makes five is sometimes a very
charming thing too."

;-)



> Secondly, since it cannot be computed it is useless.  Third, it is not the
> sort of thing that is useful for AGI in the first place.
>

I agree with these two statements

-- ben G



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

Abram,
Solomoff Induction would produce poor "predictions" if it could be used to
compute them.  Secondly, since it cannot be computed it is useless.  Third,
it is not the sort of thing that is useful for AGI in the first place.

You could experiment with finite possible ways to produce a string and see
how useful the idea is, both as an abstraction and as an actual AGI tool.
Have you tried this?  An example is a word program that complete a word as
you are typing.

As far as Matt's complaint.  I haven't yet been able to find a way that
could be used to prove that Solomonoff Induction does not do what Matt
claims it does, but I have yet to see an explanation of a proof that it
does.  When you are dealing with unverifiable pseudo-abstractions you are
dealing with something that cannot be proven.  All we can work on is whether
or not the idea seems to make sense as an abstraction.

As I said, the starting point would be to develop simpler problems and see
how they behave as you build up more complicated problems.

Jim

On Thu, Jul 8, 2010 at 5:15 PM, Abram Demski  wrote:

> Yes, Jim, you seem to be mixing arguments here. I cannot tell which of the
> following you intend:
>
> 1) Solomonoff induction is useless because it would produce very bad
> predictions if we could compute them.
> 2) Solomonoff induction is useless because we can't compute its
> predictions.
>
> Are you trying to reject #1 and assert #2, reject #2 and assert #1, or
> assert both #1 and #2?
>
> Or some third statement?
>
> --Abram
>
>
> On Wed, Jul 7, 2010 at 7:09 PM, Matt Mahoney  wrote:
>
>>   Who is talking about efficiency? An infinite sequence of uncomputable
>> values is still just as uncomputable. I don't disagree that AIXI and
>> Solomonoff induction are not computable. But you are also arguing that they
>> are wrong.
>>
>>
>> -- Matt Mahoney, matmaho...@yahoo.com
>>
>>
>>  --------------
>> *From:* Jim Bromer 
>> *To:* agi 
>> *Sent:* Wed, July 7, 2010 6:40:52 PM
>> *Subject:* Re: [agi] Solomonoff Induction is Not "Universal" and
>> Probability is not "Prediction"
>>
>> Matt,
>> But you are still saying that Solomonoff Induction has to be recomputed
>> for each possible combination of bit value aren't you?  Although this
>> doesn't matter when you are dealing with infinite computations in the first
>> place, it does matter when you are wondering if this has anything to do with
>> AGI and compression efficiencies.
>> Jim Bromer
>> On Wed, Jul 7, 2010 at 5:44 PM, Matt Mahoney wrote:
>>
>>>Jim Bromer wrote:
>>> > But, a more interesting question is, given that the first digits are
>>> 000, what are the chances that the next digit will be 1?  Dim Induction will
>>> report .5, which of course is nonsense and a whole less useful than making a
>>> rough guess.
>>>
>>> Wrong. The probability of a 1 is p(0001)/(p()+p(0001)) where the
>>> probabilities are computed using Solomonoff induction. A program that
>>> outputs  will be shorter in most languages than a program that outputs
>>> 0001, so 0 is the most likely next bit.
>>>
>>> More generally, probability and prediction are equivalent by the chain
>>> rule. Given any 2 strings x followed by y, the prediction p(y|x) =
>>> p(xy)/p(x).
>>>
>>>
>>> -- Matt Mahoney, matmaho...@yahoo.com
>>>
>>>
>>>  --
>>> *From:* Jim Bromer 
>>> *To:* agi 
>>> *Sent:* Wed, July 7, 2010 10:10:37 AM
>>> *Subject:* [agi] Solomonoff Induction is Not "Universal" and Probability
>>> is not "Prediction"
>>>
>>> Suppose you have sets of "programs" that produce two strings.  One set of
>>> outputs is 00 and the other is 11. Now suppose you used these sets
>>> of programs to chart the probabilities of the output of the strings.  If the
>>> two strings were each output by the same number of programs then you'd have
>>> a .5 probability that either string would be output.  That's ok.  But, a
>>> more interesting question is, given that the first digits are 000, what are
>>> the chances that the next digit will be 1?  Dim Induction will report .5,
>>> which of course is nonsense and a whole less useful than making a rough
>>> guess.
>>>
>>> But, of course, Solomonoff Induction purports to be able, if it was
>>> feasible, to compute the possibilities for all possible programs.  Ok, but
>>> now, try thinking 

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Abram Demski]

Yes, Jim, you seem to be mixing arguments here. I cannot tell which of the
following you intend:

1) Solomonoff induction is useless because it would produce very bad
predictions if we could compute them.
2) Solomonoff induction is useless because we can't compute its predictions.

Are you trying to reject #1 and assert #2, reject #2 and assert #1, or
assert both #1 and #2?

Or some third statement?

--Abram

On Wed, Jul 7, 2010 at 7:09 PM, Matt Mahoney  wrote:

> Who is talking about efficiency? An infinite sequence of uncomputable
> values is still just as uncomputable. I don't disagree that AIXI and
> Solomonoff induction are not computable. But you are also arguing that they
> are wrong.
>
>
> -- Matt Mahoney, matmaho...@yahoo.com
>
>
> --
> *From:* Jim Bromer 
> *To:* agi 
> *Sent:* Wed, July 7, 2010 6:40:52 PM
> *Subject:* Re: [agi] Solomonoff Induction is Not "Universal" and
> Probability is not "Prediction"
>
> Matt,
> But you are still saying that Solomonoff Induction has to be recomputed for
> each possible combination of bit value aren't you?  Although this doesn't
> matter when you are dealing with infinite computations in the first place,
> it does matter when you are wondering if this has anything to do with AGI
> and compression efficiencies.
> Jim Bromer
> On Wed, Jul 7, 2010 at 5:44 PM, Matt Mahoney  wrote:
>
>>Jim Bromer wrote:
>> > But, a more interesting question is, given that the first digits are
>> 000, what are the chances that the next digit will be 1?  Dim Induction will
>> report .5, which of course is nonsense and a whole less useful than making a
>> rough guess.
>>
>> Wrong. The probability of a 1 is p(0001)/(p()+p(0001)) where the
>> probabilities are computed using Solomonoff induction. A program that
>> outputs  will be shorter in most languages than a program that outputs
>> 0001, so 0 is the most likely next bit.
>>
>> More generally, probability and prediction are equivalent by the chain
>> rule. Given any 2 strings x followed by y, the prediction p(y|x) =
>> p(xy)/p(x).
>>
>>
>> -- Matt Mahoney, matmaho...@yahoo.com
>>
>>
>>  --
>> *From:* Jim Bromer 
>> *To:* agi 
>> *Sent:* Wed, July 7, 2010 10:10:37 AM
>> *Subject:* [agi] Solomonoff Induction is Not "Universal" and Probability
>> is not "Prediction"
>>
>> Suppose you have sets of "programs" that produce two strings.  One set of
>> outputs is 00 and the other is 11. Now suppose you used these sets
>> of programs to chart the probabilities of the output of the strings.  If the
>> two strings were each output by the same number of programs then you'd have
>> a .5 probability that either string would be output.  That's ok.  But, a
>> more interesting question is, given that the first digits are 000, what are
>> the chances that the next digit will be 1?  Dim Induction will report .5,
>> which of course is nonsense and a whole less useful than making a rough
>> guess.
>>
>> But, of course, Solomonoff Induction purports to be able, if it was
>> feasible, to compute the possibilities for all possible programs.  Ok, but
>> now, try thinking about this a little bit.  If you have ever tried writing
>> random program instructions what do you usually get?  Well, I'll take a
>> hazard and guess (a lot better than the bogus method of confusing shallow
>> probability with "prediction" in my example) and say that you will get a lot
>> of programs that crash.  Well, most of my experiments with that have ended
>> up with programs that go into an infinite loop or which crash.  Now on a
>> universal Turing machine, the results would probably look a little
>> different.  Some strings will output nothing and go into an infinite loop.
>> Some programs will output something and then either stop outputting anything
>> or start outputting an infinite loop of the same substring.  Other programs
>> will go on to infinity producing something that looks like random strings.
>> But the idea that all possible programs would produce well distributed
>> strings is complete hogwash.  Since Solomonoff Induction does not define
>> what kind of programs should be used, the assumption that the distribution
>> would produce useful data is absurd.  In particular, the use of the method
>> to determine the probability based given an initial string (as in what
>> follows given the first digits are 000) is wrong as in really wrong.  The
>> idea that this crude probability can be used

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Matt Mahoney]

Who is talking about efficiency? An infinite sequence of uncomputable values is 
still just as uncomputable. I don't disagree that AIXI and Solomonoff induction 
are not computable. But you are also arguing that they are wrong.

 -- Matt Mahoney, matmaho...@yahoo.com





From: Jim Bromer 
To: agi 
Sent: Wed, July 7, 2010 6:40:52 PM
Subject: Re: [agi] Solomonoff Induction is Not "Universal" and Probability is 
not "Prediction"


Matt,
But you are still saying that Solomonoff Induction has to be recomputed for 
each 
possible combination of bit value aren't you?  Although this doesn't matter 
when 
you are dealing with infinite computations in the first place, it does matter 
when you are wondering if this has anything to do with AGI and compression 
efficiencies.
Jim Bromer

On Wed, Jul 7, 2010 at 5:44 PM, Matt Mahoney  wrote:

 Jim Bromer wrote:
>> But, a more interesting question is, given that the first digits are 000, 
>> what 
>>are the chances that the next digit will be 1?  Dim Induction will report .5, 
>>which of course is nonsense and a whole less useful than making a rough guess.
>
>
>Wrong. The probability of a 1 is p(0001)/(p()+p(0001)) where the 
>probabilities are computed using Solomonoff induction. A program that outputs 
> will be shorter in most languages than a program that outputs 0001, so 0 
>is 
>the most likely next bit.
>
>
>More generally, probability and prediction are equivalent by the chain rule. 
>Given any 2 strings x followed by y, the prediction p(y|x) = p(xy)/p(x).
>
> -- Matt Mahoney, matmaho...@yahoo.com 
>
>
>
>
>
________________
 From: Jim Bromer 
>To: agi 
>Sent: Wed, July 7, 2010 10:10:37 AM
>Subject: [agi] Solomonoff Induction is Not "Universal" and Probability is not 
>"Prediction"
> 
>
>
>Suppose you have sets of "programs" that produce two strings.  One set of 
>outputs is 00 and the other is 11. Now suppose you used these sets of 
>programs to chart the probabilities of the output of the strings.  If the two 
>strings were each output by the same number of programs then you'd have a .5 
>probability that either string would be output.  That's ok.  But, a more 
>interesting question is, given that the first digits are 000, what are the 
>chances that the next digit will be 1?  Dim Induction will report .5, which of 
>course is nonsense and a whole less useful than making a rough guess.
> 
>But, of course, Solomonoff Induction purports to be able, if it was feasible, 
>to 
>compute the possibilities for all possible programs.  Ok, but now, try 
>thinking 
>about this a little bit.  If you have ever tried writing random program 
>instructions what do you usually get?  Well, I'll take a hazard and guess (a 
>lot 
>better than the bogus method of confusing shallow probability with 
>"prediction" 
>in my example) and say that you will get a lot of programs that crash.  Well, 
>most of my experiments with that have ended up with programs that go into an 
>infinite loop or which crash.  Now on a universal Turing machine, the results 
>would probably look a little different.  Some strings will output nothing and 
>go 
>into an infinite loop.  Some programs will output something and then either 
>stop 
>outputting anything or start outputting an infinite loop of the same 
>substring.  
>Other programs will go on to infinity producing something that looks like 
>random 
>strings.  But the idea that all possible programs would produce well 
>distributed 
>strings is complete hogwash.  Since Solomonoff Induction does not define what 
>kind of programs should be used, the assumption that the distribution would 
>produce useful data is absurd.  In particular, the use of the method to 
>determine the probability based given an initial string (as in what follows 
>given the first digits are 000) is wrong as in really wrong.  The idea that 
>this 
>crude probability can be used as "prediction" is unsophisticated.
> 
>Of course you could develop an infinite set of Solomonoff Induction values for 
>each possible given initial sequence of digits.  Hey when you're working with 
>infeasible functions why not dream anything?
> 
>I might be wrong of course.  Maybe there is something you guys haven't been 
>able 
>to get across to me.  Even if you can think for yourself you can still make 
>mistakes.  So if anyone has actually tried writing a program to output all 
>possible programs (up to some feasible point) on a Turing Machine simulator, 
>let 
>me know how it went.
> 
>Jim Bromer
> 
>agi | Archives  | Modify Your Subscription  
>agi | Archiv

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

Matt,
But you are still saying that Solomonoff Induction has to be recomputed for
each possible combination of bit value aren't you?  Although this doesn't
matter when you are dealing with infinite computations in the first place,
it does matter when you are wondering if this has anything to do with AGI
and compression efficiencies.
Jim Bromer
On Wed, Jul 7, 2010 at 5:44 PM, Matt Mahoney  wrote:

>Jim Bromer wrote:
> > But, a more interesting question is, given that the first digits are 000,
> what are the chances that the next digit will be 1?  Dim Induction will
> report .5, which of course is nonsense and a whole less useful than making a
> rough guess.
>
> Wrong. The probability of a 1 is p(0001)/(p()+p(0001)) where the
> probabilities are computed using Solomonoff induction. A program that
> outputs  will be shorter in most languages than a program that outputs
> 0001, so 0 is the most likely next bit.
>
> More generally, probability and prediction are equivalent by the chain
> rule. Given any 2 strings x followed by y, the prediction p(y|x) =
> p(xy)/p(x).
>
>
> -- Matt Mahoney, matmaho...@yahoo.com
>
>
>  --
> *From:* Jim Bromer 
> *To:* agi 
> *Sent:* Wed, July 7, 2010 10:10:37 AM
> *Subject:* [agi] Solomonoff Induction is Not "Universal" and Probability
> is not "Prediction"
>
> Suppose you have sets of "programs" that produce two strings.  One set of
> outputs is 00 and the other is 11. Now suppose you used these sets
> of programs to chart the probabilities of the output of the strings.  If the
> two strings were each output by the same number of programs then you'd have
> a .5 probability that either string would be output.  That's ok.  But, a
> more interesting question is, given that the first digits are 000, what are
> the chances that the next digit will be 1?  Dim Induction will report .5,
> which of course is nonsense and a whole less useful than making a rough
> guess.
>
> But, of course, Solomonoff Induction purports to be able, if it was
> feasible, to compute the possibilities for all possible programs.  Ok, but
> now, try thinking about this a little bit.  If you have ever tried writing
> random program instructions what do you usually get?  Well, I'll take a
> hazard and guess (a lot better than the bogus method of confusing shallow
> probability with "prediction" in my example) and say that you will get a lot
> of programs that crash.  Well, most of my experiments with that have ended
> up with programs that go into an infinite loop or which crash.  Now on a
> universal Turing machine, the results would probably look a little
> different.  Some strings will output nothing and go into an infinite loop.
> Some programs will output something and then either stop outputting anything
> or start outputting an infinite loop of the same substring.  Other programs
> will go on to infinity producing something that looks like random strings.
> But the idea that all possible programs would produce well distributed
> strings is complete hogwash.  Since Solomonoff Induction does not define
> what kind of programs should be used, the assumption that the distribution
> would produce useful data is absurd.  In particular, the use of the method
> to determine the probability based given an initial string (as in what
> follows given the first digits are 000) is wrong as in really wrong.  The
> idea that this crude probability can be used as "prediction" is
> unsophisticated.
>
> Of course you could develop an infinite set of Solomonoff Induction values
> for each possible given initial sequence of digits.  Hey when you're working
> with infeasible functions why not dream anything?
>
> I might be wrong of course.  Maybe there is something you guys
> haven't been able to get across to me.  Even if you can think for yourself
> you can still make mistakes.  So if anyone has actually tried writing a
> program to output all possible programs (up to some feasible point) on a
> Turing Machine simulator, let me know how it went.
>
> Jim Bromer
>
>*agi* | Archives <https://www.listbox.com/member/archive/303/=now>
> <https://www.listbox.com/member/archive/rss/303/> | 
> Modify<https://www.listbox.com/member/?&;>Your Subscription
> <http://www.listbox.com/>
>*agi* | Archives <https://www.listbox.com/member/archive/303/=now>
> <https://www.listbox.com/member/archive/rss/303/> | 
> Modify<https://www.listbox.com/member/?&;>Your Subscription
> <http://www.listbox.com/>
>



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Matt Mahoney]

 Jim Bromer wrote:
> But, a more interesting question is, given that the first digits are 000, 
> what 
>are the chances that the next digit will be 1?  Dim Induction will report .5, 
>which of course is nonsense and a whole less useful than making a rough guess.

Wrong. The probability of a 1 is p(0001)/(p()+p(0001)) where the 
probabilities are computed using Solomonoff induction. A program that outputs 
 will be shorter in most languages than a program that outputs 0001, so 0 
is 
the most likely next bit.

More generally, probability and prediction are equivalent by the chain rule. 
Given any 2 strings x followed by y, the prediction p(y|x) = p(xy)/p(x).

 -- Matt Mahoney, matmaho...@yahoo.com





From: Jim Bromer 
To: agi 
Sent: Wed, July 7, 2010 10:10:37 AM
Subject: [agi] Solomonoff Induction is Not "Universal" and Probability is not 
"Prediction"


Suppose you have sets of "programs" that produce two strings.  One set of 
outputs is 00 and the other is 11. Now suppose you used these sets of 
programs to chart the probabilities of the output of the strings.  If the two 
strings were each output by the same number of programs then you'd have a .5 
probability that either string would be output.  That's ok.  But, a more 
interesting question is, given that the first digits are 000, what are the 
chances that the next digit will be 1?  Dim Induction will report .5, which of 
course is nonsense and a whole less useful than making a rough guess.
 
But, of course, Solomonoff Induction purports to be able, if it was feasible, 
to 
compute the possibilities for all possible programs.  Ok, but now, try thinking 
about this a little bit.  If you have ever tried writing random program 
instructions what do you usually get?  Well, I'll take a hazard and guess (a 
lot 
better than the bogus method of confusing shallow probability with "prediction" 
in my example) and say that you will get a lot of programs that crash.  Well, 
most of my experiments with that have ended up with programs that go into an 
infinite loop or which crash.  Now on a universal Turing machine, the results 
would probably look a little different.  Some strings will output nothing and 
go 
into an infinite loop.  Some programs will output something and then either 
stop 
outputting anything or start outputting an infinite loop of the same substring. 
 
Other programs will go on to infinity producing something that looks like 
random 
strings.  But the idea that all possible programs would produce well 
distributed 
strings is complete hogwash.  Since Solomonoff Induction does not define what 
kind of programs should be used, the assumption that the distribution would 
produce useful data is absurd.  In particular, the use of the method to 
determine the probability based given an initial string (as in what follows 
given the first digits are 000) is wrong as in really wrong.  The idea that 
this 
crude probability can be used as "prediction" is unsophisticated.
 
Of course you could develop an infinite set of Solomonoff Induction values for 
each possible given initial sequence of digits.  Hey when you're working with 
infeasible functions why not dream anything?
 
I might be wrong of course.  Maybe there is something you guys haven't been 
able 
to get across to me.  Even if you can think for yourself you can still make 
mistakes.  So if anyone has actually tried writing a program to output all 
possible programs (up to some feasible point) on a Turing Machine simulator, 
let 
me know how it went.
 
Jim Bromer
 
agi | Archives  | Modify Your Subscription  


---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

Abram,
I don't think you are right.  The reason is that Solomonoff Induction does
not produce a true universal probability for any given first digits.  To do
so it would have to be capable of representing the probability of any
(computable)  sequence that follows any (computable) string of given first
digits.

Yes, if a high proportion of programs produce 00, it will be able to
register that as string as more probable, but the information on what the
next digits will be, given some input, will not be represented in anything
that resembled "compression".  For instance, if you had 62 bits and wanted
to know what the probability of the next two bits were, you would have to
have done the infinite calculations of a Solomonoff Induction for each of
the 2^62 possible combination of bits that represented the possible input to
your problem.

I might be wrong, but I don't see where all this is "information" is being
hidden if I am.  On the other hand, if I am right (or even partially right)
I don't understand why seemingly smart people are excited about this as a
possible AGI method.

We in AGI specifically want to know the answer to the kind of question:
Given some partially defined situation, how could a computer best figure out
what is going on.  Most computer situations are going to be represented by
kilobytes or megabytes these days, not in strings of 32 bits or less.  If
there was an abstraction that could help us think about these things, it
could help even if the ideal would be way beyond any feasible technology.
And there is an abstraction like this that can help us.  Applied
probability.  We can think about these ideas in the terms of strings if we
want to but the key is that WE have to work out the details because we see
the problems differently.  There is nothing that I have seen in Solomonoff
Induction that suggests that this is an adequate or even useful method to
use.  On the other hand I would not be talking about this if it weren't for
Solomonoff so maybe I just don't share your enthusiasm.  If I have
misunderstood something then all I can say is that I am still waiting for
someone to explain it in a way that I can understand.

Jim

On Wed, Jul 7, 2010 at 1:58 PM, Abram Demski  wrote:

> Jim,
>
> I am unable to find the actual objection to Solomonoff in what you wrote
> (save for that it's "wrong as in really wrong").
>
> It's true that a lot of programs won't produce any output. That just means
> they won't alter the prediction.
>
> It's also true that a lot of programs will produce random-looking or
> boring-looking output. This just means that Solomonoff will have some
> expectation of those things. To use your example, given 000, the chances
> that the next digit will be 0 will be fairly high thanks to boring programs
> which just output lots of zeros. (Not sure why you mention the idea that it
> might be .5? This sounds like "no induction" rather than "dim induction"...)
>
> --Abram
>
>   On Wed, Jul 7, 2010 at 10:10 AM, Jim Bromer  wrote:
>
>>   Suppose you have sets of "programs" that produce two strings.  One set
>> of outputs is 00 and the other is 11. Now suppose you used these
>> sets of programs to chart the probabilities of the output of the strings.
>> If the two strings were each output by the same number of programs then
>> you'd have a .5 probability that either string would be output.  That's ok.
>> But, a more interesting question is, given that the first digits are 000,
>> what are the chances that the next digit will be 1?  Dim Induction will
>> report .5, which of course is nonsense and a whole less useful than making a
>> rough guess.
>>
>> But, of course, Solomonoff Induction purports to be able, if it was
>> feasible, to compute the possibilities for all possible programs.  Ok, but
>> now, try thinking about this a little bit.  If you have ever tried writing
>> random program instructions what do you usually get?  Well, I'll take a
>> hazard and guess (a lot better than the bogus method of confusing shallow
>> probability with "prediction" in my example) and say that you will get a lot
>> of programs that crash.  Well, most of my experiments with that have ended
>> up with programs that go into an infinite loop or which crash.  Now on a
>> universal Turing machine, the results would probably look a little
>> different.  Some strings will output nothing and go into an infinite loop.
>> Some programs will output something and then either stop outputting anything
>> or start outputting an infinite loop of the same substring.  Other programs
>> will go on to infinity producing something that looks like random strings.
>> But the idea that all possible programs would produce well distributed
>> strings is complete hogwash.  Since Solomonoff Induction does not define
>> what kind of programs should be used, the assumption that the distribution
>> would produce useful data is absurd.  In particular, the use of the method
>> to determine the probability based given an in

Re: [agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Abram Demski]

Jim,

I am unable to find the actual objection to Solomonoff in what you wrote
(save for that it's "wrong as in really wrong").

It's true that a lot of programs won't produce any output. That just means
they won't alter the prediction.

It's also true that a lot of programs will produce random-looking or
boring-looking output. This just means that Solomonoff will have some
expectation of those things. To use your example, given 000, the chances
that the next digit will be 0 will be fairly high thanks to boring programs
which just output lots of zeros. (Not sure why you mention the idea that it
might be .5? This sounds like "no induction" rather than "dim induction"...)

--Abram

On Wed, Jul 7, 2010 at 10:10 AM, Jim Bromer  wrote:

> Suppose you have sets of "programs" that produce two strings.  One set of
> outputs is 00 and the other is 11. Now suppose you used these sets
> of programs to chart the probabilities of the output of the strings.  If the
> two strings were each output by the same number of programs then you'd have
> a .5 probability that either string would be output.  That's ok.  But, a
> more interesting question is, given that the first digits are 000, what are
> the chances that the next digit will be 1?  Dim Induction will report .5,
> which of course is nonsense and a whole less useful than making a rough
> guess.
>
> But, of course, Solomonoff Induction purports to be able, if it was
> feasible, to compute the possibilities for all possible programs.  Ok, but
> now, try thinking about this a little bit.  If you have ever tried writing
> random program instructions what do you usually get?  Well, I'll take a
> hazard and guess (a lot better than the bogus method of confusing shallow
> probability with "prediction" in my example) and say that you will get a lot
> of programs that crash.  Well, most of my experiments with that have ended
> up with programs that go into an infinite loop or which crash.  Now on a
> universal Turing machine, the results would probably look a little
> different.  Some strings will output nothing and go into an infinite loop.
> Some programs will output something and then either stop outputting anything
> or start outputting an infinite loop of the same substring.  Other programs
> will go on to infinity producing something that looks like random strings.
> But the idea that all possible programs would produce well distributed
> strings is complete hogwash.  Since Solomonoff Induction does not define
> what kind of programs should be used, the assumption that the distribution
> would produce useful data is absurd.  In particular, the use of the method
> to determine the probability based given an initial string (as in what
> follows given the first digits are 000) is wrong as in really wrong.  The
> idea that this crude probability can be used as "prediction" is
> unsophisticated.
>
> Of course you could develop an infinite set of Solomonoff Induction values
> for each possible given initial sequence of digits.  Hey when you're working
> with infeasible functions why not dream anything?
>
> I might be wrong of course.  Maybe there is something you guys
> haven't been able to get across to me.  Even if you can think for yourself
> you can still make mistakes.  So if anyone has actually tried writing a
> program to output all possible programs (up to some feasible point) on a
> Turing Machine simulator, let me know how it went.
>
> Jim Bromer
>
>*agi* | Archives 
>  | 
> ModifyYour Subscription
> 
>



-- 
Abram Demski
http://lo-tho.blogspot.com/
http://groups.google.com/group/one-logic



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com

[agi] Solomonoff Induction is Not "Universal" and Probability is not "Prediction"

[Jim Bromer]

Suppose you have sets of "programs" that produce two strings.  One set of
outputs is 00 and the other is 11. Now suppose you used these sets
of programs to chart the probabilities of the output of the strings.  If the
two strings were each output by the same number of programs then you'd have
a .5 probability that either string would be output.  That's ok.  But, a
more interesting question is, given that the first digits are 000, what are
the chances that the next digit will be 1?  Dim Induction will report .5,
which of course is nonsense and a whole less useful than making a rough
guess.

But, of course, Solomonoff Induction purports to be able, if it was
feasible, to compute the possibilities for all possible programs.  Ok, but
now, try thinking about this a little bit.  If you have ever tried writing
random program instructions what do you usually get?  Well, I'll take a
hazard and guess (a lot better than the bogus method of confusing shallow
probability with "prediction" in my example) and say that you will get a lot
of programs that crash.  Well, most of my experiments with that have ended
up with programs that go into an infinite loop or which crash.  Now on a
universal Turing machine, the results would probably look a little
different.  Some strings will output nothing and go into an infinite loop.
Some programs will output something and then either stop outputting anything
or start outputting an infinite loop of the same substring.  Other programs
will go on to infinity producing something that looks like random strings.
But the idea that all possible programs would produce well distributed
strings is complete hogwash.  Since Solomonoff Induction does not define
what kind of programs should be used, the assumption that the distribution
would produce useful data is absurd.  In particular, the use of the method
to determine the probability based given an initial string (as in what
follows given the first digits are 000) is wrong as in really wrong.  The
idea that this crude probability can be used as "prediction" is
unsophisticated.

Of course you could develop an infinite set of Solomonoff Induction values
for each possible given initial sequence of digits.  Hey when you're working
with infeasible functions why not dream anything?

I might be wrong of course.  Maybe there is something you guys
haven't been able to get across to me.  Even if you can think for yourself
you can still make mistakes.  So if anyone has actually tried writing a
program to output all possible programs (up to some feasible point) on a
Turing Machine simulator, let me know how it went.

Jim Bromer



---
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com