Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Dec 13, 2007, at 9:46 AM, Don Dailey wrote:


I suspect there is more than 100 ELO between ranks at 9x9.


Me too ...

Christoph

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Jason House]

On Dec 14, 2007 10:55 AM, Nick Wedd <[EMAIL PROTECTED]> wrote:

> I'll bet that if someone ever does write a go-playing program that
> adapts its play in the light of what happens in the games it plays, I'll
> eventually be able to train it to make some _really_ bad moves.



That trick works against humans too.  I've stopped playing igowin for that
very reason...
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Nick Wedd]

In message <[EMAIL PROTECTED]>, terry mcintyre 
<[EMAIL PROTECTED]> writes

- Original Message 
From: Rémi Coulom <[EMAIL PROTECTED]>


For instance, against computers, I estimate that Crazy Stone improved
about 3 stones between this summer and now. But it clearly did not
improve 3 stones on KGS. I vaguely remember that Sylvain also noticed
that MoGo could beat GNU go with a 4-stone handicap, but was only 2
stones stronger than GNU on KGS.


We human players have a nasty habit of adapting to the weaknesses of
programs, and patching our own. I played a program
which totally does not know the Chinese opening. I always wind up
playing five large fuseki points, as it slowly plays five stones all
on one side of the board. No matter how often I beat it, it will not
adapt. Whereas, if a human player gets hit on the head often enough,
he'll try something different.


I'll bet that if someone ever does write a go-playing program that 
adapts its play in the light of what happens in the games it plays, I'll 
eventually be able to train it to make some _really_ bad moves.


Nick


This may account for improving versus another program, but not so much
versus humans.  The weaknesses of your computer opponents are more
consistent.
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
it now.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


--
Nick Wedd[EMAIL PROTECTED]
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christopher Rosin]

Hi.  My program greenpeep is currently UCT-based, with some MoGo-like
enhancements and some additional learning.
I described it more here:
http://computer-go.org/pipermail/computer-go/2007-October/011438.html
http://computer-go.org/pipermail/computer-go/2007-November/011865.html

Regarding the current discussion, at this point I have very little data for
greenpeep against humans.

-Chris Rosin


On Dec 13, 2007 12:09 PM, David Fotland <[EMAIL PROTECTED]> wrote:

> Isn't Greenpeep an alpha-beta searcher, not UCT/MC?
>
> Since Go ranks are based an handicap stones, and 100 ELO points implies a
> particular winning percentage, it would be an unlikely coincidence if 1
> rank
> is 100 ELO points.  Any web site that claims this must be wrong :) and
> should have little credibility.
>
> David
>
>
> >
> > The strongest bot on CGOS all time list seems to be  greenpeep0.5.1
> >  with a
> > rating
> > of 2621.
> >
> > This implies it is almost equal to a 5 Dan player - which doesn't sound
> > right to me.However,  this could be fluky since it is as at the
> > extreme end of the scale.  It would be great if this same program
> > could play some strong humans at the equivalent time control on KGS at
> > 9x9 and we could adjust the difference between ranks accordingly.
> >
> > I suspect there is more than 100 ELO between ranks at 9x9.
> >
> > - Don
> >
> >
> >
> > Don Dailey wrote:
> > > Christoph,
> > >
> > > Your bayeselo rating is 1942 on CGOS.  I compiled a table that
> > has
> > > all players with 50 games or more which can be found here:
> > >
> > >   http://cgos.boardspace.net/9x9/hof2.html
> > >
> > >
> > > - Don
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Christoph Birk wrote:
> > >
> > >> On Tue, 11 Dec 2007, Don Dailey wrote:
> > >>
> > >>> Christoph,
> > >>> Let me know when you are finished,  what name you are playing under
> > and
> > >>> I will do the bayeselo thing to get a better figure.
> > >>>
> > >> I am playing using the 'tast-3k' account. Right, now I have 71 games
> > >> and a rating of 1979 ELO.
> > >>
> > >>
> > >>>  Also, I can
> > >>> throw out any games that were irregular if you can identify them,
> > such
> > >>> as if  a match started when you were not looking or your interface
> > got
> > >>> glitchy or something.
> > >>>
> > >> Since I added the GUI I lost no games due to software problems.
> > >> Only a few "won" games lost to human stupidity :-)
> > >>
> > >> I will take a break over the holidays, maybe playing a few more
> > >> games in the new year, but I guess for my purposes a zero-point
> > >>"3k-AGA ~=~ 2000 CGOS-ELO"
> > >> is close enough. Unless we get some other (AGA or KGS) rated
> > >> players it not make sense to get a more precise rating for the
> > >> scale.
> > >>
> > >> Christoph
> > >> ___
> > >> computer-go mailing list
> > >> computer-go@computer-go.org
> > >> http://www.computer-go.org/mailman/listinfo/computer-go/
> > >>
> > >>
> > >
> > >
> > ___
> > computer-go mailing list
> > computer-go@computer-go.org
> > http://www.computer-go.org/mailman/listinfo/computer-go/
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

>
> What I mean is that if human player H beats computer C1 65% of the
> time, and computer C2 also beats computer C1 65% of the time, then I
> would expect that H would be stronger than C2, especially if both C1
> and C2 are MC programs. If it is the case, then it would make it
> difficult to compare human scale to computer scale. But that is just
> my intuition.
>
> For instance, against computers, I estimate that Crazy Stone improved
> about 3 stones between this summer and now. But it clearly did not
> improve 3 stones on KGS. I vaguely remember that Sylvain also noticed
> that MoGo could beat GNU go with a 4-stone handicap, but was only 2
> stones stronger than GNU on KGS.
This happened in computer chess many years ago.There was a period of
time when chess playing computers were relatively unknown but then
suddenly became very common.This is anecdotal,   but it appears that
for a  certain amount of time the computers continued to improve, but at
the same time humans adapted very quickly to them. Humans quickly
became educated.  They didn't become educated until computers
approached their playing strength.  

So there is a fairly significant ELO advantage to the human that has
experience playing computers.   I think it's very possible that this is
what you observed.The advantage of a human however has limits.   You
are not going to beat a player 1000 ELO stronger just because you know
how he plays.

Also, we don't see any serious discrepancy in computer vs human ratings
in chess although it was always imagined.   If you look in the sky you
can imagine interesting shapes,  but only because you are "looking" for
them. Whenever we think we observe something based on a few data
points it's extremely subject to error.   Thus people often believed
that some given program might be 200 ELO stronger than other programs
but that it would translate into something very modest against
humans.This NEVER turned out to be true - it was fantasy  based on
the continuing need to believe  that the improvements were just too good
to be true.  

A little common sense will tell you that this cannot be true but to a
very limited extent if any.You would have to believe that the
in-transitive rubber band stretches to infinity. Or you have to do
what Einstein did and regretted,  which is to impose an artificial
explanation such as some kind of constant that pushes this back at
higher strength levels.  The main point is that in computer chess
computers improved approximately 2000 ELO against humans over a couple
of decades or so.But when it's claimed that ELO improvement against
other computers  is 4 times that,  you imply 8000 ELO points for
computer vs computer! This is  clearly not the case.  That was
generally the claim I heard - 100 ELO improvement but it's someones
"belief" that against humans it's only about 25 ELO.  Nonsense.

I could see one things possibly happening however.   You might make a
real improvement that doesn't hit a human weakness that hard and at the
local limited  horizon it really may not translate to the same 
improvement against humans.But intransitivity in playing strength is
like a rubber band.You can only stretch it so far and it fights
back. You are not going to get  A beats B 99% of the time,   B beats
C 99% of the time,  but  A cannot beat C.Not unless you go way out
of your way to construct programs that have this behavior.   

I don't believe multi-dimensional playing strength  is much of an
issue.  It exists, but it's not severe.   You are not going to have to
worry that some 4 dan player will beat the world champion because he
happens to have a playing style that the world champion cannot handle.  
   Otherwise all rating/ranking systems would be useless.And it
would be easy to  construct some really ludicrous cases of intransitivity. 


- Don












>
> Rémi
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[terry mcintyre]

- Original Message 
From: Rémi Coulom <[EMAIL PROTECTED]>

> For instance, against computers, I estimate that Crazy Stone improved 
> about 3 stones between this summer and now. But it clearly did not 
> improve 3 stones on KGS. I vaguely remember that Sylvain also noticed 
> that MoGo could beat GNU go with a 4-stone handicap, but was only 2 
> stones stronger than GNU on KGS.


We human players have a nasty habit of adapting to the weaknesses of programs, 
and patching our own. I played a program 
which totally does not know the Chinese opening. I always wind up playing five 
large fuseki points, as it slowly plays five stones all 
on one side of the board. No matter how often I beat it, it will not adapt. 
Whereas, if a human player gets hit on the head often enough, 
he'll try something different. 

This may account for improving versus another program, but not so much versus 
humans.  The weaknesses of your computer opponents are more consistent.




  

Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  
http://tools.search.yahoo.com/newsearch/category.php?category=shopping___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Rémi Coulom]

Don Dailey wrote:

I don't really know what you mean by "one-dimensional."   My
understanding of playing strength is that it's not one-dimensional
meaning that it is foiled by in-transitivities between players with
different styles.You may be able to beat me,  but I might be able to
beat someone that you cannot.  If that's what you are saying how
does the kyu/dan system  handle it that makes it superior to ELO for
predicting who will win?Is there some mechanism that measures
playing styles?I don't see this.  
  


That's what I am saying. The kyu/dan system does not handle it better.


What I THINK you mean is that the gap between various GO ranks is not
consistent in ELO terms.   In other words there is no single constant
that works in my simple formula. I definitely think this is probably
the case but surely it can be easily handled by a slightly more
sophisticated formula that "fits the curve."
  


This is not what I meant, but I agree.

What I mean is that if human player H beats computer C1 65% of the time, 
and computer C2 also beats computer C1 65% of the time, then I would 
expect that H would be stronger than C2, especially if both C1 and C2 
are MC programs. If it is the case, then it would make it difficult to 
compare human scale to computer scale. But that is just my intuition.


For instance, against computers, I estimate that Crazy Stone improved 
about 3 stones between this summer and now. But it clearly did not 
improve 3 stones on KGS. I vaguely remember that Sylvain also noticed 
that MoGo could beat GNU go with a 4-stone handicap, but was only 2 
stones stronger than GNU on KGS.


Rémi
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Jason House]

On Dec 13, 2007 4:51 PM, Don Dailey <[EMAIL PROTECTED]> wrote:

> Do you have a suggestion for a specific mechanism for this?


I was mostly just thinking a file that cgos looks for that includes bot
names and the preferences.  The don't play list would need obvious
restrictions like what you've already suggested.  A preferred opponent was
intended more to get additional games against a particular bot in a shorter
period of time.  I was thinking it'd be tough to do and would only affect
the probability of playing a particular opponent.  Honestly, I wouldn't care
if these extra games didn't count for a rating.  If the bias isn't too bad,
maybe it wouldn't really matter that much.

Here's a motivational example:
  I'm a bot owner who's noticed that against bot X, I lose 80% of my games
because of some peculiar bug in my code.  I fix that code and, like a well
behaved CGOS participant, use a new login.  I then have to wait for my
rating to (somewhat) stabilize and then wait for the games against that bot
to accumulate enough samples to be sure I fixed the problem.  Letting new
bots specify their starting rating may help this process.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Are you suggesting a mechanism that allows you to turn this off and on
at will and that is separate from the naming and password convention?   

One thing I definitely would not do is allow you to select opponents you
prefer to play or not to play - whatever control we have will be limited
to our own bots. The main reason for CGOS was to ensure variety and
guarantee that players could not be "selective" about their opponents
the way bots on the chess servers are. It is claimed that many
players on the chess servers manipulate their ratings by carefully
selecting their opponents. I think this is CGOS strength, it does
not allow this.This makes ratings far more trustworthy. This is
also why you are not allowed to abort a game (once you see who your
opponent is.)   Otherwise it would be very easy to use this as a
mechanism to control your own pairings.  

Do you have a suggestion for a specific mechanism for this?

- Don
   



Jason House wrote:
>
>
> On Dec 13, 2007 4:01 PM, Don Dailey <[EMAIL PROTECTED]
> > wrote:
>
> I don't want to add more mechanisms.   You can build your own
> mechanism
> by making your own password naming convention or bot naming
> convention.For instance you can use the underscore character to
> build separate families of bots and still keep your own branding. 
>
>
> And if I then lose interest in self-play games (because I've seen
> enough to prove to myself whatever I was looking for)?  Do I then
> create a separate account with the same bot running on the same
> hardware and either discard historical data or manually combine the
> results?
>
> Maybe a more generic mechanism of opponent preferences would work? 
> Then I could specify bots I don't want to play (my family members) or
> others I want to play more (my rival?).  Obviously, the don't play
> might be restricted to bots in the same family or that use the same
> password. 
> 
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Regarding correspondance with human ranks, and handicap value, I cannot
tell yet. It is very clear to me that the Elo-rating model is very wrong
for the game of Go, because strength is not one-dimensional, especially
when mixing bots and humans. The best way to evaluate a bot in terms of
human rating is to make it play against humans, on KGS for instance.
Unfortunately, there is no 9x9 rating there. I will compute 9x9 ratings
with the KGS data I have.

I'm only interested in measuring the ELO gaps between 9x9 players of
different (19x19) rankings.   This you can do by simply taking the
statistics of wins and losses between players of various strengths. 

I don't really know what you mean by "one-dimensional."   My
understanding of playing strength is that it's not one-dimensional
meaning that it is foiled by in-transitivities between players with
different styles.You may be able to beat me,  but I might be able to
beat someone that you cannot.  If that's what you are saying how
does the kyu/dan system  handle it that makes it superior to ELO for
predicting who will win?Is there some mechanism that measures
playing styles?I don't see this.  

What I THINK you mean is that the gap between various GO ranks is not
consistent in ELO terms.   In other words there is no single constant
that works in my simple formula. I definitely think this is probably
the case but surely it can be easily handled by a slightly more
sophisticated formula that "fits the curve."

So surely, the average 2 dan player will beat the average 1 dan player
some statistically measurable percentage of the time at 9x9 go.   This
is what I want to know.Then I want to know if that percentage is the
same at different points in the scale.   If not,  then we find an
appropriate fit statistically.   

Once this is done then we still have the problem of calibrating CGOS -
we have to determine which ELO rating on CGOS corresponds to 1 dan (or
some arbitrary AGA or KGS ranking.)

Once all of this is done, we at least have something that doesn't yet
exist.   A credible way to claim your 9x9 program would likely hold it's
own against a 19x19 player of a given level.

Of course this will be somewhat noisy,  as any ranking system is.   It
will be subject to in-transitivities just like ELO and go ranking
are.  But you have to admit that there has been talk about certain
9x9 programs playing at the "dan level" and so on. Technically this
makes no sense,  but intuitively we know exactly what we mean when we
say that - we mean that it is the "equal" of a 1 dan 19x19 player.   
This is what I want to capture as a footnote on the 9x9 server.

I agree with you about program playing different versions of
themselves.   I can throw out games where a program plays another
verison of itself if you want to study that too (I would go by
password.)Just let me know and I will run another "hall of fame"
using that criteria or I can send you the data from cgos in a compact (1
line per game) representation if you think it would be useful to help
you understand this.   Or I can send you the pgn files I produced to be
compatible with bayeselo.


- Don


 
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Jason House]

On Dec 13, 2007 4:01 PM, Don Dailey <[EMAIL PROTECTED]> wrote:

> I don't want to add more mechanisms.   You can build your own mechanism
> by making your own password naming convention or bot naming
> convention.For instance you can use the underscore character to
> build separate families of bots and still keep your own branding.


And if I then lose interest in self-play games (because I've seen enough to
prove to myself whatever I was looking for)?  Do I then create a separate
account with the same bot running on the same hardware and either discard
historical data or manually combine the results?

Maybe a more generic mechanism of opponent preferences would work?  Then I
could specify bots I don't want to play (my family members) or others I want
to play more (my rival?).  Obviously, the don't play might be restricted to
bots in the same family or that use the same password.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

I don't want to add more mechanisms.   You can build your own mechanism
by making your own password naming convention or bot naming
convention.For instance you can use the underscore character to
build separate families of bots and still keep your own branding. 

We might at some point make a web page that groups bots by family so
that it's publicly visible and easy to see which bots do not play each
other.  

> PS: Will this new feature be committed into the subversion repository
for CGOS?

Yes, when I do this I will use the sourceforge system.  

- Don


 
Jason House wrote:
>
>
> On Dec 13, 2007 2:37 PM, Don Dailey <[EMAIL PROTECTED]
> > wrote:
>
> I am considering to enforce this basic protocol on the server soon:
>
>  Programs of the same "family" will not be paired against each
> other.
>
>
> I frequently look at the games between my bot version more than I look
> at them with other bots.  This is unfortunately a side-effect of
> having few (9x9) bots in the 1200-1400 ELO range.  I like using
> similar names and the same password since it lets me keep my sanity
> when switching between machines!
>
> If the family feature is added, I'd like another mechanism to control
> the family settings.  I suspect that a programmer's decision to have
> their bots play each other (or not) may change with time.  It'd be
> nice if that was supported.
>
> PS: Will this new feature be committed into the subversion repository
> for CGOS?
> 
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Chris Fant]

> On Dec 13, 2007 3:09 PM, David Fotland <[EMAIL PROTECTED]> wrote:
> > Isn't Greenpeep an alpha-beta searcher, not UCT/MC?
>
> I could have sworn I heard it described as UCT/MC with MoGo-like
> enhancements.
>

Same here.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Jason House]

On Dec 13, 2007 3:09 PM, David Fotland <[EMAIL PROTECTED]> wrote:

> Isn't Greenpeep an alpha-beta searcher, not UCT/MC?


I could have sworn I heard it described as UCT/MC with MoGo-like
enhancements.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Jason House]

On Dec 13, 2007 2:37 PM, Don Dailey <[EMAIL PROTECTED]> wrote:

>  I am considering to enforce this basic protocol on the server soon:
>
>  Programs of the same "family" will not be paired against each other.
>

I frequently look at the games between my bot version more than I look at
them with other bots.  This is unfortunately a side-effect of having few
(9x9) bots in the 1200-1400 ELO range.  I like using similar names and the
same password since it lets me keep my sanity when switching between
machines!

If the family feature is added, I'd like another mechanism to control the
family settings.  I suspect that a programmer's decision to have their bots
play each other (or not) may change with time.  It'd be nice if that was
supported.

PS: Will this new feature be committed into the subversion repository for
CGOS?
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

David Fotland wrote:
> Isn't Greenpeep an alpha-beta searcher, not UCT/MC?
>
> Since Go ranks are based an handicap stones, and 100 ELO points implies a
> particular winning percentage, it would be an unlikely coincidence if 1 rank
> is 100 ELO points.  Any web site that claims this must be wrong :) and
> should have little credibility.
>   
I don't believe it's 100 ELO either.   It's probably something between
100 and 200.

> David
>
>
>   
>> The strongest bot on CGOS all time list seems to be  greenpeep0.5.1
>>  with a
>> rating
>> of 2621.
>>
>> This implies it is almost equal to a 5 Dan player - which doesn't sound
>> right to me.However,  this could be fluky since it is as at the
>> extreme end of the scale.  It would be great if this same program
>> could play some strong humans at the equivalent time control on KGS at
>> 9x9 and we could adjust the difference between ranks accordingly.
>>
>> I suspect there is more than 100 ELO between ranks at 9x9.
>>
>> - Don
>>
>>
>>
>> Don Dailey wrote:
>> 
>>> Christoph,
>>>
>>> Your bayeselo rating is 1942 on CGOS.  I compiled a table that
>>>   
>> has
>> 
>>> all players with 50 games or more which can be found here:
>>>
>>>   http://cgos.boardspace.net/9x9/hof2.html
>>>
>>>
>>> - Don
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Christoph Birk wrote:
>>>
>>>   
 On Tue, 11 Dec 2007, Don Dailey wrote:

 
> Christoph,
> Let me know when you are finished,  what name you are playing under
>   
>> and
>> 
> I will do the bayeselo thing to get a better figure.
>
>   
 I am playing using the 'tast-3k' account. Right, now I have 71 games
 and a rating of 1979 ELO.


 
>  Also, I can
> throw out any games that were irregular if you can identify them,
>   
>> such
>> 
> as if  a match started when you were not looking or your interface
>   
>> got
>> 
> glitchy or something.
>
>   
 Since I added the GUI I lost no games due to software problems.
 Only a few "won" games lost to human stupidity :-)

 I will take a break over the holidays, maybe playing a few more
 games in the new year, but I guess for my purposes a zero-point
"3k-AGA ~=~ 2000 CGOS-ELO"
 is close enough. Unless we get some other (AGA or KGS) rated
 players it not make sense to get a more precise rating for the
 scale.

 Christoph
 ___
 computer-go mailing list
 computer-go@computer-go.org
 http://www.computer-go.org/mailman/listinfo/computer-go/


 
>>>   
>> ___
>> computer-go mailing list
>> computer-go@computer-go.org
>> http://www.computer-go.org/mailman/listinfo/computer-go/
>> 
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
>   
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

RE: [computer-go] Where and How to Test the Strong Programs?

[David Fotland]

Isn't Greenpeep an alpha-beta searcher, not UCT/MC?

Since Go ranks are based an handicap stones, and 100 ELO points implies a
particular winning percentage, it would be an unlikely coincidence if 1 rank
is 100 ELO points.  Any web site that claims this must be wrong :) and
should have little credibility.

David


> 
> The strongest bot on CGOS all time list seems to be  greenpeep0.5.1
>  with a
> rating
> of 2621.
> 
> This implies it is almost equal to a 5 Dan player - which doesn't sound
> right to me.However,  this could be fluky since it is as at the
> extreme end of the scale.  It would be great if this same program
> could play some strong humans at the equivalent time control on KGS at
> 9x9 and we could adjust the difference between ranks accordingly.
> 
> I suspect there is more than 100 ELO between ranks at 9x9.
> 
> - Don
> 
> 
> 
> Don Dailey wrote:
> > Christoph,
> >
> > Your bayeselo rating is 1942 on CGOS.  I compiled a table that
> has
> > all players with 50 games or more which can be found here:
> >
> >   http://cgos.boardspace.net/9x9/hof2.html
> >
> >
> > - Don
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Christoph Birk wrote:
> >
> >> On Tue, 11 Dec 2007, Don Dailey wrote:
> >>
> >>> Christoph,
> >>> Let me know when you are finished,  what name you are playing under
> and
> >>> I will do the bayeselo thing to get a better figure.
> >>>
> >> I am playing using the 'tast-3k' account. Right, now I have 71 games
> >> and a rating of 1979 ELO.
> >>
> >>
> >>>  Also, I can
> >>> throw out any games that were irregular if you can identify them,
> such
> >>> as if  a match started when you were not looking or your interface
> got
> >>> glitchy or something.
> >>>
> >> Since I added the GUI I lost no games due to software problems.
> >> Only a few "won" games lost to human stupidity :-)
> >>
> >> I will take a break over the holidays, maybe playing a few more
> >> games in the new year, but I guess for my purposes a zero-point
> >>"3k-AGA ~=~ 2000 CGOS-ELO"
> >> is close enough. Unless we get some other (AGA or KGS) rated
> >> players it not make sense to get a more precise rating for the
> >> scale.
> >>
> >> Christoph
> >> ___
> >> computer-go mailing list
> >> computer-go@computer-go.org
> >> http://www.computer-go.org/mailman/listinfo/computer-go/
> >>
> >>
> >
> >
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Hideki Kato]

Hi Rémi ,

Rémi Coulom: <[EMAIL PROTECTED]>:
>Don Dailey wrote:
>> It would be great if you would provide recommendations for a simple
>> conversion formula when you are ready based on this study.   Also,
>> if you have any suggestions in general for CGOS ratings the
>> cgos-developers would be willing to listen to your suggestions.
>>
>> - Don
>My suggestion would be to tell programmers to use a different login each 
>time they change version or hardware (most do that, already), and use 
>bayeselo to rank the programs.

Sure, it should be recommended strongly.  Before overall ratings, I
was not so serious to change login names as I just want to know recent
rating which is enough.  In fact, I misestimated the time consumption
of my latest version of ggmc and it lost many games by time for a
week or so.  I noticed it and have changed its setting of time control
a few days ago.  Its rating improved much and so, the difference
between its overall (2011) and current (2043) ratings is relatively
large.  I strongly believe that this kind of _minor_ fixing are made
frequently in past.

As a conclusion, current overall ratings are hard to believe or,
perhaps,  may include lots of error.  I'd like to suggest to postpone
this excellent idea until almost all peticipants surely do change
their login names if they modify, not only changing versions, their
programs, and to exclude older programs from counting.  

-Hideki

>This would be best if combined with a mechanism to recognize that two 
>logins are versions of the same program (for instance, if they use the 
>same password), and avoid pairing them.
>
>Regarding correspondance with human ranks, and handicap value, I cannot 
>tell yet. It is very clear to me that the Elo-rating model is very wrong 
>for the game of Go, because strength is not one-dimensional, especially 
>when mixing bots and humans. The best way to evaluate a bot in terms of 
>human rating is to make it play against humans, on KGS for instance. 
>Unfortunately, there is no 9x9 rating there. I will compute 9x9 ratings 
>with the KGS data I have.
>
>What I have observed with Crazy Stone is that gaining Elo points against 
>humans is more difficult than gaining Elo points against GNU Go, which 
>is more difficult than gaining Elo points against MC programs, which is 
>more difficult than gaining Elo points against itself. But it is more an 
>intuition than a scientific study.
>
>Rémi
>___
>computer-go mailing list
>computer-go@computer-go.org
>http://www.computer-go.org/mailman/listinfo/computer-go/
--
[EMAIL PROTECTED] (Kato)
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

From time to time I have put highly experimental and very different
programs on CGOS and I don't care if they play themselves


What I meant to say is that I don't care if they play other programs of
mine.

- Don


Don Dailey wrote:
> I am considering to enforce this basic protocol on the server soon:
>
>  Programs of the same "family" will not be paired against each other.
>
> A family of programs have the same name up to the first hyphen and the
> same password.
>
> So if I have these programs:
>
> Name   password
> -----
> Lazarus-1.2foobar
> Lazarus-1.3foobar
> Lazarus-1.4foobar
> Lazarus-1.5winniepooh
>
> Then Lazarus-1.5 will be allowed to play either of the other programs
> listed,  but those other programs will not be allow to play each other
> as they are considered relatives.  
>
> We cannot prevent programs from playing each other no matter what we
> do, they can always change the name and password.   However this gives
> the programmers the ability to prevent multiple versions of his
> program from playing each other if he chooses. Most programmers
> probably log onto CGOS in order to play other peoples programs.
>
> >From time to time I have put highly experimental and very different
> programs on CGOS and I don't care if they play themselves - they are
> not really different versions of the same program.   In this case I
> always give them different names anyway.   I pretty much always use
> the same password so I can control this easily with the name. 
>
> - Don
>
>
>
>
>
>   
>
> Rémi Coulom wrote:
>> Don Dailey wrote:
>>> It would be great if you would provide recommendations for a simple
>>> conversion formula when you are ready based on this study.   Also,
>>> if you have any suggestions in general for CGOS ratings the
>>> cgos-developers would be willing to listen to your suggestions.
>>>
>>> - Don
>> My suggestion would be to tell programmers to use a different login
>> each time they change version or hardware (most do that, already),
>> and use bayeselo to rank the programs.
>>
>> This would be best if combined with a mechanism to recognize that two
>> logins are versions of the same program (for instance, if they use
>> the same password), and avoid pairing them.
>>
>> Regarding correspondance with human ranks, and handicap value, I
>> cannot tell yet. It is very clear to me that the Elo-rating model is
>> very wrong for the game of Go, because strength is not
>> one-dimensional, especially when mixing bots and humans. The best way
>> to evaluate a bot in terms of human rating is to make it play against
>> humans, on KGS for instance. Unfortunately, there is no 9x9 rating
>> there. I will compute 9x9 ratings with the KGS data I have.
>>
>> What I have observed with Crazy Stone is that gaining Elo points
>> against humans is more difficult than gaining Elo points against GNU
>> Go, which is more difficult than gaining Elo points against MC
>> programs, which is more difficult than gaining Elo points against
>> itself. But it is more an intuition than a scientific study.
>>
>> Rémi
>> ___
>> computer-go mailing list
>> computer-go@computer-go.org
>> http://www.computer-go.org/mailman/listinfo/computer-go/
>>
> 
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

I am considering to enforce this basic protocol on the server soon:

 Programs of the same "family" will not be paired against each other.

A family of programs have the same name up to the first hyphen and the
same password.

So if I have these programs:

Name   password
-----
Lazarus-1.2foobar
Lazarus-1.3foobar
Lazarus-1.4foobar
Lazarus-1.5winniepooh

Then Lazarus-1.5 will be allowed to play either of the other programs
listed,  but those other programs will not be allow to play each other
as they are considered relatives.  

We cannot prevent programs from playing each other no matter what we do,
they can always change the name and password.   However this gives the
programmers the ability to prevent multiple versions of his program from
playing each other if he chooses. Most programmers probably log onto
CGOS in order to play other peoples programs.

>From time to time I have put highly experimental and very different
programs on CGOS and I don't care if they play themselves - they are not
really different versions of the same program.   In this case I always
give them different names anyway.   I pretty much always use the same
password so I can control this easily with the name. 

- Don



   

  

Rémi Coulom wrote:
> Don Dailey wrote:
>> It would be great if you would provide recommendations for a simple
>> conversion formula when you are ready based on this study.   Also,
>> if you have any suggestions in general for CGOS ratings the
>> cgos-developers would be willing to listen to your suggestions.
>>
>> - Don
> My suggestion would be to tell programmers to use a different login
> each time they change version or hardware (most do that, already), and
> use bayeselo to rank the programs.
>
> This would be best if combined with a mechanism to recognize that two
> logins are versions of the same program (for instance, if they use the
> same password), and avoid pairing them.
>
> Regarding correspondance with human ranks, and handicap value, I
> cannot tell yet. It is very clear to me that the Elo-rating model is
> very wrong for the game of Go, because strength is not
> one-dimensional, especially when mixing bots and humans. The best way
> to evaluate a bot in terms of human rating is to make it play against
> humans, on KGS for instance. Unfortunately, there is no 9x9 rating
> there. I will compute 9x9 ratings with the KGS data I have.
>
> What I have observed with Crazy Stone is that gaining Elo points
> against humans is more difficult than gaining Elo points against GNU
> Go, which is more difficult than gaining Elo points against MC
> programs, which is more difficult than gaining Elo points against
> itself. But it is more an intuition than a scientific study.
>
> Rémi
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Rémi Coulom]

Don Dailey wrote:

It would be great if you would provide recommendations for a simple
conversion formula when you are ready based on this study.   Also,
if you have any suggestions in general for CGOS ratings the
cgos-developers would be willing to listen to your suggestions.

- Don
My suggestion would be to tell programmers to use a different login each 
time they change version or hardware (most do that, already), and use 
bayeselo to rank the programs.


This would be best if combined with a mechanism to recognize that two 
logins are versions of the same program (for instance, if they use the 
same password), and avoid pairing them.


Regarding correspondance with human ranks, and handicap value, I cannot 
tell yet. It is very clear to me that the Elo-rating model is very wrong 
for the game of Go, because strength is not one-dimensional, especially 
when mixing bots and humans. The best way to evaluate a bot in terms of 
human rating is to make it play against humans, on KGS for instance. 
Unfortunately, there is no 9x9 rating there. I will compute 9x9 ratings 
with the KGS data I have.


What I have observed with Crazy Stone is that gaining Elo points against 
humans is more difficult than gaining Elo points against GNU Go, which 
is more difficult than gaining Elo points against MC programs, which is 
more difficult than gaining Elo points against itself. But it is more an 
intuition than a scientific study.


Rémi
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Hideki Kato wrote:
> Hi Don,
>
> There are not enough evidence to believe this.
>   
I realize that - and I don't expect it to be perfect.   But it's a
starting point.I would like to put something up pretty soon just to
have something to debate about :-) Seriously,  if we make some kind
of stab at it there is no harm done.I would of course put a
disclaimer on the main web page and make it clear that this formula is
highly speculative and that it will improve as we get more information. 

- Don



> Tast-3k has too few matches against each program, less than ten games 
> and has no matches against strongest programs including Crazy Stone, 
> MoGo and greenpeep.  In addition, there seems some bias, that is, 
> his winning rate against gnugo-3.7.10(s) is less than 50% but is 
> 100% against FatMan while their ratings are almost the same, about 
> 1800 ELO.
> http://cgos.boardspace.net/9x9/cross/tast-3k.html
>
> -Hideki
>
> Don Dailey: <[EMAIL PROTECTED]>:
>   
>> I'm going to estimate that 100 ELO is roughly 1 rank based on this:
>>
>> http://en.wikipedia.org/wiki/Go_ranks_and_ratings
>>
>> This may not hold for 9x9.If a 1 kyu beats a 2kyu about 64% of the
>> time in an even game at 19x19,  it doesn't imply that he will do the
>> same at 9x9,  but until I have a reason to believe differently I will
>> set this as our first guess in the formula.We can always come back
>> and modify this later.
>>
>> We may be able to borrow KGS data of well established players playing
>> 9x9 games against each other to estimate this. Would anyone like to
>> volunteer to do this?
>>
>> So as it stands, our formula is:
>>
>>  3ky = 1942 cgos_elo
>>
>>   dan =  (your_elo - 2142) / 100   
>>   kyu  = -dan + 1
>>
>>1 Dan  =  2242 cgos
>>1 Kyu   =  2142 cgos
>>2 Kyu   =  2042 cgos
>>3 Kyu   =  1942 cgos
>>
>> The strongest bot on CGOS all time list seems to be  greenpeep0.5.1
>>  with a rating
>> of 2621.
>>
>> This implies it is almost equal to a 5 Dan player - which doesn't sound
>> right to me.However,  this could be fluky since it is as at the
>> extreme end of the scale.  It would be great if this same program
>> could play some strong humans at the equivalent time control on KGS at
>> 9x9 and we could adjust the difference between ranks accordingly.   
>>
>> I suspect there is more than 100 ELO between ranks at 9x9.  
>>
>> - Don
>>
>>
>>
>> Don Dailey wrote:
>> 
>>> Christoph,
>>>
>>> Your bayeselo rating is 1942 on CGOS.  I compiled a table that has
>>> all players with 50 games or more which can be found here:
>>>
>>>   http://cgos.boardspace.net/9x9/hof2.html
>>>
>>>
>>> - Don
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Christoph Birk wrote:
>>>   
>>>   
 On Tue, 11 Dec 2007, Don Dailey wrote:
 
 
> Christoph,
> Let me know when you are finished,  what name you are playing under and
> I will do the bayeselo thing to get a better figure.
>   
>   
 I am playing using the 'tast-3k' account. Right, now I have 71 games
 and a rating of 1979 ELO.

 
 
>  Also, I can
> throw out any games that were irregular if you can identify them,  such
> as if  a match started when you were not looking or your interface got
> glitchy or something.
>   
>   
 Since I added the GUI I lost no games due to software problems.
 Only a few "won" games lost to human stupidity :-)

 I will take a break over the holidays, maybe playing a few more
 games in the new year, but I guess for my purposes a zero-point
"3k-AGA ~=~ 2000 CGOS-ELO"
 is close enough. Unless we get some other (AGA or KGS) rated
 players it not make sense to get a more precise rating for the
 scale.

 Christoph
 ___
 computer-go mailing list
 computer-go@computer-go.org
 http://www.computer-go.org/mailman/listinfo/computer-go/

 
 
>>>   
>>>   
>> ___
>> computer-go mailing list
>> computer-go@computer-go.org
>> http://www.computer-go.org/mailman/listinfo/computer-go/
>> 
> --
> [EMAIL PROTECTED] (Kato)
>
>   
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Hideki Kato]

Hi Don,

There are not enough evidence to believe this.

Tast-3k has too few matches against each program, less than ten games 
and has no matches against strongest programs including Crazy Stone, 
MoGo and greenpeep.  In addition, there seems some bias, that is, 
his winning rate against gnugo-3.7.10(s) is less than 50% but is 
100% against FatMan while their ratings are almost the same, about 
1800 ELO.
http://cgos.boardspace.net/9x9/cross/tast-3k.html

-Hideki

Don Dailey: <[EMAIL PROTECTED]>:
>I'm going to estimate that 100 ELO is roughly 1 rank based on this:
>
> http://en.wikipedia.org/wiki/Go_ranks_and_ratings
>
>This may not hold for 9x9.If a 1 kyu beats a 2kyu about 64% of the
>time in an even game at 19x19,  it doesn't imply that he will do the
>same at 9x9,  but until I have a reason to believe differently I will
>set this as our first guess in the formula.We can always come back
>and modify this later.
>
>We may be able to borrow KGS data of well established players playing
>9x9 games against each other to estimate this. Would anyone like to
>volunteer to do this?
>
>So as it stands, our formula is:
>
>  3ky = 1942 cgos_elo
>
>   dan =  (your_elo - 2142) / 100   
>   kyu  = -dan + 1
>
>1 Dan  =  2242 cgos
>1 Kyu   =  2142 cgos
>2 Kyu   =  2042 cgos
>3 Kyu   =  1942 cgos
>
>The strongest bot on CGOS all time list seems to be  greenpeep0.5.1
> with a rating
>of 2621.
>
>This implies it is almost equal to a 5 Dan player - which doesn't sound
>right to me.However,  this could be fluky since it is as at the
>extreme end of the scale.  It would be great if this same program
>could play some strong humans at the equivalent time control on KGS at
>9x9 and we could adjust the difference between ranks accordingly.   
>
>I suspect there is more than 100 ELO between ranks at 9x9.  
>
>- Don
>
>
>
>Don Dailey wrote:
>> Christoph,
>>
>> Your bayeselo rating is 1942 on CGOS.  I compiled a table that has
>> all players with 50 games or more which can be found here:
>>
>>   http://cgos.boardspace.net/9x9/hof2.html
>>
>>
>> - Don
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Christoph Birk wrote:
>>   
>>> On Tue, 11 Dec 2007, Don Dailey wrote:
>>> 
 Christoph,
 Let me know when you are finished,  what name you are playing under and
 I will do the bayeselo thing to get a better figure.
   
>>> I am playing using the 'tast-3k' account. Right, now I have 71 games
>>> and a rating of 1979 ELO.
>>>
>>> 
  Also, I can
 throw out any games that were irregular if you can identify them,  such
 as if  a match started when you were not looking or your interface got
 glitchy or something.
   
>>> Since I added the GUI I lost no games due to software problems.
>>> Only a few "won" games lost to human stupidity :-)
>>>
>>> I will take a break over the holidays, maybe playing a few more
>>> games in the new year, but I guess for my purposes a zero-point
>>>"3k-AGA ~=~ 2000 CGOS-ELO"
>>> is close enough. Unless we get some other (AGA or KGS) rated
>>> players it not make sense to get a more precise rating for the
>>> scale.
>>>
>>> Christoph
>>> ___
>>> computer-go mailing list
>>> computer-go@computer-go.org
>>> http://www.computer-go.org/mailman/listinfo/computer-go/
>>>
>>> 
>>
>>   
>___
>computer-go mailing list
>computer-go@computer-go.org
>http://www.computer-go.org/mailman/listinfo/computer-go/
--
[EMAIL PROTECTED] (Kato)
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

It would be great if you would provide recommendations for a simple
conversion formula when you are ready based on this study.   Also,
if you have any suggestions in general for CGOS ratings the
cgos-developers would be willing to listen to your suggestions.

- Don



Rémi Coulom wrote:
> Don Dailey wrote:
>> We may be able to borrow KGS data of well established players playing
>> 9x9 games against each other to estimate this. Would anyone like to
>> volunteer to do this?
>
> Bill Shubert kindly provided this data to me. I am working on a study
> about rating systems for the game of Go. I expect I will make a
> preliminary version available in January.
>
> Rémi
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Rémi Coulom]

Don Dailey wrote:

We may be able to borrow KGS data of well established players playing
9x9 games against each other to estimate this. Would anyone like to
volunteer to do this?


Bill Shubert kindly provided this data to me. I am working on a study 
about rating systems for the game of Go. I expect I will make a 
preliminary version available in January.


Rémi
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

I'm going to estimate that 100 ELO is roughly 1 rank based on this:

 http://en.wikipedia.org/wiki/Go_ranks_and_ratings

This may not hold for 9x9.If a 1 kyu beats a 2kyu about 64% of the
time in an even game at 19x19,  it doesn't imply that he will do the
same at 9x9,  but until I have a reason to believe differently I will
set this as our first guess in the formula.We can always come back
and modify this later.

We may be able to borrow KGS data of well established players playing
9x9 games against each other to estimate this. Would anyone like to
volunteer to do this?

So as it stands, our formula is:

  3ky = 1942 cgos_elo

   dan =  (your_elo - 2142) / 100   
   kyu  = -dan + 1

1 Dan  =  2242 cgos
1 Kyu   =  2142 cgos
2 Kyu   =  2042 cgos
3 Kyu   =  1942 cgos

The strongest bot on CGOS all time list seems to be  greenpeep0.5.1
 with a rating
of 2621.

This implies it is almost equal to a 5 Dan player - which doesn't sound
right to me.However,  this could be fluky since it is as at the
extreme end of the scale.  It would be great if this same program
could play some strong humans at the equivalent time control on KGS at
9x9 and we could adjust the difference between ranks accordingly.   

I suspect there is more than 100 ELO between ranks at 9x9.  

- Don



Don Dailey wrote:
> Christoph,
>
> Your bayeselo rating is 1942 on CGOS.  I compiled a table that has
> all players with 50 games or more which can be found here:
>
>   http://cgos.boardspace.net/9x9/hof2.html
>
>
> - Don
>
>
>
>
>
>
>
>
>
>
> Christoph Birk wrote:
>   
>> On Tue, 11 Dec 2007, Don Dailey wrote:
>> 
>>> Christoph,
>>> Let me know when you are finished,  what name you are playing under and
>>> I will do the bayeselo thing to get a better figure.
>>>   
>> I am playing using the 'tast-3k' account. Right, now I have 71 games
>> and a rating of 1979 ELO.
>>
>> 
>>>  Also, I can
>>> throw out any games that were irregular if you can identify them,  such
>>> as if  a match started when you were not looking or your interface got
>>> glitchy or something.
>>>   
>> Since I added the GUI I lost no games due to software problems.
>> Only a few "won" games lost to human stupidity :-)
>>
>> I will take a break over the holidays, maybe playing a few more
>> games in the new year, but I guess for my purposes a zero-point
>>"3k-AGA ~=~ 2000 CGOS-ELO"
>> is close enough. Unless we get some other (AGA or KGS) rated
>> players it not make sense to get a more precise rating for the
>> scale.
>>
>> Christoph
>> ___
>> computer-go mailing list
>> computer-go@computer-go.org
>> http://www.computer-go.org/mailman/listinfo/computer-go/
>>
>> 
>
>   
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Christoph,

Your bayeselo rating is 1942 on CGOS.  I compiled a table that has
all players with 50 games or more which can be found here:

  http://cgos.boardspace.net/9x9/hof2.html


- Don










Christoph Birk wrote:
> On Tue, 11 Dec 2007, Don Dailey wrote:
>> Christoph,
>> Let me know when you are finished,  what name you are playing under and
>> I will do the bayeselo thing to get a better figure.
>
> I am playing using the 'tast-3k' account. Right, now I have 71 games
> and a rating of 1979 ELO.
>
>>  Also, I can
>> throw out any games that were irregular if you can identify them,  such
>> as if  a match started when you were not looking or your interface got
>> glitchy or something.
>
> Since I added the GUI I lost no games due to software problems.
> Only a few "won" games lost to human stupidity :-)
>
> I will take a break over the holidays, maybe playing a few more
> games in the new year, but I guess for my purposes a zero-point
>"3k-AGA ~=~ 2000 CGOS-ELO"
> is close enough. Unless we get some other (AGA or KGS) rated
> players it not make sense to get a more precise rating for the
> scale.
>
> Christoph
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 11 Dec 2007, Don Dailey wrote:

Christoph,
Let me know when you are finished,  what name you are playing under and
I will do the bayeselo thing to get a better figure.


I am playing using the 'tast-3k' account. Right, now I have 71 games
and a rating of 1979 ELO.


 Also, I can
throw out any games that were irregular if you can identify them,  such
as if  a match started when you were not looking or your interface got
glitchy or something.


Since I added the GUI I lost no games due to software problems.
Only a few "won" games lost to human stupidity :-)

I will take a break over the holidays, maybe playing a few more
games in the new year, but I guess for my purposes a zero-point
   "3k-AGA ~=~ 2000 CGOS-ELO"
is close enough. Unless we get some other (AGA or KGS) rated
players it not make sense to get a more precise rating for the
scale.

Christoph
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Alain Baeckeroot]

Le mercredi 12 décembre 2007, Ben Lambrechts a écrit :
> 
> > How do AGA ratings compare to KGS?
> Sensei's Library is your friend ;o)
> http://senseis.xmp.net/?RankWorldwideComparison
> 

I believe this page has not been updated since last year change
on kgs ranking scale.

Kgs have the big advantage of letting some bots get a rank, which is very
convenient.

Alain

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Ben Lambrechts]

How do AGA ratings compare to KGS?

Sensei's Library is your friend ;o)
http://senseis.xmp.net/?RankWorldwideComparison

Ben
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Michael Alford]

I have lurked here for a long time, I find this newsletter very 
interesting, because I play Go, I am not a programmer. I agree with 
everything Mr Foltand had to say, but would like to add a small bit 
about IGS ranks. A few years ago, IGS changed its ranking system so as 
to anchor from the top, ie a 9d on IGS was a 9d pro, making it the 
closest to "real world" rankings. I think the strength of IGS ranks has 
slipped a bit since cyberoro came online, but is still probably closest 
to "real world" ranks. I would suggest not using AGA or KGS or any other 
 ranking system for establishing your CGOS ranks, recruit some IGS 
players with solid (over 100 rated games) ranks, and have them play with 
your various engines.


back to lurking,
Michael

David Fotland wrote:

I think AGA and KGS are pretty close.  AGA is a "real" rating system in that
ratings are earned in sanctioned tournaments so they are not disrupted by
casual games.  http://www.usgo.org/ratings/default.html  


European ratings (also from tournaments) are perhaps 2 stones tougher.  Many
think they are more reasonable, since most feel the top of the amateur
rating should be 7 dan or lower, and top AGA ratings are higher than that.
Top pros that have participated in AGA tournaments have ratings about 10
dan, and there are many amateur 8 dans.

Japanese ratings are less tough.  Japanese amateur ratings can be purchased,
with a test, so there has been inflation, and I don't think there is a
national rating organization like in USA and Europe. 


Korean and Chinese are very tough, since they think a 1 dan amateur should
be close to professional strength.

So, I'm AGA 3 dan, but I would have a tough time playing as a 1 dan in
Europe, and I play at 4 dan in clubs in Japan.

I tried playing in a club as 3 dan in China once, and got totally crushed.

My preference would be a scale that is fixed at the top, with 9 dan pros at
9 dan.  This would put 1 dan pros at about 7 dan and top amateurs at 6 dan,
with a few 7 dans.  This is pretty close to the European scale.  Top
tournament pros almost never lose to pro 1 dans, so there are 300 or more
ELO points between the top amateurs and the top professionals.

So perhaps top human play is 3500 or more on the cgos 19x19 scale.  That's
12 ranks above 2000, with the higher ranks having more ELO points per rank.

David


-Original Message-
From: [EMAIL PROTECTED] [mailto:computer-go-
[EMAIL PROTECTED] On Behalf Of Don Dailey
Sent: Tuesday, December 11, 2007 6:37 PM
To: computer-go
Subject: Re: [computer-go] Where and How to Test the Strong Programs?

I feel that we probably need several more players to have much
accuracy,  but I don't mind starting the best educated guess we can
muster - it can be modified at a later time.

How do AGA ratings compare to other systems?   Is any particular system
considered (defacto or otherwise) more of a "standard" than some other?

How do AGA ratings compare to KGS?

- Don



Christoph Birk wrote:

It looks like my (3k AGA) CGOS rating (tast-3k) is converging around
2000 ELO. That gives us a zero-point but we need at least one more
rated player (better more) to get the scale.
If you would like to use my GUI please contact me by private email
at ccbirk at gmail dot com.

Christoph
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/



___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

RE: [computer-go] Where and How to Test the Strong Programs?

[David Fotland]

I think AGA and KGS are pretty close.  AGA is a "real" rating system in that
ratings are earned in sanctioned tournaments so they are not disrupted by
casual games.  http://www.usgo.org/ratings/default.html  

European ratings (also from tournaments) are perhaps 2 stones tougher.  Many
think they are more reasonable, since most feel the top of the amateur
rating should be 7 dan or lower, and top AGA ratings are higher than that.
Top pros that have participated in AGA tournaments have ratings about 10
dan, and there are many amateur 8 dans.

Japanese ratings are less tough.  Japanese amateur ratings can be purchased,
with a test, so there has been inflation, and I don't think there is a
national rating organization like in USA and Europe. 

Korean and Chinese are very tough, since they think a 1 dan amateur should
be close to professional strength.

So, I'm AGA 3 dan, but I would have a tough time playing as a 1 dan in
Europe, and I play at 4 dan in clubs in Japan.

I tried playing in a club as 3 dan in China once, and got totally crushed.

My preference would be a scale that is fixed at the top, with 9 dan pros at
9 dan.  This would put 1 dan pros at about 7 dan and top amateurs at 6 dan,
with a few 7 dans.  This is pretty close to the European scale.  Top
tournament pros almost never lose to pro 1 dans, so there are 300 or more
ELO points between the top amateurs and the top professionals.

So perhaps top human play is 3500 or more on the cgos 19x19 scale.  That's
12 ranks above 2000, with the higher ranks having more ELO points per rank.

David

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:computer-go-
> [EMAIL PROTECTED] On Behalf Of Don Dailey
> Sent: Tuesday, December 11, 2007 6:37 PM
> To: computer-go
> Subject: Re: [computer-go] Where and How to Test the Strong Programs?
> 
> I feel that we probably need several more players to have much
> accuracy,  but I don't mind starting the best educated guess we can
> muster - it can be modified at a later time.
> 
> How do AGA ratings compare to other systems?   Is any particular system
> considered (defacto or otherwise) more of a "standard" than some other?
> 
> How do AGA ratings compare to KGS?
> 
> - Don
> 
> 
> 
> Christoph Birk wrote:
> > It looks like my (3k AGA) CGOS rating (tast-3k) is converging around
> > 2000 ELO. That gives us a zero-point but we need at least one more
> > rated player (better more) to get the scale.
> > If you would like to use my GUI please contact me by private email
> > at ccbirk at gmail dot com.
> >
> > Christoph
> > ___
> > computer-go mailing list
> > computer-go@computer-go.org
> > http://www.computer-go.org/mailman/listinfo/computer-go/
> >
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

I feel that we probably need several more players to have much
accuracy,  but I don't mind starting the best educated guess we can
muster - it can be modified at a later time.

How do AGA ratings compare to other systems?   Is any particular system
considered (defacto or otherwise) more of a "standard" than some other?

How do AGA ratings compare to KGS?

- Don



Christoph Birk wrote:
> It looks like my (3k AGA) CGOS rating (tast-3k) is converging around
> 2000 ELO. That gives us a zero-point but we need at least one more
> rated player (better more) to get the scale.
> If you would like to use my GUI please contact me by private email
> at ccbirk at gmail dot com.
>
> Christoph
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Christoph,

Let me know when you are finished,  what name you are playing under and
I will do the bayeselo thing to get a better figure.  Also, I can
throw out any games that were irregular if you can identify them,  such
as if  a match started when you were not looking or your interface got
glitchy or something.

- Don


Christoph Birk wrote:
> It looks like my (3k AGA) CGOS rating (tast-3k) is converging around
> 2000 ELO. That gives us a zero-point but we need at least one more
> rated player (better more) to get the scale.
> If you would like to use my GUI please contact me by private email
> at ccbirk at gmail dot com.
>
> Christoph
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 11 Dec 2007, Don Dailey wrote:

Are you playing on CGOS? Did you actually build your own GUI for this?


As I wrote in a previous email, I re-used my 'myCtest' program
but replaced the 'genmove' command with a simple GUI. Just took
me a few hours.


I don't want people playing on CGOS as a general rule except under
controlled circumstance for this purpose,  but not just for fun.


Don't worry, I don't plan to play "for fun" on CGOS. I just wanted
to establish a conversion from ELO rating to 'human rank'.
Now we need 1 or 2 stronger players to get the scale.

Christoph

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Are you playing on CGOS? Did you actually build your own GUI for this?

I don't want people playing on CGOS as a general rule except under
controlled circumstance for this purpose,  but not just for fun.

I discovered that it's easy to use gtpadapter from gogui and play on
CGOS.   The only problem is that you have to be very careful to avoid
illegal moves such as KO and suicide. Probably a good player would
rarely do this but I played 2 or 3 games (not taking them seriously) and
forfeited due to playing quickly without checking around.  

- Don



Christoph Birk wrote:
> It looks like my (3k AGA) CGOS rating (tast-3k) is converging around
> 2000 ELO. That gives us a zero-point but we need at least one more
> rated player (better more) to get the scale.
> If you would like to use my GUI please contact me by private email
> at ccbirk at gmail dot com.
>
> Christoph
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

It looks like my (3k AGA) CGOS rating (tast-3k) is converging around
2000 ELO. That gives us a zero-point but we need at least one more rated 
player (better more) to get the scale.

If you would like to use my GUI please contact me by private email
at ccbirk at gmail dot com.

Christoph
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Rémi Coulom]

Christoph Birk wrote:

On Tue, 4 Dec 2007, Christoph Birk wrote:

On Tue, 4 Dec 2007, Don Dailey wrote:

It would be awkward at best.   I could build a client to do this,  but
the human would have to be willing to sit and play games at the moment
they were scheduled.


You are right ... it's very awkward. I lost one game by typo
and another by time.


I added a simple GUI to my program to play on CGOS ...
but I am not doing very well :-(

Any (dan) volunteer(s) ?

Christoph
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
I noticed handtalk on CGOS. Does anybody knows if it is the Chinese 
program ? I thought it may be a human player, because it does not play 
so many games in a row, and appeared when we talked about gtpdisplay. If 
it is really the Chinese HandTalk program, then it is stronger than I 
would have expected of a classical program.


Rémi
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 4 Dec 2007, Christoph Birk wrote:

On Tue, 4 Dec 2007, Don Dailey wrote:

It would be awkward at best.   I could build a client to do this,  but
the human would have to be willing to sit and play games at the moment
they were scheduled.


You are right ... it's very awkward. I lost one game by typo
and another by time.


I added a simple GUI to my program to play on CGOS ...
but I am not doing very well :-(

Any (dan) volunteer(s) ?

Christoph
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Wed, 5 Dec 2007, Joel Veness wrote:

I have been thinking about making a version of Goanna (~2250 on CGOS)
public, once it plays in a human friendly way.


Thanks,
Christoph
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Joel Veness]

Hi Christoph,

I have been thinking about making a version of Goanna (~2250 on CGOS)
public, once it plays in a human friendly way.

At the moment, it is nearly unusable for fun human vs computer matches
because of a lack of opening book (slow first few moves), and
ridiculous endgame play.

Considering how much time I have been putting into this project
lately, it is not going to be happening till at least the end of
January.

Joel

On Dec 5, 2007 8:13 AM, Christoph Birk <[EMAIL PROTECTED]> wrote:
> On Tue, 4 Dec 2007, Gunnar Farnebäck wrote:
> > terry mcintyre wrote:
> >> Some of the MonteGNU code was just released on CVS. Check out Gnugo's
> >> development pages.
> >
> > Don't expect that code to do better than 2000 on CGOS though
> > (mgtest2). The remaining code used by MonteGNU is still too messy.
>
> That's why I asked for 'MonteGNU'. It has a well established rating
> on CGOS (23670 games).
> I there any other (fairly) strong CGOS 9x9 program that's available
> for download AND that has an reliable rating on CGOS?
>
> Christoph
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

You must also avoid suicide moves! I also tried playing on top of an
existing stone and it didn't allow that - but any other kind of illegal
move (by cgos rules) is passed through and causes a CGOS forfeit.

There is a config file option,  perhaps there is way to configure it to
a particular set of rules?

- Don


Don Dailey wrote:
> I just tried gtpdisplay and it worked the first time!The only
> problem is that I tried to make an illegal ko move.
>
> On linux, I just put gtpdisplay as the name of the program and it
> worked.   
>
> It looks like it could also be used to watch your program play on CGOS, 
> just provide a program name as an argument to the program.
>
> I don't see any way to make it reject superko however - so you must take
> care not to make a superko move. 
>
> - Don
>
>
>
>
> Christoph Birk wrote:
>   
>> On Tue, 4 Dec 2007, Don Dailey wrote:
>> 
 Yes, that would work.
 Some humans also could play on CGOS (just for a while) to establish
 a conversion from CGOS-ELO to human-ranks.

 
>>> It would be awkward at best.   I could build a client to do this,  but
>>> the human would have to be willing to sit and play games at the moment
>>> they were scheduled.
>>>
>>> It would be better to get a handful of players to AGREE in advance to
>>> play particular programs on KGS but under very CGOS like conditions
>>> including the time control. They should know in advance they are
>>> part of the experiment and that they should not experiment but should
>>> play full strength and not abandon the games or take-back moves, etc.
>>>   
>> Yes, this would be better ... but will it happen?
>> I quickly hacked my program to allow me to make moves
>> and I am playing currently as test-3k (my AGA rating)
>> (so much for 'C' beeing slow for prototyping :-)
>>
>> Christoph
>>
>> ___
>> computer-go mailing list
>> computer-go@computer-go.org
>> http://www.computer-go.org/mailman/listinfo/computer-go/
>>
>> 
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
>   
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

I just tried gtpdisplay and it worked the first time!The only
problem is that I tried to make an illegal ko move.

On linux, I just put gtpdisplay as the name of the program and it
worked.   

It looks like it could also be used to watch your program play on CGOS, 
just provide a program name as an argument to the program.

I don't see any way to make it reject superko however - so you must take
care not to make a superko move. 

- Don




Christoph Birk wrote:
> On Tue, 4 Dec 2007, Don Dailey wrote:
>>> Yes, that would work.
>>> Some humans also could play on CGOS (just for a while) to establish
>>> a conversion from CGOS-ELO to human-ranks.
>>>
>> It would be awkward at best.   I could build a client to do this,  but
>> the human would have to be willing to sit and play games at the moment
>> they were scheduled.
>>
>> It would be better to get a handful of players to AGREE in advance to
>> play particular programs on KGS but under very CGOS like conditions
>> including the time control. They should know in advance they are
>> part of the experiment and that they should not experiment but should
>> play full strength and not abandon the games or take-back moves, etc.
>
> Yes, this would be better ... but will it happen?
> I quickly hacked my program to allow me to make moves
> and I am playing currently as test-3k (my AGA rating)
> (so much for 'C' beeing slow for prototyping :-)
>
> Christoph
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

How does it deal with other gtp commands sent to it?Perhaps it can
be used.   Maybe Christoph can experiment with it.

- Don


Rémi Coulom wrote:
> Don Dailey wrote:
>> I saw that you made an illegal move! 
>> The way to do this is to the take the viewing client and hack it.   
>> Then you would get a nice gui and legal move testing (at the least the
>> package to do legal move testing is there even if it's not being used.)
>>
>> If you are typing your moves in manually,  you could at least pull the
>> object oriented gogame package out and use it to verify the moves so you
>> don't mistype an illegal move.   It's dirt simple to use.
>>
>> I could probably do a gui since I plan to build a graphical engine
>> client anyway.But I don't really want humans playing except as a
>> special experiment. 
>>
>> - Don
> Note that the "gtpdisplay" tool that comes with gogui does this
> already. You enter moves in the GUI, and they are sent as reply to the
> "genmove" command.
>
> http://gogui.sourceforge.net/doc/reference-gtpdisplay.html
>
> Rémi
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

I was wondering if gogui could be used - it would have to emulate a go
program somehow.   But gogui is a controller, not a program.

However I know it comes with all kinds of filters to do various
things.   If it can be made to act like a go engine (where a human is
the "brains") then it could be connected to the cgo3.tcl script directly.

- Don


Rémi Coulom wrote:
> Don Dailey wrote:
>> I saw that you made an illegal move! 
>> The way to do this is to the take the viewing client and hack it.   
>> Then you would get a nice gui and legal move testing (at the least the
>> package to do legal move testing is there even if it's not being used.)
>>
>> If you are typing your moves in manually,  you could at least pull the
>> object oriented gogame package out and use it to verify the moves so you
>> don't mistype an illegal move.   It's dirt simple to use.
>>
>> I could probably do a gui since I plan to build a graphical engine
>> client anyway.But I don't really want humans playing except as a
>> special experiment. 
>>
>> - Don
> Note that the "gtpdisplay" tool that comes with gogui does this
> already. You enter moves in the GUI, and they are sent as reply to the
> "genmove" command.
>
> http://gogui.sourceforge.net/doc/reference-gtpdisplay.html
>
> Rémi
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Rémi Coulom]

Don Dailey wrote:
I saw that you made an illegal move!  

The way to do this is to the take the viewing client and hack it.
Then you would get a nice gui and legal move testing (at the least the

package to do legal move testing is there even if it's not being used.)

If you are typing your moves in manually,  you could at least pull the
object oriented gogame package out and use it to verify the moves so you
don't mistype an illegal move.   It's dirt simple to use.

I could probably do a gui since I plan to build a graphical engine
client anyway.But I don't really want humans playing except as a
special experiment.  



- Don
Note that the "gtpdisplay" tool that comes with gogui does this already. 
You enter moves in the GUI, and they are sent as reply to the "genmove" 
command.


http://gogui.sourceforge.net/doc/reference-gtpdisplay.html

Rémi
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 4 Dec 2007, Don Dailey wrote:

But I don't really want humans playing except as a
special experiment.


I agree. But it's an interesting experiment ...

Christoph


___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 4 Dec 2007, Don Dailey wrote:

It would be awkward at best.   I could build a client to do this,  but
the human would have to be willing to sit and play games at the moment
they were scheduled.


You are right ... it's very awkward. I lost one game by typo
and another by time.

Christoph

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

I saw that you made an illegal move!  

The way to do this is to the take the viewing client and hack it.
Then you would get a nice gui and legal move testing (at the least the
package to do legal move testing is there even if it's not being used.)

If you are typing your moves in manually,  you could at least pull the
object oriented gogame package out and use it to verify the moves so you
don't mistype an illegal move.   It's dirt simple to use.

I could probably do a gui since I plan to build a graphical engine
client anyway.But I don't really want humans playing except as a
special experiment.  


- Don




Christoph Birk wrote:
> On Tue, 4 Dec 2007, Don Dailey wrote:
>>> Yes, that would work.
>>> Some humans also could play on CGOS (just for a while) to establish
>>> a conversion from CGOS-ELO to human-ranks.
>>>
>> It would be awkward at best.   I could build a client to do this,  but
>> the human would have to be willing to sit and play games at the moment
>> they were scheduled.
>>
>> It would be better to get a handful of players to AGREE in advance to
>> play particular programs on KGS but under very CGOS like conditions
>> including the time control. They should know in advance they are
>> part of the experiment and that they should not experiment but should
>> play full strength and not abandon the games or take-back moves, etc.
>
> Yes, this would be better ... but will it happen?
> I quickly hacked my program to allow me to make moves
> and I am playing currently as test-3k (my AGA rating)
> (so much for 'C' beeing slow for prototyping :-)
>
> Christoph
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 4 Dec 2007, Don Dailey wrote:

Yes, that would work.
Some humans also could play on CGOS (just for a while) to establish
a conversion from CGOS-ELO to human-ranks.


It would be awkward at best.   I could build a client to do this,  but
the human would have to be willing to sit and play games at the moment
they were scheduled.

It would be better to get a handful of players to AGREE in advance to
play particular programs on KGS but under very CGOS like conditions
including the time control. They should know in advance they are
part of the experiment and that they should not experiment but should
play full strength and not abandon the games or take-back moves, etc.


Yes, this would be better ... but will it happen?
I quickly hacked my program to allow me to make moves
and I am playing currently as test-3k (my AGA rating)
(so much for 'C' beeing slow for prototyping :-)

Christoph

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Jason House]

Maybe it should be an official tournament on KGS.  We should probably
make it invitation only for bots and open to 1d+ from KGS.  For
invitation, maybe it should be 2200+ ELO bots?

Looking at http://cgos.boardspace.net/9x9/standings.html, that seems to be:
GreenPeep (2550)
Zen (2472)
MoGo (not listed, but obviously near the top)
Goanna (2251)
CrazyStone (2214, 2416?)
Leela (2205)
Valkyria (2202)

If the threshold is 2400 ELO, the list may be more reasonable:
GreenPeep (2550)
Zen (2472)
CrazyStone (2416?)
MoGo (...)

On 12/4/07, Don Dailey <[EMAIL PROTECTED]> wrote:
>
>
> Christoph Birk wrote:
> > On Tue, 4 Dec 2007, Chris Fant wrote:
> >> MoGo.  But it seems that it hasn't been playing recently (anyway, you
> >> would have had no idea of the settings and hardware used).  You could
> >> play against it on your own hardware to understand it's strength
> >> against a human, and let it get a CGOS rating using the same hardware
> >> whenever you are not playing it.
> >
> > Yes, that would work.
> > Some humans also could play on CGOS (just for a while) to establish
> > a conversion from CGOS-ELO to human-ranks.
> >
> > Christoph
>
> It would be awkward at best.   I could build a client to do this,  but
> the human would have to be willing to sit and play games at the moment
> they were scheduled.
>
> It would be better to get a handful of players to AGREE in advance to
> play particular programs on KGS but under very CGOS like conditions
> including the time control. They should know in advance they are
> part of the experiment and that they should not experiment but should
> play full strength and not abandon the games or take-back moves, etc.
>
> - Don
>
>
>
> >
> > ___
> > computer-go mailing list
> > computer-go@computer-go.org
> > http://www.computer-go.org/mailman/listinfo/computer-go/
> >
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Christoph Birk wrote:
> On Tue, 4 Dec 2007, Chris Fant wrote:
>> MoGo.  But it seems that it hasn't been playing recently (anyway, you
>> would have had no idea of the settings and hardware used).  You could
>> play against it on your own hardware to understand it's strength
>> against a human, and let it get a CGOS rating using the same hardware
>> whenever you are not playing it.
>
> Yes, that would work.
> Some humans also could play on CGOS (just for a while) to establish
> a conversion from CGOS-ELO to human-ranks.
>
> Christoph

It would be awkward at best.   I could build a client to do this,  but
the human would have to be willing to sit and play games at the moment
they were scheduled.  

It would be better to get a handful of players to AGREE in advance to
play particular programs on KGS but under very CGOS like conditions
including the time control. They should know in advance they are
part of the experiment and that they should not experiment but should
play full strength and not abandon the games or take-back moves, etc.

- Don
 


>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 4 Dec 2007, Chris Fant wrote:

MoGo.  But it seems that it hasn't been playing recently (anyway, you
would have had no idea of the settings and hardware used).  You could
play against it on your own hardware to understand it's strength
against a human, and let it get a CGOS rating using the same hardware
whenever you are not playing it.


Yes, that would work.
Some humans also could play on CGOS (just for a while) to establish
a conversion from CGOS-ELO to human-ranks.

Christoph

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Chris Fant]

MoGo.  But it seems that it hasn't been playing recently (anyway, you
would have had no idea of the settings and hardware used).  You could
play against it on your own hardware to understand it's strength
against a human, and let it get a CGOS rating using the same hardware
whenever you are not playing it.


On Dec 4, 2007 4:13 PM, Christoph Birk <[EMAIL PROTECTED]> wrote:
> On Tue, 4 Dec 2007, Gunnar Farnebäck wrote:
> > terry mcintyre wrote:
> >> Some of the MonteGNU code was just released on CVS. Check out Gnugo's
> >> development pages.
> >
> > Don't expect that code to do better than 2000 on CGOS though
> > (mgtest2). The remaining code used by MonteGNU is still too messy.
>
> That's why I asked for 'MonteGNU'. It has a well established rating
> on CGOS (23670 games).
> I there any other (fairly) strong CGOS 9x9 program that's available
> for download AND that has an reliable rating on CGOS?
>
> Christoph
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 4 Dec 2007, Gunnar Farnebäck wrote:

terry mcintyre wrote:

Some of the MonteGNU code was just released on CVS. Check out Gnugo's
development pages.


Don't expect that code to do better than 2000 on CGOS though
(mgtest2). The remaining code used by MonteGNU is still too messy.


That's why I asked for 'MonteGNU'. It has a well established rating
on CGOS (23670 games).
I there any other (fairly) strong CGOS 9x9 program that's available
for download AND that has an reliable rating on CGOS?

Christoph
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

On Tue, 4 Dec 2007, Don Dailey wrote:

note:  this is only to estimate the playing strength relative to a 19x19
player since there is no real system that makes sense for 9x9. I
would simple put this on the crosstable web pages in parenthesis.  e.g.

  Rated: 2410  (1.1d est.)


I don't think this makes sense until it's calibrated.

Christoph

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Gunnar Farnebäck]

terry mcintyre wrote:
> Some of the MonteGNU code was just released on CVS. Check out Gnugo's
> development pages.

Don't expect that code to do better than 2000 on CGOS though
(mgtest2). The remaining code used by MonteGNU is still too messy.

/Gunnar
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Let's make a wild guess.What if I made the web site report
approximate strength using the following formula:

  dan = (elo - 2300) / 100  

So a 2400 player is 1 dan,  a 2500 player is 2 dan etc.

Here is a table:

  2300 - 1.0 kyu
  2310 - 0.9 kyu
  2320 - 0.8 kyu
  ...
  2400 - 1.0 dan
  2410 - 1.1 dan

Does this scale and these constants seem reasonable?   Am I off a class
in either direction?

note:  this is only to estimate the playing strength relative to a 19x19
player since there is no real system that makes sense for 9x9. I
would simple put this on the crosstable web pages in parenthesis.  e.g.   

   Rated: 2410  (1.1d est.)


- Don






Christoph Birk wrote:
> Robert Jasiek wrote:
>>> Where can one play the latest versions of MoGo or other, similarly
>>> strong programs?
>
> Would it be possible to publish the MonteGNU code?
> If yes, then a few dan-players could play each at least 20 games
> against it and publish their results. That would allow for a
> rough estimate of MonteGNU's strength (9x9 CGOS = 2100 ELO)
> and a (first) normalization point for CGOS.
>
> Christoph
>
> ___
> computer-go mailing list
> computer-go@computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[terry mcintyre]

 Some of the MonteGNU code was just released on CVS. Check out Gnugo's 
development pages. 
 
Terry McIntyre <[EMAIL PROTECTED]>
They mean to govern well; but they mean to govern. They promise to be kind 
masters; but they mean to be masters. -- Daniel Webster

- Original Message 
From: Christoph Birk <[EMAIL PROTECTED]>
To: computer-go 
Sent: Tuesday, December 4, 2007 12:14:24 PM
Subject: Re: [computer-go] Where and How to Test the Strong Programs?


Robert Jasiek wrote:
>> Where can one play the latest versions of MoGo or other, similarly
 strong 
>> programs?

Would it be possible to publish the MonteGNU code?
If yes, then a few dan-players could play each at least 20 games
against it and publish their results. That would allow for a
rough estimate of MonteGNU's strength (9x9 CGOS = 2100 ELO)
and a (first) normalization point for CGOS.

Christoph

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/






  

Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  
http://tools.search.yahoo.com/newsearch/category.php?category=shopping___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Rémi Coulom]

Rémi Coulom wrote:

Hi,

13x13 StoneCrazy is currently connected to CGOS (computer go room). It 
will stay there for about 24h.


Rémi 
So far, it lost 1 game against 3d, and 2 games against 2d. In this game, 
it started a nice ko fight at move 69 (but lost):

http://files.gokgs.com/games/2007/12/4/BadRobot-StoneCrazy.sgf

Semeais in the corner with a nakade generate huge catastrophes. Unless I 
fix this, it will still lose to 5k players when they occur, whatever 
computational power I use.


I have also connected Crazy Stone on a core 2 duo to 9x9 CGOS, and will 
leave it there for a while.


Rémi
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Christoph Birk]

Robert Jasiek wrote:
Where can one play the latest versions of MoGo or other, similarly strong 
programs?


Would it be possible to publish the MonteGNU code?
If yes, then a few dan-players could play each at least 20 games
against it and publish their results. That would allow for a
rough estimate of MonteGNU's strength (9x9 CGOS = 2100 ELO)
and a (first) normalization point for CGOS.

Christoph

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[David Doshay]

At the Cotsen Open the encouragement is a prize for the best program.

It has not been very satisfying for me to have SlugGo win it the past
two years by the default of being the only program present. I would
be much happier to have others show up too. I have heard from one
programmer who says he will enter a program next year.

In the past the prize was cash, but at my request they changed it to
a trophy this year ... money disappears but trophies last. If others
are going to enter programs next year I encourage them to tell the
organizers which is more valuable to them.

Cheers,
David



On 4, Dec 2007, at 10:27 AM, Robert Jasiek wrote:


David Doshay wrote:

When tournament organizers allow and encourage it!


Some (local) European tournaments would allow it. (Some have  
already done it.) "Encourage" - not yet :)


--
robert jasiek
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Robert Jasiek]

David Doshay wrote:

When tournament organizers allow and encourage it!


Some (local) European tournaments would allow it. (Some have already 
done it.) "Encourage" - not yet :)


--
robert jasiek
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[David Doshay]

On 4, Dec 2007, at 3:59 AM, Robert Jasiek wrote:


When will we see the strong programs entering real world tournaments?



When tournament organizers allow and encourage it! At this time AGA  
rules are that games against computers are not counted in a human  
player's ranking.


The Cotsen Open allows programs to enter, so I am there every year,  
with SlugGo presently entered as a 10 kyu on a full board. SlugGo has  
won against 8 Kyu and lost to 12 kyu players.


I am expecting that there will be something happening at the US Go  
Congress 2008 in Portland.


Cheers,
David

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Robert Jasiek wrote:
> Where can one play the latest versions of MoGo or other, similarly
> strong programs? It is said that some programs are on KGS, but I
> cannot find them. How to find them? Is it possible to play against
> them as a human on CGOS? I, German 5d, would want to play even games
> on 19x19, 13x13, or 9x9 to learn more about their current playing
> level. How would that be used for assessing the program's strength?
> E.g., KGS ratings are inaccurate by +-600; the standard deviation of
> an EGF high dan rating I subjectively estimate as roughly +-70. So IMO
> ratings are not useful to assess a program's strength against humans.
> When will we see the strong programs entering real world tournaments?
>
It would be nice to "normalize" the ratings of CGOS to human terms.   If
we had a great deal of data points,  we could construct an ELO to
kyu/dan  formula which could be posted along with the ELO ratings on the
web page. For instance here is a stab:

dan = (elo - 2400) / 100

The issue is which constants to use and is this form accurate enough to
encompass a fairly large range of rankings?   Can it be reasonably
accurate at 15k and still accurate at 2 Dan if the constants are
correctly chosen or does it need to be more sophisticated? At any
rate, I'm more interested in accuracy near the levels of the top
programs.  

Since we are comparing to human players and also comparing 19x19
rankings to 9x9 rankings, this is a little dicey - but it will still
work.What we want to know is how a given program is likely to do
against a particular human player and this is just an estimate.The
human player is of course assumed to be a 19x19 rated player.

The other problem is that there is no universal calibration - a 1 Dan
KGS player may be 2 Dan in some other system.But perhaps KGS is
universal enough for our needs.

We can rate any number of players on KGS by just playing them for a
hundred games or so - but a lot of games are aborted,  or time added or
take-backs are made, etc.  Also, the games should be played at the
same time control as on CGOS.

- Don


___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Rémi Coulom]

Robert Jasiek wrote:
Where can one play the latest versions of MoGo or other, similarly 
strong programs? It is said that some programs are on KGS, but I 
cannot find them. How to find them? Is it possible to play against 
them as a human on CGOS? I, German 5d, would want to play even games 
on 19x19, 13x13, or 9x9 to learn more about their current playing 
level. How would that be used for assessing the program's strength? 
E.g., KGS ratings are inaccurate by +-600; the standard deviation of 
an EGF high dan rating I subjectively estimate as roughly +-70. So IMO 
ratings are not useful to assess a program's strength against humans. 
When will we see the strong programs entering real world tournaments?



Hi,

13x13 StoneCrazy is currently connected to CGOS (computer go room). It 
will stay there for about 24h.


Rémi
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Jason House]

On 12/4/07, Chris Fant <[EMAIL PROTECTED]> wrote:
> > What I consider more of an issue is that MoGo seems to be very
> > sensitive to (undocumented) configuration options.  Such issues
> > probably exist with all engines.  It'd probably be smarter to set up a
> > day where strong bots would connect to CGOS and invite dan-level
> > players to challenge them.
>
> You mean KGS, right?  I don't think humans on CGOS are an appropriate
> direction for that server.

Yes, KGS, sorry.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Chris Fant]

> What I consider more of an issue is that MoGo seems to be very
> sensitive to (undocumented) configuration options.  Such issues
> probably exist with all engines.  It'd probably be smarter to set up a
> day where strong bots would connect to CGOS and invite dan-level
> players to challenge them.

You mean KGS, right?  I don't think humans on CGOS are an appropriate
direction for that server.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Jason House]

On 12/4/07, Chris Fant <[EMAIL PROTECTED]> wrote:
> > > Where can one play the latest versions of MoGo or other, similarly
> > > strong programs?
> >
> > But Mogo is now a free program.You can get a copy, find some good
> > hardware and play at 9x9 and 19x19.
> >
>
> But the released version is probably not the latest.

Most strong bots will probably re-release when significant
enhancements have been made.  I think recent MoGo updates have been to
support enhanced hardware (AKA, use MPI).  On a normal computer, I'd
assume it plays at about the same level.

What I consider more of an issue is that MoGo seems to be very
sensitive to (undocumented) configuration options.  Such issues
probably exist with all engines.  It'd probably be smarter to set up a
day where strong bots would connect to CGOS and invite dan-level
players to challenge them.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Chris Fant]

> > Where can one play the latest versions of MoGo or other, similarly
> > strong programs? It is said that some programs are on KGS, but I
> > cannot find them. How to find them? Is it possible to play against
> > them as a human on CGOS?
> CGOS is designed for computer/computer only.You could modify the
> client to accept moves or build a client and pretend you are a bot - but
> you have no control over the scheduling algorithm - you would be forced
> to accept whatever pairing and color was assigned to you.
>
> But Mogo is now a free program.You can get a copy, find some good
> hardware and play at 9x9 and 19x19.
>

But the released version is probably not the latest.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Where and How to Test the Strong Programs?

[Don Dailey]

Robert Jasiek wrote:
> Where can one play the latest versions of MoGo or other, similarly
> strong programs? It is said that some programs are on KGS, but I
> cannot find them. How to find them? Is it possible to play against
> them as a human on CGOS? 
CGOS is designed for computer/computer only.You could modify the
client to accept moves or build a client and pretend you are a bot - but
you have no control over the scheduling algorithm - you would be forced
to accept whatever pairing and color was assigned to you.

But Mogo is now a free program.You can get a copy, find some good
hardware and play at 9x9 and 19x19.

- Don




> I, German 5d, would want to play even games on 19x19, 13x13, or 9x9 to
> learn more about their current playing level. How would that be used
> for assessing the program's strength? E.g., KGS ratings are inaccurate
> by +-600; the standard deviation of an EGF high dan rating I
> subjectively estimate as roughly +-70. So IMO ratings are not useful
> to assess a program's strength against humans. When will we see the
> strong programs entering real world tournaments?
>
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/