Re: [computer-go] erm...
There's some value to human-human games in this proposed tournament, I think. Some humans might play or worse at 5 minute time controls. Comparison with longer games might be interesting. Terry McIntyre <[EMAIL PROTECTED]> They mean to govern well; but they mean to govern. They promise to be kind masters; but they mean to be masters. -- Daniel Webster - Original Message From: steve uurtamo <[EMAIL PROTECTED]> To: computer-go Sent: Wednesday, December 5, 2007 8:18:34 AM Subject: Re: [computer-go] erm... two followups, and i'm sorry for not referencing the original notes directly: i) i agree that 9x9 has fewer standard deviations of skill. there's simply less to be good at (ladders are tiny, life and death can only be so large, the difference between influence and territory is skewed, etc.). this doesn't mean that "dan level play" is meaningless, and i'm sorry for making that implication. ii) i do think that organized play by a range (not too low, or people will get irritated) of bots against a large batch of humans with solid kgs ranks is a good idea. a tournament is the best way to do this, and i don't even think that strictly human->bot games are necessary: human->human games will give us quite a bit of information about 9x9 "ranks" as well, and how the conversion should go. that data is probably at least as useful as the human->bot games. also, there's no real reason to exclude bot->bot games on kgs. it'll provide equally good data about how a tournament works. to this end, if anyone wants to organize a 9x9 tournament on kgs (i'm not sure how this is done) where computer players are allowed (encouraged) to play, with cgos-like time controls, i'd be happy to do the ELO->rank analysis after the fact (although i imagine that jacques would do a better job. :) the more games the better, and with reasonable time controls, there's no reason that this couldn't be something like the ironman tournaments that kgs occasionally runs, in the sense that a multi-day tournament during various times of the day would provide for a rich and wide variety of players. i'd ignore everything but the game results, of course, and would just need kgs rank at start of tournament for each player, and elo information at start of tournament for each bot. s. Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
Nick Wedd wrote: > In message <[EMAIL PROTECTED]>, steve uurtamo > <[EMAIL PROTECTED]> writes > > < snip > > >> to this end, if anyone wants to organize a 9x9 tournament on >> kgs (i'm not sure how this is done) where computer players >> are allowed (encouraged) to play, with cgos-like time controls, > > I think that only a few people on KGS have the power to run > tournaments. I am one of those people, and I have been assigned the > power only so that I can run bot tournaments. I believe that an event > which included a minority of human players would be allowed (though of > course I shall accept wms' view on this). For our purposes, this should be an invitational tournament. It should include well known bots that "we believe" are in the low kyu - low dan range - perhaps gnugo at the bottom end. Then it should included well known stable/consistent human players. They should have well established ratings. Preferably it should be bot vs human, but I seriously doubt KGS has a provision for this. It should be composed of many rounds - so perhaps it should be a round robin tournament. > > I assume that if I organise a tournament in the way that I do bot > tournaments, but without the "robot" restriction, both bots and humans > will be able to play. But I'm not sure about any of this, so I would > start by running a test tournament, with fast time limits so that not > too much time is wasted when things go wrong. > > I do not have the power to tell it not to pair humans with humans. > > I cannot set cgos-like time controls, with their element of Fischer > time. CGOS doesn't use Fischer time - it is absolute time control. (1/4 second is added to the clock for each move - but I don't think that is particularly relevant.Just set the time control for 5 minutes total time per player. - Don > The nearest thing that KGS supports is absolute time plus a very fast > overtime (either Canadian or byo-yomi). > > Nick > >> i'd be happy to do the ELO->rank analysis after the fact (although >> i imagine that jacques would do a better job. :) >> >> the more games the better, and with reasonable time controls, >> there's no reason that this couldn't be something like the ironman >> tournaments that kgs occasionally runs, in the sense that a >> multi-day tournament during various times of the day would provide >> for a rich and wide variety of players. i'd ignore everything but >> the game results, of course, and would just need kgs rank at start >> of tournament for each player, and elo information at start of >> tournament for each bot. > ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
On Wed, 5 Dec 2007, Jacques Basaldúa wrote: The correct answer should be: dan does not exist for board sizes other than 19. And it is impossible to define a consistent scale made of 30 levels for 9x9. Perhaps 9x9 has only 10 or 12 levels. Defining a step in level as being a standard deviation above the other player in the normal distribution that justifies the Elo system. I think you miss my point. I am just trying to measure how many ELO points would the averatge X dan human get on CGOS. Christoph ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
In message <[EMAIL PROTECTED]>, steve uurtamo <[EMAIL PROTECTED]> writes < snip > to this end, if anyone wants to organize a 9x9 tournament on kgs (i'm not sure how this is done) where computer players are allowed (encouraged) to play, with cgos-like time controls, I think that only a few people on KGS have the power to run tournaments. I am one of those people, and I have been assigned the power only so that I can run bot tournaments. I believe that an event which included a minority of human players would be allowed (though of course I shall accept wms' view on this). I assume that if I organise a tournament in the way that I do bot tournaments, but without the "robot" restriction, both bots and humans will be able to play. But I'm not sure about any of this, so I would start by running a test tournament, with fast time limits so that not too much time is wasted when things go wrong. I do not have the power to tell it not to pair humans with humans. I cannot set cgos-like time controls, with their element of Fischer time. The nearest thing that KGS supports is absolute time plus a very fast overtime (either Canadian or byo-yomi). Nick i'd be happy to do the ELO->rank analysis after the fact (although i imagine that jacques would do a better job. :) the more games the better, and with reasonable time controls, there's no reason that this couldn't be something like the ironman tournaments that kgs occasionally runs, in the sense that a multi-day tournament during various times of the day would provide for a rich and wide variety of players. i'd ignore everything but the game results, of course, and would just need kgs rank at start of tournament for each player, and elo information at start of tournament for each bot. -- Nick Wedd[EMAIL PROTECTED] ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
two followups, and i'm sorry for not referencing the original notes directly: i) i agree that 9x9 has fewer standard deviations of skill. there's simply less to be good at (ladders are tiny, life and death can only be so large, the difference between influence and territory is skewed, etc.). this doesn't mean that "dan level play" is meaningless, and i'm sorry for making that implication. ii) i do think that organized play by a range (not too low, or people will get irritated) of bots against a large batch of humans with solid kgs ranks is a good idea. a tournament is the best way to do this, and i don't even think that strictly human->bot games are necessary: human->human games will give us quite a bit of information about 9x9 "ranks" as well, and how the conversion should go. that data is probably at least as useful as the human->bot games. also, there's no real reason to exclude bot->bot games on kgs. it'll provide equally good data about how a tournament works. to this end, if anyone wants to organize a 9x9 tournament on kgs (i'm not sure how this is done) where computer players are allowed (encouraged) to play, with cgos-like time controls, i'd be happy to do the ELO->rank analysis after the fact (although i imagine that jacques would do a better job. :) the more games the better, and with reasonable time controls, there's no reason that this couldn't be something like the ironman tournaments that kgs occasionally runs, in the sense that a multi-day tournament during various times of the day would provide for a rich and wide variety of players. i'd ignore everything but the game results, of course, and would just need kgs rank at start of tournament for each player, and elo information at start of tournament for each bot. s. Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm.../ CGoTournaments
Hi all, this discussion about ratings and playing strength of Go programs on 19x19 and 9x9 with and without handicap stones etc. is exactly the field I try to find an approach in my tournaments! Most classic Go programs were designed only to be as strong as possible in 19x19 and seamed to be much weaker in 9x9. Most newer Go programs were designed to be as strong as possible in 9x9 and seam to be much weaker in 19x19. Meanwhile some programmers make efforts to optimize their Go programs for all three boardsizes 9x9, 13x13 and 19x19 so there could be NOW a new possibility to use these programs and their playing results to propose a new standard correlation of the kyu and dan grades between the different boardsizes and with different komi / handicap stones! I remember when HandTalk won my first 9x9 tournament in 2000 the author Chen Zhixing sent me his 2001version of GoeMate for further tournaments and answered, it was never optimized for 9x9 and it should be possible to make a much stronger 9x9 Go program - perhaps he even tried some efforts in this direction, I never heard more of him since... All the years, GoeMate from Chen Zhixing and Go++ from M.Reiss were the two strongest 19x19 Go programs in the world but important tournaments with motivating prize money became rare. In my now ongoing tournament with 100 games/pairing for 19x19 I started to let play GnuGo 3.7.11 on different levels (10 and 15) against Go++ version 5.0, GoeMate2001, FunGo, ManyFaces v11 and some other older Go programs. I plan reporting results+games on a special web site I am working on. In 2008 I want to include some of the newer available Go programs like MoGo or Leela or ask for new versions of all the other new top Go programs around...! My aim is to add one new version or new program with up to 1000 19x19 games played every month, to have my new 9x9 tournament continue running aside with the same engines and to let play weaker programs again with handicap! Stefan Jacques Basaldúa wrote: > > Don Dailey wrote: > > The assumption is that 1 Dan 19x19 = 1 Dan > > 9x9 and on average this will be true. > > That is precisely my point. It is a very strong assumption because > 19x19 has lots of things: joseki study, global board evaluation, etc. > that do not exist in 9x9. This assumption is only an approximation. It > is like guessing that a world champion 100 meter runner will also be > a world champion long jump athlete. That may happen (Carl Lewis), but > usually it is not the case, although the correlation is evident. > Perhaps the word champion in 19x19 is in the top-50 at 9x9. Nobody will > be surprised if a 3 dan beats a 5 dan consistently in 9x9, because it > is a different game, but of course, a 10 kyu won't have a chance. > > The correct answer should be: dan does not exist for board sizes other > than 19. And it is impossible to define a consistent scale made of 30 > levels for 9x9. Perhaps 9x9 has only 10 or 12 levels. Defining a step > in level as being a standard deviation above the other player in the > normal distribution that justifies the Elo system. > > (BTW. The best proof that the 19x19 level does not match the 9x9 > level are the UCT programs themselves.) > > Jacques. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
Don Dailey wrote: The assumption is that 1 Dan 19x19 = 1 Dan 9x9 and on average this will be true. That is precisely my point. It is a very strong assumption because 19x19 has lots of things: joseki study, global board evaluation, etc. that do not exist in 9x9. This assumption is only an approximation. It is like guessing that a world champion 100 meter runner will also be a world champion long jump athlete. That may happen (Carl Lewis), but usually it is not the case, although the correlation is evident. Perhaps the word champion in 19x19 is in the top-50 at 9x9. Nobody will be surprised if a 3 dan beats a 5 dan consistently in 9x9, because it is a different game, but of course, a 10 kyu won't have a chance. The correct answer should be: dan does not exist for board sizes other than 19. And it is impossible to define a consistent scale made of 30 levels for 9x9. Perhaps 9x9 has only 10 or 12 levels. Defining a step in level as being a standard deviation above the other player in the normal distribution that justifies the Elo system. (BTW. The best proof that the 19x19 level does not match the 9x9 level are the UCT programs themselves.) Jacques. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
Don Dailey wrote: > you can still judge the quality of your opponent by looking at his 19x19 KGS ranking. Rather by looking at his real world ranking. A human real world rank may be off by 1 while a human KGS rank may be off by 6 ranks. -- robert jasiek ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
Darren Cook wrote: >> 9x9 games is a bit silly. it doesn't actually capture any extra >> information about the program, since there's no such thing as >> a 9x9 rank to compare with/against, much less a dan rank. >> > > I disagree. In my studies of 9x9, over a number of years, the human > 19x19 rank generally carries over to 9x9. E.g. a 9p consistently beats a > 3p, a 6d consistently beats a 3d, a 1d consistently beats a 5k. Whether > they are playing at 19x19 or 9x9. > That's my assumption too. Although it's possible some players are better at one board size than another I don't believe the effect is likely to be major. Even if a strong player never played a 9x9 game I believe he would quickly adapt to it after 2 or 3 games. > (As an aside, my conclusion from this, which I personally think is very > important, is that the main element of human go strength is shape, > tesuji and life/death reading.) > > UCT programs are a bit different, in that (as they are "brute-forcing" > in the number of legal moves) their strength does not scale up to higher > board sizes in the same way human strength does. But no-one is claiming > otherwise which is why people are referring to the 9x9 strength and the > 19x19 strength of Mogo and CrazyStone as separate things. > It's interesting to me that 19x19 go programs start out weaker relative to humans than 9x9 go programs. But I ran experiments that indicated wonderful scaling behavior in 19x19 go. If anything they improved MORE with depth. Of course I could only test at the low end of the skill range with a weak UCT program (an older Lazarus version.) The game at 19x19 is more difficult for computers, but this is very true of humans too. It's possible that strong humans just don't specialize in 9x9 go so perhaps this is a partial explanation.I think the good UCT programs are about 2-3 kyu lower at 19x19 relative to humans. Could it be that really it's the humans weaker at 9x9 since strong humans don't really specialize in 9x9 go? - Don > But, when we say Mogo is 3d at 9x9, it is fair to say that a human > player with a 3d rank at 19x19 will enjoy a challenging game with it. > > Darren > > ___ > computer-go mailing list > computer-go@computer-go.org > http://www.computer-go.org/mailman/listinfo/computer-go/ > > ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
> 9x9 games is a bit silly. it doesn't actually capture any extra > information about the program, since there's no such thing as > a 9x9 rank to compare with/against, much less a dan rank. I disagree. In my studies of 9x9, over a number of years, the human 19x19 rank generally carries over to 9x9. E.g. a 9p consistently beats a 3p, a 6d consistently beats a 3d, a 1d consistently beats a 5k. Whether they are playing at 19x19 or 9x9. (As an aside, my conclusion from this, which I personally think is very important, is that the main element of human go strength is shape, tesuji and life/death reading.) UCT programs are a bit different, in that (as they are "brute-forcing" in the number of legal moves) their strength does not scale up to higher board sizes in the same way human strength does. But no-one is claiming otherwise which is why people are referring to the 9x9 strength and the 19x19 strength of Mogo and CrazyStone as separate things. But, when we say Mogo is 3d at 9x9, it is fair to say that a human player with a 3d rank at 19x19 will enjoy a challenging game with it. Darren ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
I would like to add that we are calibrating against 19x19 players. Even though their ratings are based on 19x19 play we just want a mapping from 19x19 dan to 9x9 cgos. The assumption is that 1 Dan 19x19 = 1 Dan 9x9 and on average this will be true. We don't expect to get a perfect table out of this - just a rough rule of thumb. I would mention that KGS in a sense does this. When you play 9x9 they still display your rating even if you believe there is no correlation or relevance.Of course they don't rate 9x9 games but you can still judge the quality of your opponent by looking at his 19x19 KGS ranking. - Don Don Dailey wrote: > But it does have real meaning. People talk about Dan level 9x9 go > programs and so all I'm looking for is a way to instrument this in a > meaningful way. > > If a 9x9 program is estimated to be 2 dan on CGOS, it means a typical 1 > dan player will lose to it and a typical 3 dan player will beat it. > It's a matter of adjusting the constants to make sure this is so. > > We cannot use the kyu/dan system for handicap on 9x9, it doesn't work > and that's understood - we are just trying to get a sense of how these > 9x9 programs compare to humans. > > - Don > > > > > ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
But it does have real meaning. People talk about Dan level 9x9 go programs and so all I'm looking for is a way to instrument this in a meaningful way. If a 9x9 program is estimated to be 2 dan on CGOS, it means a typical 1 dan player will lose to it and a typical 3 dan player will beat it. It's a matter of adjusting the constants to make sure this is so. We cannot use the kyu/dan system for handicap on 9x9, it doesn't work and that's understood - we are just trying to get a sense of how these 9x9 programs compare to humans. - Don steve uurtamo wrote: > not to put too fine a point on it, but "estimating dan ranks" via > 9x9 games is a bit silly. it doesn't actually capture any extra > information about the program, since there's no such thing as > a 9x9 rank to compare with/against, much less a dan rank. > > ELO works well because it's strictly arbitrary relative to the > anchor. human kyu/dan ranks are arbitrary as well, but there > are thousands of people feeding ELO (or something like it) in > and out of the system, and there's an easily measured "top end". > > i'm just saying, it will look a bit silly -- imagine if when chess > programs started to completely dominate something like 5x5 chess, > the strongest programs were labeled "master level play". > > well, sure, at 5x5 chess... > > there's a reason that KGS won't assign a rank to a player that > only plays 9x9 games. the rating system isn't really built for > a board that small, in the sense that handicaps are massively > nonlinear at that size. the main motivation for kyu/dan rankings > is to establish handicap stones, after all... > > s. > > > > > > > Looking for last minute shopping deals? > Find them fast with Yahoo! Search. > http://tools.search.yahoo.com/newsearch/category.php?category=shopping > ___ > computer-go mailing list > computer-go@computer-go.org > http://www.computer-go.org/mailman/listinfo/computer-go/ > > ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] erm...
On Tue, 4 Dec 2007, steve uurtamo wrote: not to put too fine a point on it, but "estimating dan ranks" via 9x9 games is a bit silly. it doesn't actually capture any extra information about the program, since there's no such thing as a 9x9 rank to compare with/against, much less a dan rank. I don't claim that this "9x9 CGOS rank" will be equal to ranks on 19x19. But it should work the other way round. If several "human ranked players" played on CGOS and established an ELO rating we would have a good estimate of how strong GGOS ratings are. Christoph ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
[computer-go] erm...
not to put too fine a point on it, but "estimating dan ranks" via 9x9 games is a bit silly. it doesn't actually capture any extra information about the program, since there's no such thing as a 9x9 rank to compare with/against, much less a dan rank. ELO works well because it's strictly arbitrary relative to the anchor. human kyu/dan ranks are arbitrary as well, but there are thousands of people feeding ELO (or something like it) in and out of the system, and there's an easily measured "top end". i'm just saying, it will look a bit silly -- imagine if when chess programs started to completely dominate something like 5x5 chess, the strongest programs were labeled "master level play". well, sure, at 5x5 chess... there's a reason that KGS won't assign a rank to a player that only plays 9x9 games. the rating system isn't really built for a board that small, in the sense that handicaps are massively nonlinear at that size. the main motivation for kyu/dan rankings is to establish handicap stones, after all... s. Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
