On 6/16/07, Don Dailey <[EMAIL PROTECTED]> wrote:
On Fri, 2007-06-15 at 23:54 +0200, Erik van der Werf wrote:
> So far, Steenvreter has never played on CGOS. I'm very busy with work,
> so it will take a while before I have time to put it up for some
> games. Also to be honest, I'm not really that interested. I guess CGOS
> is nice if you have no other way to evaluate the strength of your
> program, but I really like it much more to play in a tournament like
> the Computer Olympiad where I can meet other programmers face to face.
It sounds almost like you are afraid to play on CGOS. After winning a
tournament you may feel that you have a reputation to protect.
I understand it may look that way, but really the primary problem is
lack of time. Coming week I have to be in Sweden, then a conference in
Germany, and further a lot of other small things to do in between.
These things were much easier when I was working full time at
university...
I personally feel the results of hundreds of games on CGOS are more
valid than 10 games no matter how much importance or prestige is
attached to a particular tournament. 10 games just isn't statistically
significant.
The tournament was double round robin, so that's 18 games.
On the first day I lost one game because of a stupid bug, and won 2
because I was lucky to meet a weak opponent. I was only able to
recover from the unnecessary loss against Go Intellect because Crazy
Stone managed to crush Mogo twice in the normal rounds. Also Mango
could probably have done much better if it hadn't suffered from some
weird bugs, so clearly the outcome could have been quite different...
Having said that, I agree with you 100 percent about your preference.
There is nothing that compares to a real face to face tournament for the
pure excitement of it. The stress of each move is powerful motivator
for improving your program. I am also a strong supporter of ICGA having
been a member off and on since it was ICCA in the early days. I really
loved the experiences of playing in the Computer Chess tournaments they
organized and the people I got to know.
Your result was excellent, but the results were close, only a 1 game
difference between the top 3 finishers doesn't "evaluate" a program.
If you played that same tournament over and over you would very likely
see Mogo and CrazyStone winning many of them too. There is no way to
say who is better especially since the hardware isn't even equal,
Steenvreter running on the most powerful of the 3 top finishers.
You're completely right. I think Mogo and Crazystone play stronger in
the beginning of the game, and my program has to be a bit lucky to get
them into a difficult fight.
This is not something that can't be solved, I know exactly what I have
to do, but I just need time. Steenvreter was really a rush job,
hacking things together until the last day before the tournament and
no time to test properly. I was hoping to be able to catch up with the
stronger programs, but never expected it to win the tournament.
When I played in the ICCA organized tournaments we used to privately
laugh (even though we absolutely loved those well organized tournaments)
that people actually believed it was about "proving who was the best
computer player in the world." In 1993 my program won the
International Computer Chess Championship and we privately joked about
that too - we knew that we had played 2 programs that were at least a
little better and one that was MUCH better. We "lucked out" by getting
a draw against the program that was much better and wining against the 2
that we believed had a statistically better chance of beating us. We
knew we were not the best, but we prepared the program to win. We knew
it was about being good enough that we had a chance to win. A weak
program has almost no chance, but to have a "reasonable" chance of
winning a tournament you must be good enough to consistently beat the
weaker players and not so weak that you have little chance of beating 1
or 2 strong players. In a 5 round tournament which was typically what
we played in, you may only have to face 1 strong program and if you
have a chance of beating it, you can win a tournament - or someone else
may knock it out so that you don't have to.
Winning a 10 round tournament of course if much more impressive. But
it's still not a very valid indicator of who is the better player.
We were in Hong Kong one year and I think Murray Campbell, one of the
Deep Blue authors told me that they estimated their chances of winning
to be about 50%. At first that sounded ludicrous because they were the
heavy favorites and everyone knew their program was far better than any
other program. But there calculations were roughly correct. Being
best doesn't mean you will win. The problem is that you usually have to
win almost every game. If you are statistically likely to win any
particular game, you are much less statistically likely to put a string
of wins together. On the other hand, out of many program playing it's
not unlikely that one of them will get a bit lucky and put a string of
wins together. The more programs in a tournament, the less likely the
best one will win. As it turned out, they did NOT win that
tournament. I found it pretty cool that this was a surprise to most of
the players except for them.
:-)
Erik
_______________________________________________
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
