Re: [computer-go] Re: Hardware limits
Hi Joshua, Yes, I think it was implicitly understood that these chess competitions were about creating the best chess playing (non-human) entity. However human nature being what it is we attach the author to the program and judge the author through his program. However, if you create a really strong program even on modest hardware, opportunities open up. Some company, or in my case a university, will approach you and give you some support. - Don On Thu, 2009-01-15 at 00:16 -0500, Joshua Shriver wrote: When I was big into Chess programming this was a sore topic for me as well. I felt it was unfair for people competing in the WCCC to win if they had a cluster of of 100 PCs, a Cray, etc, when another person was using a P200mhz. I believe it was Dr. Hyatt that said this and it made a lot of sense to me It's not just about creating the best chess program, it's creating the best playing machine So when you look at competitions that dont have hardware limits, you can't look at it like Engine X is the best in the world. You have to look at it like Engine X + this hardware setup is the best in the world; and take it with a grain of salt. Even if you did set hardware limits, it would be a hard task. Even if it's just a single PC do you use single core or multicore? Multicore would help those who have parallelized their code but hurt others. Is it fair? There's no real line that can be drawn, because on the flip side I wouldnt find it fair if I had written a parallel engine and spent all that time and effort to only be limited to 1 core. Just my $0.02. -Josh For now I tend to be of the opinion that in competitions, one should be able to bring your own hardware or run on standard hardware provided by organizers. The restriction that the hardware be physically present allows for enough flexibility that people or teams can try different set-ups (like a row of PS3s) while avoiding having people with access to a big cluster compete with people who only have access to a PC. But similarly to the competition of building the most powerful computer in the world, I can see room for a competition between big clusters that play Go as well. One doesn't have to be to the exclusion of the other. Think of car-racing. You have drag-racing where they use rockets to cross half a mile as fast as possible and you have F1-racing where the 'hardware' is constrained within certain limits. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
The thing about computer chess is that the swift do not always win the race. Many times in the past modest hardware has beaten powerful hardware. Even Deep Blue didn't always win the tournaments it played in. They came to one competition and Campbell told me that they had estimated their winning chances at about 50/50 - which was far ahead of any of the other competitors. As it turns out, Fritz won that particular time on one of the least powerful machines that was at that tournament. - Don On Wed, 2009-01-14 at 22:27 -0800, Dave Dyer wrote: Lets look at it another way - no one would care what hardware you choose to use, unless you win. So at the very least, you ought to be able to use arbitrary hardware until it becomes established that only that class of hardware can win. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
my biased $0.02: i don't think that the point is to call it even. someone's got to win, and everyone else has to come in = 2nd place. moreover, pretending as if this is the kind of contest that can be won with money (or hardware) alone is just sour grapes. one way to make this a contest about algorithms would be to require everyone's code to run on identical hardware under the identical operating system. how much fun would that be? not very much at all. imagine that you're used to being able to fit a very frequently-accessed table entirely in ram. in fact, many of your other data structures and code flow are built around the fact that it fits entirely in ram on your box. then imagine that the contest hardware has less ram and that you get to spend 90% of your thinking time watching the machine swap, or rewrite all of your code. no thanks. sure, this is the opposite of the problem that is being described -- instead of it being a sad story for the guy who has tiny hardware, it's a sad story for the guy who has access to better hardware. in neither case is it a really sad story, however. it's just that arbitrary limits always cause problems for somebody. the *only* time where hardware could really become an issue is if everyone competing is using algorithms all of which scale at roughly the same rate, all of which parallelize at roughly the same rate, etc. talk about a boring contest! this isn't an asymptotic problem requiring an algorithmic solution. this is a fixed-size board requiring a best of show answer. whoever gets there, however they get there, deserves to win, as long as the machines are choosing their own moves. s. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
On Thu, Jan 15, 2009 at 8:47 AM, steve uurtamo uurt...@gmail.com wrote: this isn't an asymptotic problem requiring an algorithmic solution. this is a fixed-size board requiring a best of show answer. whoever gets there, however they get there, deserves to win, as long as the machines are choosing their own moves. scalability is still essential and beautiful. the unlimited class winner will always be important. hardware access is very unevenly distributed. yet access to hardware currently matters in every tournament. some people like that part of the game, some don't. there is the potential for people in this community to speak up and create a culture that rewards good algorithm design as much as possible. to do so requires at least noticing hardware power. -- - Ryan ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
RE: [computer-go] Re: Hardware limits
Still, in the unlimited class of a rally from Bagdad to Beijing, it is probably not the dragracer nor the solar car that wins, but the landrover with good road maps and a GPS. I agree with most posters that in the end, you have to find the best thing to do the job. Hardware could play a major role in your solution, but you need more than just that to win. Dave Van: computer-go-boun...@computer-go.org namens Ryan Grant Verzonden: vr 16-1-2009 3:55 Aan: computer-go Onderwerp: Re: [computer-go] Re: Hardware limits On Thu, Jan 15, 2009 at 8:47 AM, steve uurtamo uurt...@gmail.com wrote: this isn't an asymptotic problem requiring an algorithmic solution. this is a fixed-size board requiring a best of show answer. whoever gets there, however they get there, deserves to win, as long as the machines are choosing their own moves. scalability is still essential and beautiful. the unlimited class winner will always be important. hardware access is very unevenly distributed. yet access to hardware currently matters in every tournament. some people like that part of the game, some don't. there is the potential for people in this community to speak up and create a culture that rewards good algorithm design as much as possible. to do so requires at least noticing hardware power. -- - Ryan ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
Programmers work on all kinds of hardware. Making them port their code to some arbitrary standard platform is not a great idea. Just as one voice, I will not bother to port my code to a different box. So, if the competitions are all on the same hardware you are running a *Go -playing-programs-developed-on-that-platform* competition. And that sounds silly to me. I have been working on Go on moderate sized clusters (25 - 75 CPUs) for about 5 years. Most of my work has shown that it is not trivial to write a program that scales reasonably over those processors. It is far from trivial to develop and debug programs that run on clusters. More hardware is no guarantee of a stronger program and is not a simple way to get one. Cheers, David On 13, Jan 2009, at 6:26 AM, Petr Baudis wrote: Is it a _Go program_ competition? Or _Go-playing computer_ competition? I think in the former case it would make most sense to just run all the programs on the same hardware provided by the organizers. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
- Original Message From: David Doshay ddos...@mac.com Programmers work on all kinds of hardware. Making them port their code to some arbitrary standard platform is not a great idea. Just as one voice, I will not bother to port my code to a different box. So, if the competitions are all on the same hardware you are running a *Go -playing-programs-developed-on-that-platform* competition. And that sounds silly to me. I have been working on Go on moderate sized clusters (25 - 75 CPUs) for about 5 years. Most of my work has shown that it is not trivial to write a program that scales reasonably over those processors. It is far from trivial to develop and debug programs that run on clusters. More hardware is no guarantee of a stronger program and is not a simple way to get one. Amen! Furthermore, I'd strongly encourage developers using clusters, even supercompter-sized clusters -- these are the forerunners of computers which you and I will use next year, or ten years from now. The existence of strong competitors will spur candidates with more ordinary quad and octo-core computers to work harder to optimize their codes. As David Levy admitted, microprocessors have done quite creditably in competion with mainframes in the Chess world. I was delighted to see Mogo on a supercomputer pitted against a professional Go player. That experiment surely raised many interesting questions. How big would a supercomputer need to be, to beat a pro on an even game? What other refinements to the program would be needed? Could the program be optimized to use the hardware even more effectively? Would an FPGA be able to deliver the processing power needed in a smaller hardware package? I would not want the design space for computer Go to be restricted to currently available desktop computers, especially when that target advances so rapidly. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
The proposed performance-per-watt metric would probably give Sicortex a leg up. Imagine the headline: Ten MIT cyclists power supercomputer which defeats pro Go Player :D Subsequently, a fierce battle rages over whether to require cyclists to be selected randomly from the geek population, instead of letting those with superior quads take the crown. Terry McIntyre terrymcint...@yahoo.com ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
also, it's quite surprising how few watts the human brain uses. s. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
I think the whole concept of taking on performance per watt in the restricted domain of Go playing programs is silly. Are we to spend our time searching for the Transmeta cores and porting to those? Saving energy is a fine thing. Lets leave that to various hardware engineers in the semiconductor industry. Or, if you think this is such a grand idea then you should offer up the prize money and then we can all see who comes to compete for it. Lets stick to writing algorithms and use time restrictions in play against each other and against humans for our metrics. It is not the case that a successful algorithm on one CPU is a good one for putting on a cluster. Different people will try different things, some on one CPU, some to take advantage of the newer multi-core CPUs, and some will try things on multiple CPUs. It is all good, and these things are not directly interchangeable. I invite you to spend a few hours debugging an MPI application if you think there is a simple relationship between the power of the computer and the strength of the program. Putting a V12 engine in a soap-box derby car will not make it win. It is not that simple. Cheers, David On 14, Jan 2009, at 11:58 AM, Ryan Grant wrote: On Wed, Jan 14, 2009 at 9:22 AM, David Doshay ddos...@mac.com wrote: if the competitions are all on the same hardware you are running a *Go -playing-programs-developed-on-that-platform* competition. And that sounds silly to me. it would be worthwhile for this community to reward authors of efficient algorithms. authors of the best algorithms, once clearly identified, could line up for grants to port to larger architectures, to see whether their work scales. newcomers would have a strong incentive to try their ideas, even if they're not aligned with large computing resources. while we don't currently have such funding, clear community demand would be an important step in realizing it. for any architecture, we can measure which algorithms are getting the best results, per rough unit of computation resources. one very honest measurement is electricity. electricity can be estimated where it is not closely measured. here is a thread from July which tries to outline this in more detail: Elo-joules / honest clusters / Re: tournaments http://computer-go.org/pipermail/computer-go/2008-July/015357.html arrange a Green Computer Go tournament, in which the wattage used by each machine is well monitored. play a marathon to calculate rankings, then normalize based on each contestant's sum of game-relative joules. the best Elo / joules ratio determines the winner. -- - Ryan ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
On Jan 14, 2009, at 8:39 PM, David Doshay wrote: Saving energy is a fine thing. Lets leave that to various hardware engineers in the semiconductor industry. Or, if you think this is such a grand idea then you should offer up the prize money and then we can all see who comes to compete for it. Actually, this is one of IBM's sales-pitches. That their big computers use less energy per transaction than most other setups, whether they're super-computers or clusters. Maybe we can ask them to sponsor an event :-). I think we need to encourage a wide variety of approaches to make progress. Severe restrictions on the hardware to be used doesn't fit in that. But MCTS programs are known to scale well. So it's also not desirable to have a single-CPU computer compete with a 3,000 CPU cluster and call it even. So personally I'm still of the opinion that you can roughly divide competitions between 'all you can carry' and 'as large and powerful as you can arrange'. That will suffice for now. I'm sure computer-Go has quite a bit to go before it's in a similar situation as computer-chess. Mark ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
When I was big into Chess programming this was a sore topic for me as well. I felt it was unfair for people competing in the WCCC to win if they had a cluster of of 100 PCs, a Cray, etc, when another person was using a P200mhz. I believe it was Dr. Hyatt that said this and it made a lot of sense to me It's not just about creating the best chess program, it's creating the best playing machine So when you look at competitions that dont have hardware limits, you can't look at it like Engine X is the best in the world. You have to look at it like Engine X + this hardware setup is the best in the world; and take it with a grain of salt. Even if you did set hardware limits, it would be a hard task. Even if it's just a single PC do you use single core or multicore? Multicore would help those who have parallelized their code but hurt others. Is it fair? There's no real line that can be drawn, because on the flip side I wouldnt find it fair if I had written a parallel engine and spent all that time and effort to only be limited to 1 core. Just my $0.02. -Josh For now I tend to be of the opinion that in competitions, one should be able to bring your own hardware or run on standard hardware provided by organizers. The restriction that the hardware be physically present allows for enough flexibility that people or teams can try different set-ups (like a row of PS3s) while avoiding having people with access to a big cluster compete with people who only have access to a PC. But similarly to the competition of building the most powerful computer in the world, I can see room for a competition between big clusters that play Go as well. One doesn't have to be to the exclusion of the other. Think of car-racing. You have drag-racing where they use rockets to cross half a mile as fast as possible and you have F1-racing where the 'hardware' is constrained within certain limits. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
I must be out of touch, didnt know Rybka could run on a cluster :) last I checked he was about to release a smp version. Lots to catch up on. -Josh In chess, one team is firmly dominating (Rybka), and they have since last year also managed to acquire the best hardware (40 core cluster). This makes them essentially unbeatable. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
[computer-go] Re: Hardware limits
Lets look at it another way - no one would care what hardware you choose to use, unless you win. So at the very least, you ought to be able to use arbitrary hardware until it becomes established that only that class of hardware can win. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
On Sat, Jan 10, 2009 at 02:21:17PM +0100, Gian-Carlo Pascutto wrote: Mark Boon wrote: So it seems arbitrary to put limitations on the hardware. However, if two programs are essentially the same, but one side manages to bring a more powerful computer than the other, is it fair to award one program a prize and not the other? If the programmer has done the needed work to make use of that, obviously he deserves to be rewarded for that. However, how do you account for the other way, which seems far more likely to me, that a team has better program but not enough resources to buy high-end hardware to run it on, therefore losing anyway? The concern being, not only the quality of the program is rewarded, but also the amount of resources available to the team, which seems unfair to me if the goal of the competition would be to choose the best program. This scenario seems to me far more likely than the other way around, but I am not familiar with the actual practice and scenarios of Chess and Go tournaments. Maybe what should be qualified is really what kind of competitions are we talking about, and name them appropriately. Is it a _Go program_ competition? Or _Go-playing computer_ competition? I think in the former case it would make most sense to just run all the programs on the same hardware provided by the organizers. In the latter case, you do not have to worry about any restrictions on hardware at all. -- Petr Pasky Baudis The average, healthy, well-adjusted adult gets up at seven-thirty in the morning feeling just terrible. -- Jean Kerr ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
Dave Dyer wrote: I think general hardware limits are good, because they will permit more teams to be competitive without altering the nature of the competition. So in effect, it's an admission that the strength of some teams should be crippled in a completely arbitrary way, because they are to good for the others. It's nice that someone admits this in writing. -- GCP ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
On Jan 10, 2009, at 8:16 AM, Gian-Carlo Pascutto wrote: Dave Dyer wrote: I think general hardware limits are good, because they will permit more teams to be competitive without altering the nature of the competition. So in effect, it's an admission that the strength of some teams should be crippled in a completely arbitrary way, because they are to good for the others. It's nice that someone admits this in writing. Please, don't sneer. Different people have different ideas about things. If you don't agree with them, try to make them see your point of view by way of arguments. Don't get personal, it won't help you in any way. Rather the contrary, people will become less inclined to listen to you. We are trying to make computers play Go as well as possible. That inevitably has both a hardware and a software side to it. So it seems arbitrary to put limitations on the hardware. However, if two programs are essentially the same, but one side manages to bring a more powerful computer than the other, is it fair to award one program a prize and not the other? This is not an easy matter. Taking an extreme standpoint one way or the other is going to be difficult to maintain. For now I tend to be of the opinion that in competitions, one should be able to bring your own hardware or run on standard hardware provided by organizers. The restriction that the hardware be physically present allows for enough flexibility that people or teams can try different set-ups (like a row of PS3s) while avoiding having people with access to a big cluster compete with people who only have access to a PC. But similarly to the competition of building the most powerful computer in the world, I can see room for a competition between big clusters that play Go as well. One doesn't have to be to the exclusion of the other. Think of car-racing. You have drag-racing where they use rockets to cross half a mile as fast as possible and you have F1- racing where the 'hardware' is constrained within certain limits. Mark ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Re: Hardware limits
Mark Boon wrote: Please, don't sneer. ??? I have seen a lot of discussion, but no good reasons that make sense for the decision that was made. What Davy Dyer said IS a good reason, and most likely the real one. But the people in favor of the decision will not like to admit this. So it's good that at least one person realizes it. We are trying to make computers play Go as well as possible. The decision does not affect the Go tournament at all, where 3000 CPU clusters are still fully allowed to compete against single CPU programs. The thing is that in Go the small hardware is still beating the bigger one. In chess, one team is firmly dominating (Rybka), and they have since last year also managed to acquire the best hardware (40 core cluster). This makes them essentially unbeatable. And clearly, some of their commercial competitors have a problem with that. So it seems arbitrary to put limitations on the hardware. However, if two programs are essentially the same, but one side manages to bring a more powerful computer than the other, is it fair to award one program a prize and not the other? If the programmer has done the needed work to make use of that, obviously he deserves to be rewarded for that. -- GCP ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
[computer-go] Re: Hardware limits
I think general hardware limits are good, because they will permit more teams to be competitive without altering the nature of the competition. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
