On Wed, 16 Jan 2013 07:28:13 PM Pedro Rodrigues wrote: > What is the current best way to evaluate changes in the AI? > > For whoever is working with AI-related code, how do you evaluate your > changes?
I have a hairy collection of scripts that automatically runs games with 7 AIs + 1 observer from a known starting configuration, mines the log files, and produces output like this: 20130117-small-7-VH x 10 Builds 45.600 ~ 8.393 Cashins 2.000 ~ 0.894 Colony# 41.200 ~ 8.352 ColonySize 3.273 ~ 0.499 Defences 261.900 ~ 19.310 Missions 7.900 ~ 2.914 Native Demands 44.400 ~ 18.189 Native Gifts 37.100 ~ 14.202 Pioneerings 61.200 ~ 10.429 Piracies 71.200 ~ 22.004 Scoutings 150.000 ~ 26.442 Seek+Destroys 249.500 ~ 30.884 Transports 303.200 ~ 83.260 Turn Speed 3.953 ~ 0.239 Wishes 148.800 ~ 17.429 ...which means that for last night's run using the game on a small map and Very Hard rules in 10 independent runs, the number of successfully completed TransportMissions is down to 300 or so on average. This is a bit low, although not scarily so given the standard deviation near 80, but for 0.10.7 it was nearer 350, so I should probably look at that. OTOH, the other measures are pretty stable, especially the critical Colony# + ColonySize, so I can be reasonably confident that the new patch to fix the isPerson bug is not causing massive breakage. > On a related subject (as proper AI-testing many times rely on removing the > randomness of the system) , what was the reasoning behind the removal of > the PseudoRandom interface in rev.6988, and make all implementations > inherit from Java.util.random? [Terminological aside: I tend to use the term inherit / inheritance if there is a class hierarchy relationship, so as none of our classes actually extend Random I do not wish to use that term here. I would say we "use" Random. Or if this was C I would say we "include" java.util.Random.] IIRC PseudoRandom did nothing but wrap nextInt. There were a few places where other functionality of java.util.Random would have been helpful (e.g. nextDouble). At the time two other processes were either underway or planned--- giving each of the AIs its own RNG (so as to improve determinism) and routing all game-changing random generation through a logging function of some sort (like Utils.randomInt) so as to be able to track problems with random numbers (remember all the complaints we have about the combat system being rigged?). Its a couple of years ago now, so memory is fading, but in the end I expect I decided that PseudoRandom was not doing anything useful. > The use of the interface is much preferable > as it makes it clear that several implementations can be used in a given > method call (thus making the code more readable), and avoids unforeseen > consequences when using those other implementations, as they do not have > inherited methods that may give different results from those intended (or > even desirable). This would be a good argument if we had any intention of using anything other than java.util.Random to generate random numbers. I think that would be a really bad idea. I have put a lot of work into making sure that every[1] random number generated by FreeCol derives ultimately from the initial seeding of a single java.util.Random at game initialization time. Indeed, AFAICT, if the thread scheduling could be made deterministic so that the AIs always interacted with the server in the same order, an all-AI FreeCol game would be exactly repeatable. That aside, when a user says `The RNG is broken', we now can reply `Then so are all users of java.util.Random, everywhere'. [1] OK, there might still be some client-side-only random calls that do not use the main RNG, but they only alter graphical effects, not game play. Cheers, Mike Pope
signature.asc
Description: This is a digitally signed message part.
------------------------------------------------------------------------------ Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612
_______________________________________________ Freecol-developers mailing list Freecol-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freecol-developers
