> cxtest has already impressive infratructure (installation, creation of > clean environment) and nice scripting. However, it requires Metacity WM > and VNC. The Metacity WM doesn't really concern me, the VNC thing however does. This makes game testing harder since VNC isn't a part of the average game setup. We don't have to transfer all the rendering screenshots over VNC though. I am CCing Martin Pilka on this, maybe he can give us some more insight.
> I'm personally using AutoHotkey(http://www.autohotkey.com) as an > automatic testing framework backend. It's a Windows application, so it > can be run through Wine itself and it's sandboxed from X11. It looks interesting, this would also allow us to run automated tests on Windows, perhaps to track the effect of driver updates, etc. > I can imagine Winebot packages to be easily converted to an automatic > testing engine. Winebot actually covers automated installation and > creation of clean independent environment. I don't know if we have to automate the installations as well. Reinstalling the apps excludes a few factors and makes the tests more repeatable, on the other hand it is not something that occurs in the average gaming environment. Not reinstalling can bring up problems related to the game's state on the disk, like fragmenting game caches. When we see a problem we can reset the environment manually to exclude the disk content as the cause. Automated game installation is a nice to have, but not a strict requirement IMHO. Not reinstalling would ease the use of some controller application that runs inside wine, since constant installing and removing of the app can leave garbage in the C drive and the registry over time. On the other hand, an external application could remove the wineprefix and start with a completely clean Windows environment. > Running applications with different log levels, tracing output, > comparing screenshots with different levels of fuzzy comparaion > - it's not faraway. How would fuzzy comparison work?
