2009/4/3 Adam Murdoch <[email protected]> > > > Tom Eyckmans wrote: > > > > 2009/4/1 Adam Murdoch <[email protected]> > >> >> >> Tom Eyckmans wrote: >> >>> Hi guys, >>> >>> I'm working on the native test framework execution stuff, >>> >> >> Excellent >> >> I just want to let you guys know how I'm trying to implement this so you >>> can contribute ideas/ better ways of doing things. >>> >>> Global overview: >>> - recurse through compiled test classes, filter out class files that >>> contain a $ sign => testClassFilesQueue ( BlockingQueue ) >>> - testClassFilesQueue => scan for test classes (first found test class >>> determines test framework) => testInfosQueue ( BlockingQueue ) >>> - testServer / testClient >>> - server controls the forked test process: >>> - test process requests work (which test to execute next >>> (dequeued from testInfosQueue -> work/memory throthle), no more work -> >>> terminate) >>> - client sends progress events to the server => progressEventQueue => >>> these events get duplicated to BuildListener notifications and Test Report >>> output, which both need to get executed serially. >>> >>> Smarter test detection. This is done on the compiled test classes. Only >>> test classes that don't contain a $ sign are queued on a blocking queue for >>> processing. >>> >> >> I'm pretty sure it's quite possible for a test class to have a $ in it's >> name - it's just a bit unusual. We should probably support this. What was >> the thinking behind excluding classes with a $ sign? > > I had a problem with javassist, it threw an exception and couldn't read the > files with a $ sign in them, I didn't pay much attention to it at the time, > but they can indeed contain tests in them, as suggested by Peter I'll try > ASM for the detection perhaps ASM doesn't have this limitation. > >> >> >> > Interesting. I've used javassist to load up files with $ in their name > before. > It could be that I did something wrong.
> But, if ASM works and is fast enough, then that is probably a better > option, given that we drag it in to the classpath with groovy anyway. > Yes this is also something I like. > > > >> >>> When a class is identified as an actual test class, an implementation of >>> TestInfo is queued testInfosQueue, this object is a reference to the test >>> class and the type of implementation determines the way the test class needs >>> to be processed. This is needed to support executing JUnit TestCases and >>> TestSuites or TestNG Test classes and xml suite files. >>> >>> When the first test class is identified the TestFramework that identified >>> the test class is used. This causes only that framework to be used for test >>> detection from that point on. >>> >>> >> This is too dependent on the environment, when the test source contains >> both types of tests. For example, on some file systems we may find the >> test-ng tests first, and on some file systems we may find the junit tests >> first. We should do something which behaves the same way everywhere, >> something like: >> - have the build file specify which framework to use for a given test >> suite, with a default (ie what we do now) >> - or, scan for all types of tests and assert that there is exactly 1 test >> framework detected. > > Because of Peters feedback I don't think we should provide auto framework > detection because than we have to run detection during build file > evaluation, which seems extremely tricky to me (sourceDirs set on end of > build file??), then we don't know the testframework and don't know which > option object is going to/should be used. > >> >> > It would be nice if when I don't want to configure the test execution in > any way, that my tests just get executed without me doing anything in the > build file, and regardless of whether I'm using junit or testng or both. > In case of both I think we have several options that might be the case: - there is some migration happening => the user wants us to use testng to run both the testng and junit tests - the project is using both testng / junit for specific cases where one of the test frameworks is the only option => the user wants us to execute both testng and junit separately. - in case of a different test framework mix (spock / junit / testng/ ...) there might not be a possibility to have something joint executed and there is no other way but to execute every testframework separately. Because of the last possbility I'd have each test framework executed separately always, and just have multiple reports for each executed test framework. > > > >> >> To control the ports used by the server processes (over multiple gradle >>> builds running at the same time) I'm currently using a single file in >>> ~/.gradle/internal/testing/ports.used that is exclusively locked while a new >>> port is being determined or when a port is no longer used. Currently I'm >>> starting to use ports from 2000 on, I've just picked this port so this may >>> not be the best starting point. Ideally I want to detect free ports so there >>> is really no need for configuration for ports. >>> >> >> A simpler option would be to let the OS select the port when creating the >> listen socket in the build process, > > How does this work? > > > Usually you can bind to port 0, and the OS will assign a port. For example, > you can create a ServerSocket with port 0, and then query which port is > actually used by calling getLocalPort(). > That would remove the need for some detection logic in Gradle! If this works that would be extremely nice. I actually favour the client-server approach over the stdin/stdout option because with the client-server way we have the most normal forked vm running regarding stdin/stdout and we don't need to fiddle with it. > > > > I'm currently undecided where to do the output processing. >>> >> >> What do you mean by 'output processing'? > > Generating the test result XML files. > > > I think the build process is the best place to do this, as you suggest. > Ok > > > >> >> I'm currently in favour of doing this in the gradle build process as to >>> limit the classpath of the forked vm. I think we can re-use some of the Ant >>> JUnit output code but not sure if this is something that we want. >>> >>> I'd love to receive feedback on this. >>> >>> Thx, >>> >>> Tom >>> >>> >>> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> >> >
