Hi Ryan,
So. I discussed this a while ago, but here it comes again. Let me
first clear a few things from what you said.
> Only in the last month or so did I learn that -Dtests.iters doesn't really
> "work". What I mean is in regards to randomization.
This is not true. It works (as I will explain below). Try it, for
example (the annotation has the same effect as providing
-Dtests.iters=10):
@Repeat(iterations = 10)
@Seed("0")
public class Test001 extends RandomizedTest {
@Test public void test() {
System.out.println(randomAsciiOfLength(10));
}
}
I fixed the initial seed to make it reproducible. This will print:
nKtjLXhWQw
awHTHLIGAq
vEYgnxTkWv
mSAloRXtIV
iBhCJuZNzP
DHAIyqecSS
zaEoTAWAOa
CoraUrKuib
fKxUZnyQTx
beFtvsUTHc
> Each iteration currently is *exactly* the same as far as randomization
>> (each iteration uses the same master seed).
You can see above that it isn't true. Every iteration is different and
uses different randomness (and this randomness is "derived" from the
(master, iteration) pair so it is fully reproducible in each run).
>> Why not create a different seed for each iteration when -Dtests.iters is
>> used?
Let's talk about JUnit unit tests and how (any) runner should execute
them. I will demonstrate this on a simple class like this one (pseudo
code):
class Foo {
@BeforeClass beforeClassHook() {}
@Before beforeHook() {}
@Test test1() {}
@After afterHook() {}
@AfterClass afterClassHook() {}
}
There are a couple of "stages" to be executed. Simplifying a bit, it
looks like this.
0. Prerequsite
- class available, possible loaded and initialized
1. Setup:
- extract test methods
2. Execution.
- run class-before hooks (rules, @BeforeClass)
- for each test:
run before hooks (@Before, rules)
run the test itself
run after hooks (@After, rules)
- run class-after hooks (rules, @AfterClass)
For the class above, the sequence of method calls would be:
beforeClassHook()
new() // constructor
beforeHook()
test1()
afterHook()
afterClassHook()
If you were to multiply tests execution manually, you would copy-paste
the test method giving it a different name:
class Foo {
@BeforeClass beforeClassHook() {}
@Before beforeHook() {}
@Test test1() {}
@Test test2() {}
@After afterHook() {}
@AfterClass afterClassHook() {}
}
which would result in a sequence of calls like this one:
beforeClassHook()
new() // constructor
beforeHook()
test1()
afterHook()
new() // constructor (new instance)
beforeHook()
test2()
afterHook()
afterClassHook()
So, first of all, note that duplicating tests is *not* equivalent to
just looping around method body. Each execution should be run on a new
instance and wrapped with setup and teardown hooks, otherwise it's not
really an isolated JUnit test anymore (and it would be against JUnit
informal execution flow).
This is, in short, what -Dtests.iters does (and what @Repeat does) --
it replicates every test, making sure they have unique names (IDEs get
confused if they don't) and trying to work around other issues I won't
discuss here. It does work. The reason you believe it doesn't work is
because most of the stuff in LuceneTestCase is initialized at *static*
class level, which by definition is executed only once, regardless of
the number of tests in a class. Let's modify our initial example a
bit:
@Repeat(iterations = 10)
@Seed("0")
public class Test002 extends RandomizedTest {
static String s;
@BeforeClass
public static void beforeClass() {
s = randomAsciiOfLength(10);
}
@Test public void test() {
System.out.println(s);
}
}
If you run this, you'll see:
SXVNjhPdQD
SXVNjhPdQD
SXVNjhPdQD
SXVNjhPdQD
SXVNjhPdQD
SXVNjhPdQD
SXVNjhPdQD
SXVNjhPdQD
SXVNjhPdQD
SXVNjhPdQD
This works as expected because beforeClass() is invoked once (even if
every test has a different randomness available to it). LuceneTestCase
does it for performance reasons (that static initialization is fairly
costly). This ends the
JUnit part of the story.
But wait, there is more. If you take a look above at how JUnit runners
should work they load the class (or in fact are given an initialized
class) before they can do anything. So if there are static class
initializers (static { field = foo(); }) then these may get executed
well before the runner has any chance to initialize its own stuff --
that's why you *have* to use @BeforeClass methods if you want to use
RandomizedTest's randomness; doing
this:
@Repeat(iterations = 10)
@Seed("0")
public class Test003 extends RandomizedTest {
static final String s;
static {
s = randomAsciiOfLength(10);
}
@Test public void test() {
System.out.println(s);
}
}
will result in an initialization exception complaining about missing
random context:
java.lang.IllegalStateException: No context information for thread:
Thread[id=11, name=SUITE-Test003-seed#[0], state=RUNNABLE,
group=TGRP-Test003]. Is this thread running under a class
com.carrotsearch.randomizedtesting.RandomizedRunner runner context?
Add @RunWith(class
com.carrotsearch.randomizedtesting.RandomizedRunner.class) to your
test class. Make sure your code accesses random contexts within
@BeforeClass and @AfterClass boundary (for example, static test class
initializers are not permitted to access random contexts).
Finally, all the above probably begs the question why can't the runner
just wrap every iteration one level up the ladder -- re-execute static
hooks, etc.
The answer is that you are only considering the level of a single
class and the master seed is in fact used for other things, most
notably for ordering test classes for predictable execution within one
forked JVM (this is done in JUnit4-Ant task). If you re-executed one
class's static hooks with a different seed for different iterations
how would they report the "real master" seed back for re-running the
entire suite? Should it be a pair master-seed:iteration? Only the derived
class seed? Or maybe we should have two seeds -- one for ordering classes
and one for the actual class execution?
It gets really messy if you get down to details like this, not
to mention missing support from IDEs for running multiple tests of the
same class (it's already pretty complicated to make it work in
Eclipse, IntelliJ Idea, etc. when multiple identical methods are executed).
Finally:
>> different people have their own "beasting" scripts
>> that run the test essentially N times from a shell to force different
>> seeds in each iteration.
This is true. ButI don't think Mike, for example, will resign from
his scripts even if "real" class-level tests.iters would be
implemented -- Mike's
script runs tests across multiple machines via SSH; I won't have the
time for such distributed extensions in any near future.
If you want to try to implement reiteration at the 'static context'
feel free to give it a shot though; I would certainly be interested in
how you approach the problems I mentioned above. A good place to start
would be to modify JUnit4-ant (Ant task), around here:
https://github.com/carrotsearch/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L887
if you somehow associated a class with its master seed (and duplicated
classes as well), then you could fork off multiple class executions
with different seed (you'd still have to modify the forked subprocess
to accept class name and master seed; currently only one master seed
is passed at the start of each JVM.
Or you could do what I mentioned above -- separate the seed used for
class ordering from the one used for executing
every class (every iteration of a class).
Or maybe it'd be easier to modify the runner itself (then duplication
would work from IDE level)... but then you'll hit the IDE issues and
constraints... I really don't know which way is better (or worse).
That's part of the challenge I guess. :)
I'll try to look at the code again in the next few days and will give
you some feedback. I can't log to Jira for some reason, but I'll
lookup the issue number for this, it's been there for a good while.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
