[
https://issues.apache.org/jira/browse/LUCENE-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss updated LUCENE-3492:
--------------------------------
Description:
{color:red}The work on this issue is temporarily at github{color} (lots of
experiments and tweaking):
https://github.com/carrotsearch/randomizedtesting
Or directly: git clone git://github.com/carrotsearch/randomizedtesting.git
{color}
----
RandomizedRunner is a JUnit runner, so it is capable of running @Test-annotated
test cases. It
respects regular lifecycle hooks such as @Before, @After, @BeforeClass or
@AfterClass, but it
also adds the following:
Randomized, but repeatable execution and infrastructure for dealing with
randomness:
- uses pseudo-randomness (so that a given run can be repeated if given the same
starting seed)
for many things called "random" below,
- randomly shuffles test methods to ensure they don't depend on each other,
- randomly shuffles hooks (within a given class) to ensure they don't depend on
each other,
- base class RandomizedTest provides a number of methods for generating random
numbers, strings
and picking random objects from collections (again, this is fully repeatable
given the
initial seed if there are no race conditions),
- the runner provides infrastructure to augment stack traces with information
about the initial
seeds used for running the test, so that it can be repeated (or it can be
determined that
the test is not repeatable -- this indicates a problem with the test case
itself).
Thread control:
- any threads created as part of a test case are assigned the same initial
random seed
(repeatability),
- tracks and attempts to terminate any Threads that are created and not
terminated inside
a test case (not cleaning up causes a test failure),
- tracks and attempts to terminate test cases that run for too long (default
timeout: 60 seconds,
adjustable using global property or annotations),
Improved validation and lifecycle support:
- RandomizedRunner uses relaxed contracts of hook methods' accessibility (hook
methods _can_ be
private). This helps in avoiding problems with method shadowing (static
hooks) or overrides
that require tedious super.() chaining). Private hooks are always executed
and don't affect
subclasses in any way, period.
- @Listeners annotation on a test class allows it to hook into the execution
progress and listen
to events.
- @Validators annotation allows a test class to provide custom validation
strategies
(project-specific). For example a base class can request specific test case
naming strategy
(or reject JUnit3-like methods, for instance).
- RandomizedRunner does not "chain" or "suppress" exceptions happening during
execution of
a test case (including hooks). All exceptions are reported as soon as they
happened and multiple
failure reports can occur. Most environments we know of then display these
failures sequentially
allowing a clearer understanding of what actually happened first.
was:
I love the idea of randomized testing. Everyone (we at CarrotSearch, Lucene and
Solr folks) have their glue to make it possible. The question is if there's
something to pull out that others could share without having the need to import
Lucene-specific classes.
The work on this issue is on my github account (lots of experiments):
https://github.com/dweiss/randomizedtesting
Or directly: git clone git://github.com/dweiss/randomizedtesting.git
> Extract a generic framework for running randomized tests.
> ---------------------------------------------------------
>
> Key: LUCENE-3492
> URL: https://issues.apache.org/jira/browse/LUCENE-3492
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/test
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Minor
> Fix For: 4.0
>
> Attachments: Screen Shot 2011-10-06 at 12.58.02 PM.png
>
>
> {color:red}The work on this issue is temporarily at github{color} (lots of
> experiments and tweaking):
> https://github.com/carrotsearch/randomizedtesting
> Or directly: git clone git://github.com/carrotsearch/randomizedtesting.git
> {color}
> ----
> RandomizedRunner is a JUnit runner, so it is capable of running
> @Test-annotated test cases. It
> respects regular lifecycle hooks such as @Before, @After, @BeforeClass or
> @AfterClass, but it
> also adds the following:
> Randomized, but repeatable execution and infrastructure for dealing with
> randomness:
> - uses pseudo-randomness (so that a given run can be repeated if given the
> same starting seed)
> for many things called "random" below,
> - randomly shuffles test methods to ensure they don't depend on each other,
> - randomly shuffles hooks (within a given class) to ensure they don't depend
> on each other,
> - base class RandomizedTest provides a number of methods for generating
> random numbers, strings
> and picking random objects from collections (again, this is fully
> repeatable given the
> initial seed if there are no race conditions),
> - the runner provides infrastructure to augment stack traces with information
> about the initial
> seeds used for running the test, so that it can be repeated (or it can be
> determined that
> the test is not repeatable -- this indicates a problem with the test case
> itself).
> Thread control:
> - any threads created as part of a test case are assigned the same initial
> random seed
> (repeatability),
> - tracks and attempts to terminate any Threads that are created and not
> terminated inside
> a test case (not cleaning up causes a test failure),
> - tracks and attempts to terminate test cases that run for too long (default
> timeout: 60 seconds,
> adjustable using global property or annotations),
> Improved validation and lifecycle support:
> - RandomizedRunner uses relaxed contracts of hook methods' accessibility
> (hook methods _can_ be
> private). This helps in avoiding problems with method shadowing (static
> hooks) or overrides
> that require tedious super.() chaining). Private hooks are always executed
> and don't affect
> subclasses in any way, period.
> - @Listeners annotation on a test class allows it to hook into the execution
> progress and listen
> to events.
> - @Validators annotation allows a test class to provide custom validation
> strategies
> (project-specific). For example a base class can request specific test case
> naming strategy
> (or reject JUnit3-like methods, for instance).
> - RandomizedRunner does not "chain" or "suppress" exceptions happening during
> execution of
> a test case (including hooks). All exceptions are reported as soon as they
> happened and multiple
> failure reports can occur. Most environments we know of then display these
> failures sequentially
> allowing a clearer understanding of what actually happened first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
