This is a great idea! Your suggestion to do it via a JUnit test runner makes it very concrete.
Kenn On Thu, Jun 22, 2017 at 3:27 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Hi folks and especially runner developers, > > https://issues.apache.org/jira/browse/BEAM-2506 - quoting from there: > > Currently ValidatesRunner test suites run 1 pipeline per unit test. That's > a lot of small pipelines, and consumes a lot of resources especially in > case of a pretty heavyweight runner like Dataflow, so tests take a long > time and can't be run in parallel due to quota issues, etc. > > Jason Kuster says he and Davor Bonaci discussed that we could execute > multiple unit tests in a single TestPipeline. > > To further develop it: in case of Java, we could create a custom JUnit > Runner http://junit.org/junit4/javadoc/4.12/org/junit/runner/Runner.html > that would apply all the transforms and PAsserts in unit tests to a single > instance of TestPipeline (per class, rather than per method), and run the > whole thing at the end. PAssert captures the source location of its > application, so we could still report which particular test failed. > > This obviously has fewer isolation between unit test methods, cause they > effectively run in parallel instead of in sequence, so things like > per-method setup and teardown will no longer be applicable. There'll > probably be other issues. > > Anyway, this seems doable and high-impact. > > Just bringing this to the attention of the community - it seems worth > discussing and perhaps someone will be interested in developing this idea > further or implementing it. >
