Without claiming that this is the final set of categories or that they are used correctly right now, here is what I think they mean:
- ValidatesRunner tests should be tests of the runner itself, generally that it implements a primitive correctly - NeedsRunner tests should be tests of the PTransform/pipeline, assuming the runner is correct Notably, to the extent the assumption of runner correctness holds, this implies that it is OK to run NeedsRunner tests with just the direct runner. Pragmatically, in the Java SDK & IOs, this is not the current breakdown. - In the Java SDK the NeedsRunner category is probably used more to flag "run this with just the direct runner" than to express the semantic intent. That isn't so bad; it is very close to the right usage. - There are IOs that had RunnableOnService tests which are now ValidatesRunner tests. While the ability to run an IO does validate a runner, this is really an integration test of the IO. If they are to be run with just the direct runner they don't need any annotation, because the IO can take a test-scoped dependency on the direct runner. So it mostly makes sense to tag those tests for which it is profitable to run against all runners. I think the question of IO ITs with the intent to run across runners is currently under design discussion and I would defer to other people on the best way to do that. It could be a new category, or it could be a different design pattern entirely. Kenn On Mon, Mar 27, 2017 at 11:55 AM, Eugene Kirpichov < [email protected]> wrote: > Kenn - can you also remind for everybody, what is the difference between > @NeedsRunner and @ValidatesRunner, and when should one use one or the > other? I always find myself confused about this especially in code reviews. > > On Mon, Mar 27, 2017 at 11:32 AM Kenneth Knowles <[email protected]> > wrote: > > > Hi all, > > > > I just merged the rename from RunnableOnService to ValidatesRunner in the > > Java codebase (Python was already there) > > https://github.com/apache/beam/pull/2157. > > > > I'm sure there will be stragglers throughout our docs, etc, so please do > > help me catch them and fix them. And start learning to say > > "ValidatesRunner" in conversation :-) > > > > Kenn > > > > On Thu, Nov 10, 2016 at 1:01 PM, Lukasz Cwik <[email protected]> > > wrote: > > > > > The default is a crashing runner which throws an exception if its > > executed. > > > This makes SDK core/examples/... not depend on any implemented runners. > > > > > > On Thu, Nov 10, 2016 at 12:37 PM, Robert Bradshaw < > > > [email protected]> wrote: > > > > > > > +1 to ValidatesRunner. I'd be nice if it were (optionally?) > > > > parameterized by which feature it validates. > > > > > > > > @NeedsRunner is odd, as using a runner is the most natural way to > > > > write many (most) tests, but an annotation should be used to mark the > > > > exception, not the norm. (I'd just assume a runner is available for > > > > all tests, e.g. CoreTests depends on DirectRunner depends on Core). > > > > > > > > On Thu, Nov 10, 2016 at 10:14 AM, Mark Liu > <[email protected] > > > > > > > wrote: > > > > > +1 ValidatesRunner > > > > > > > > > > On Thu, Nov 10, 2016 at 8:40 AM, Kenneth Knowles > > > <[email protected] > > > > > > > > > > wrote: > > > > > > > > > >> Nice. I like ValidatesRunner. > > > > >> > > > > >> On Nov 10, 2016 03:39, "Amit Sela" <[email protected]> wrote: > > > > >> > > > > >> > How about @ValidatesRunner ? > > > > >> > Seems to complement @NeedsRunner as well. > > > > >> > > > > > >> > On Thu, Nov 10, 2016 at 9:47 AM Aljoscha Krettek < > > > [email protected] > > > > > > > > > >> > wrote: > > > > >> > > > > > >> > > +1 > > > > >> > > > > > > >> > > What I would really like to see is automatic derivation of the > > > > >> capability > > > > >> > > matrix from an extended Runner Test Suite. (As outlined in > > Thomas' > > > > >> doc). > > > > >> > > > > > > >> > > On Wed, 9 Nov 2016 at 21:42 Kenneth Knowles > > > <[email protected] > > > > > > > > > >> > > wrote: > > > > >> > > > > > > >> > > > Huge +1 to this. > > > > >> > > > > > > > >> > > > The two categories I care most about are: > > > > >> > > > > > > > >> > > > 1. Tests that need a runner, but are testing the other > "thing > > > > under > > > > >> > > test"; > > > > >> > > > today this is NeedsRunner. > > > > >> > > > 2. Tests that are intended to test a runner; today this is > > > > >> > > > RunnableOnService. > > > > >> > > > > > > > >> > > > Actually the lines are not necessary clear between them, > but I > > > > think > > > > >> we > > > > >> > > can > > > > >> > > > make good choices, like we already do. > > > > >> > > > > > > > >> > > > The idea of two categories with a common superclass actually > > > has a > > > > >> > > pitfall: > > > > >> > > > what if a test is put in the superclass category, when it > does > > > not > > > > >> > have a > > > > >> > > > clear meaning? And also, I don't have any good ideas for > > names. > > > > >> > > > > > > > >> > > > So I think just replacing RunnableOnService with RunnerTest > to > > > > make > > > > >> > clear > > > > >> > > > that it is there just to test the runner is good. We might > > also > > > > want > > > > >> > > > RunnerIntegrationTest extends NeedsRunner to use in the IO > > > > modules. > > > > >> > > > > > > > >> > > > See also Thomas's doc on capability matrix testing* which is > > > > aimed at > > > > >> > > case > > > > >> > > > 2. Those tests should all have a category from the doc, or a > > new > > > > one > > > > >> > > added. > > > > >> > > > > > > > >> > > > * > > > > >> > > > > > > > >> > > > > > > > >> > > > > https://docs.google.com/document/d/1fICxq32t9yWn9qXhmT07xpclHeHX2 > > > > >> > VlUyVtpi2WzzGM/edit > > > > >> > > > > > > > >> > > > Kenn > > > > >> > > > > > > > >> > > > On Wed, Nov 9, 2016 at 12:20 PM, Jean-Baptiste Onofré < > > > > >> [email protected] > > > > >> > > > > > > >> > > > wrote: > > > > >> > > > > > > > >> > > > > Hi Mark, > > > > >> > > > > > > > > >> > > > > Generally speaking, I agree. > > > > >> > > > > > > > > >> > > > > As RunnableOnService extends NeedsRunner, @TestsWithRunner > > or > > > > >> > > > @RunOnRunner > > > > >> > > > > sound clearer. > > > > >> > > > > > > > > >> > > > > Regards > > > > >> > > > > JB > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > On 11/09/2016 09:00 PM, Mark Liu wrote: > > > > >> > > > > > > > > >> > > > >> Hi all, > > > > >> > > > >> > > > > >> > > > >> I'm working on building RunnableOnService in Python SDK. > > > After > > > > >> > having > > > > >> > > > >> discussions with folks, "RunnableOnService" looks like > not > > a > > > > very > > > > >> > > > >> intuitive > > > > >> > > > >> name for those unit tests that require runners and build > > > > >> lightweight > > > > >> > > > >> pipelines to test specific components. Especially, they > > don't > > > > have > > > > >> > to > > > > >> > > > run > > > > >> > > > >> on a service. > > > > >> > > > >> > > > > >> > > > >> So I want to raise this idea to the community and see if > > > anyone > > > > >> have > > > > >> > > > >> similar thoughts. Maybe we can come up with a name this > is > > > > tight > > > > >> to > > > > >> > > > >> runner. > > > > >> > > > >> Currently, I have two names in my head: > > > > >> > > > >> > > > > >> > > > >> - TestsWithRunners > > > > >> > > > >> - RunnerExecutable > > > > >> > > > >> > > > > >> > > > >> Any thoughts? > > > > >> > > > >> > > > > >> > > > >> Thanks, > > > > >> > > > >> Mark > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > > -- > > > > >> > > > > Jean-Baptiste Onofré > > > > >> > > > > [email protected] > > > > >> > > > > http://blog.nanthrax.net > > > > >> > > > > Talend - http://www.talend.com > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > >
