On Tue, Sep 25, 2012 at 4:25 PM, Rob Godfrey <rob.j.godf...@gmail.com>wrote:
> > The reason the python tests *won't* work as a spec are precisely > > because they cannot "correspond to a piece of functionality". > > > > > I'm not sure that you have put forward a case that they "can not" only that > they "do not". I agree that the current tests are a fair way from meeting > the requirements of a functional spec, but fundamentally there should be a > 1:1 correspondence between a functional spec and a test suite. Whether one > defines the spec explicitly (and then writes an implicit test suite) or > implicitly (via the mechanism of a well defined set of tests) is really a > style issue I think. > > > > The tests are really good at what they do: verifying that functions > > work as expected. Keep them coming. > > > > > Actually I disagree. The tests currently aren't all that great. When trying > to get the tests to pass against the Java codebase there's really no help > in trying to work out what the tests are supposed to do (though some - such > as against the message API are much better by having a much clearer > definition of what it is that they are testing). > > > > But the tests are lousy at specifying what can be done. I pulled up > > messenger.py as a random example. > > 1. Not a single comment. > > > > +1 > > > > 2. Magic numbers. > > > > +1 > > > > 3. Magic process flow (setup, teardown part of some framework > > unrelated to messenger. What's with assert? What's the thread model?). > > > > +1 > > > > 4. No enumeration of functions to call, function arg lists, arg > > defaults, arg units, and so on. > > > > Not sure exactly what you mean here, though if you are saying that the > tests should in some way be categorized by which API function they are > testing, then I completely agree. > > > > 5. What are prerequisites? > > > > As a counterexample to using tests as specs, take a look at > > cpp/src/tests/acl.py and tell me how C++ broker ACLs work. > > No amount of ACL python test code will be a decent spec from > > which users can understand and write ACL rules. > > > > > I'm not sure that one can prove a negative by counter-example. My personal > view is that we *can* define a spec implicitly through tests, but that we > haven't yet proved that conjecture. Ultimately a (functional) test suite > and a functional spec should have some sort of one-to-one correspondence > and if they are defined in separate forms then there is a risk of > divergence (and code is generally more precise than text). However for many > people (probably including myself) it is easier to first write the positive > statement (the spec) and then define the negative (via the test). > > Ultimately the means (test vs. doc) is less important than just ensuring > that there is an adequate definition somewhere, which - to Rajith's > original point - I'm not sure there is yet (at least not written down). > I think the term 'functional spec' is possibly a bit distracting here as different people have different ideas of what that means, so let's forget about that for a minute and look at what we actually have and how it is working/not working. 1. We have an interface definition (in python), and a test suite/TCK (in python). 2. We have two implementations of this interface, the C implementation, and the Java implementation. On the one hand this is a very typical arrangement: An API, a test suite, and a set of implementations. On the other hand this is somewhat unusual in that the interface definition and test suite are in a different language than either implementation. What's incredibly powerful about this is that we have the ability to enforce with code consistent behaviour between the two implementations. Please don't underestimate how useful this is. We have many years of experience transcoding protocol implementations from one language to another without this capability, and I would say we have proven conclusively at this point that to attempt this kind of thing without a common test suite is guaranteed disaster. That said, there are some less than ideal things about the current setup: 1. The interface definition is scattered and difficult to see. It's pretty much a function of the C header files translated to python via swig, and this is not exactly idiomatic for python or Java. 2. Understanding the interface definition and test suite requires too much familiarity with the C implementation. I've already taken some steps to address both (1) and (2). Namely defining the idiomatic python API, thereby bringing together these scattered definitions into one place, and started writing new tests in terms of it rather than directly against the ad-hoc interface as defined by swig. Of course at this point much of the test suite is still written against the ad-hoc swig bindings, and the idiomatic python API itself has no API-doc as much of that documentation is still living inside the C header files via doxygen, and I really don't want to have the same documentation in two places. I believe if we move that documentation into the python interface definition, however, it would be possibly to extract via pydoc and related tools to serve as a source for API-doc for the C and Java impls, and any other bindings as well, and right now I think we would be better served by having that documentation live in python rather than C (modulo of course the C related portions). There is of course a back log of work here that can't be done instantaneously, however I suggest we create a tests component in JIRA, and if people run into issues understanding a specific test or a given aspect of the interface definition we can file JIRAs there and give priority to rewriting those tests against the idiomatic interface as well as bringing together all available documentation into the idiomatic API definition. --Rafael
