Can we split this into 1) test code boilerplate - I have big issues with removing this
2) capturing tests that are expected to fail due to issues in TDML vs. in the scala test driver code. - I'm on board with this. With respect to (1) the test code boilerplate.... The ability to start from a unit test one-liner in the IDE and debug a test is not a nice-to-have. It is 100% the way I do all my work on Daffodil. Furthermore, it is the way I *want* to work on Daffodil. It keeps the emphasis on test-driven development, etc. Furthermore, I depend on running large bulk test runs, within the IDE, and seeing the one-by-one little check boxes (or capture of the individual test error and console output) for each individual test. So it's not just can I run them easily. It's can I run 1000+ of them, and then use the IDE feature to click through the 15 that failed one by one. It is completely worth keeping all the test boilerplate in order to have this. This is a big part of where I get personal "joy" from working on Daffodil. You take this away, and it's no longer anywhere near as much fun to work on. Mess with peoples' fun at your peril 🙂 If you can come up with a way for me to get exactly that behavior - where a TDML file of tests behaves exactly like a file of JUnit tests in the IDE, then I could deal with that. E.g., a pre-processor step like 'sbt setupTests' that scrapes all the TDML files and outputs all the scala test JUnit files automatically would be one way to maybe do that. The test driver files could then not be checked-in I think. But the goal here should not be to "accommodate IDE users" somehow. The goal should be perfect seamless execution that follows ordinary IDE idioms in the ordinary way, like developing any other library software with unit tests in it. A few one-time setup hiccups are tolerable. Once things are setup, then it has to be "IDE-ordinary/JUnit-ordinary" for the test-edit-debug-compile-repeat cycle. Without that I believe we exclude developers who want powerful IDE support. Like me. Both Eclipse and IntelliJ currently achieve this in my view. IntelliJ simply has better Scala-language support, so we seem to prefer it. But Eclipse works fine, for the degree of scala support it offers. For issue (2): capturing tests that are supposed to fail in the TDML not in the scala test driver code. This sounds really really good. I think moving all parameters currently expressed in scala test driver files onto the TDML as additional attributes on the testSuite element or on the parserTestCase and unparserTestCase elements is great. Some scala-test-suite parameters specify how the TDML file XML is to be read, such as with or without validation on. Capturing that flag on the TDML testSuite element requires reading every TDML file twice - once without validation, then once again to validate it once you rule out the use of the no-validate flag. This latter would be the normal case, not the exceptional case. But the overhead here might be trivial in the grand scheme of things. I think tests that are marked as expected to fail (hopefully with a JIRA ticket marking that), I think they should be run anyway for that implementation, and the fact that they fail verified. This insures they are run periodically and so will capture and characterize failing behavior. Some bugs fix themselves (as part of library upgrades, or as part of fixing other bugs), and this would identify those for us. When I comb through the bugs periodically there's often the notion "I think that may already be fixed." It would be great for the test suite to tell us. They do need to be somehow identified so when they fail as expected, that's not counted as a passing test, but as a known failing test that is still expected to fail. JUnit can't distinguish this I think, but a test script could, or a message. Though we have thousands of tests, dozens of which might be these "expected to fail" variety, and we don't want too much output noise. One line per saying "Expected Failure DAFFODIL-9876" would perhaps be ok. I would suggest the properties not combine implementation and failure documentation into one attribute, but rather since such a test can be a negative or positive test case, we need a place to put strings that are expected to find in the failure. We'd have to invent a new child element to express those like: <tdml:bug implementation="daffodil" info="DAFFODIL-9876"> <tdml:expect>some string here</tdml:expect> </tdml:bug> Shorter without the expect info: <tdml:bug info="DAFFODIL-9876"/> Notice here I inferred the implementation attribute from the info attribute (first token). We should allow multiple such bug child elements to report about different implementations, or if there are multiple tickets to highlight. ________________________________ From: Steve Lawrence <[email protected]> Sent: Monday, January 11, 2021 8:58 AM To: [email protected] <[email protected]> Subject: Re: rfc: sbt-tdml This is great! Additional comments inline: On 1/9/21 12:20 PM, Interrante, John A (GE Research, US) wrote: > I like the idea of removing the Scala TDML test boilerplate as much as > possible. Does your sbt plugin handle the use case where some Scala test > files define customized or multiple TDML runners in the same file? For > example, see > daffodil-test/src/test/scala/org/apache/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala > where two runners are defined for the same TDML file (expressions.tdml, one > standard runner and one customized runner with validateDFDLSchemas = false) > and several other customized runners are defined (expressions2.tdml with > compileAllTopLevel = true). Even if your sbt plugin can define only a > standard runner for each TDML file, it would be nice to remove most of the > Scala TDML test code except for the code using customized runners. I'm sure > there will be some edge cases where the sbt plugin must not run a TDML file > with a standard runner because it works only with a customized runner, but > the ability to specify configuration files in TDML or tell the sbt plugin to > skip some TDML files may help somewhat. I suspect if we ever need the same TDML file to be run with different TDML Runner configurations, then we would just need two different TDML files. It's maybe a bit of a headache, but certainly much cleaner, and is a pretty uncommon occurrence. Having this plugin read the TDML file for attributes defined on the testSuite would be a good way to configure the runner. > We need to make sure we preserve the ability to skip some TDML test cases > with a string explaining why they are failing (usually a JIRA bug number). > If we comment out failing test cases directly in the TDML file then we take > the risk that these tests will bitrot faster once the TDML processor no > longer parses them. I looked at > daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd and I think > we could modify the TDML schema to support skipping failing test cases. Both > parserTestCase and unparserTestCase use the same attributeGroup > (testCaseAttribs). We could add a new optional string-type attribute > "reasonForSkipping" to testCaseAttribs and modify the sbt plugin or the TDML > processor to automatically skip such test cases (perhaps printing their names > and/or the reasons for skipping them, but we don't want to make the test > output too big). Agreed, parser/unparserTestCases definitely need a way to specify that this test is expected to fail and should be skipped. We sortof already have this capability with the "implementations" attribute. If that attribute doesn't contain "daffodil" that it is skipped when testing Daffodil. So you could do something like implementations="" or implementations="ibm" (if a test only works in IBM). Unfortunately, this doesn't required that bug information be included, which is pretty important. It would maybe be nice if a test would only be skipped if bug information were included. I wonder if maybe we need to rethink the "implementations" attribute? We might consider instead something like <implementation>-failure="details". So for example, a test that fails on Daffodil might include daffodil-failure="DAFFODIL-1234" This way, in order to have the TDML runner skip a test, you MUST include this attribute which can provide details. This detail can then be included in the output when a test is skipped. Additionally it makes it easy to grep for failed tests and related bug numbers. > Nice idea, it will be good to see how far it can reduce the size of the Scala > test boilerplate code! Oh, it occurs to me that sometimes it's useful to > debug a TDML test case directly from the IDE by running a Scala test method > under the debugger. How often do developers need to debug existing or new > TDML test cases, which would require them to write throw-away Scala test > methods by hand whenever necessary? Maybe we can code up some sort of > universal Scala entry point we can run from the IDE which can run any desired > TDML test case by editing (at most) one or two lines of code. Yeah, I think this is probably the the most important functionality to have, and probably isn't viable if IDE support isn't there. How does this integrate with IDEs? Maybe it "just works" for IDEs that support sbt? What about if an IDE supports BSP since our current version of SBT supports BSP now? But what about IDE's like eclipse. I'm not sure if any devs use Eclipse anymore since IntelliJ seems much better, but maybe something to consider? Can we claim we only support IDE's that support SBT/BSP? Also, SBT has the testOnly command to run a single testsuite/test. Is that supported with this? For example, currently we can run sbt testOnly org.apache.daffodil....TestSuite -- --tests=test_regex So it takes a test suite and an optional regex of tests in that suite to run. Having this similar functionality would be pretty critical. Presumably, it would be very similar to the above, but maybe just a path to the TDML file instead, e.g. sbt testOnly org/apache/daffodil/.../TestSuite.tdml -- --tests=test_regex Using the path to the tdml file as the test suite name avoids issues with duplicate test suite names in TDML files. Also makes for easily copy/pasting a path when you want to test a tdml file. Some other thoughts: Does anything output when a test fails? Or is that still to be implemented? I'm also curious how this works regarding Daffodil versions. Presumably this depends on a certain Daffodil version? What happens if a repo has a different Daffodil dependency? Does this override that? Or does this just expect that Daffodil is already a dependency and uses that version? And hopefully our TDMLRUnner API is stable enough that things just work? We definitely have cases where we need to run tests with different Daffodil versions, it would be nice if we didn't have to also change the plugin version to match. We also need to consider how this gets published. Do you plan to maintain and publish this separately as an outside plugin that Daffodil just depends on an uses? Or do you plant to have it merged into a Daffodil repo? If so, I wonder if it lives in the Daffodil repo so Daffodil always has the latest, and then this is published when there are Daffodil releases? Or maybe it's a separate Apache repo that follows the same Apache release process/requirements, but can be on a different schedule than Daffodil? If all the above issue can potentially be resolved, I think there's a good chance this could replace all of our boiler plate, which I'm in big favor of. > -----Original Message----- > From: John Wass <[email protected]> > Sent: Friday, January 8, 2021 4:27 PM > To: [email protected] > Subject: EXT: rfc: sbt-tdml > > https://github.com/jw3/sbt-tdml > I needed to get smarter on SBT plugins for a task and tinkered with something > in Daffodil to do it. > > The goal here was to remove the TDML test boilerplate. I threw this at the > daffodil-test project with surpsisingly sane results on the first try. See > the bottom of the readme. > > The learning process is pretty much complete, so not sure where this goes, > but wanted see if there are any comments. >
