This is great! Additional comments inline: On 1/9/21 12:20 PM, Interrante, John A (GE Research, US) wrote: > I like the idea of removing the Scala TDML test boilerplate as much as > possible. Does your sbt plugin handle the use case where some Scala test > files define customized or multiple TDML runners in the same file? For > example, see > daffodil-test/src/test/scala/org/apache/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala > where two runners are defined for the same TDML file (expressions.tdml, one > standard runner and one customized runner with validateDFDLSchemas = false) > and several other customized runners are defined (expressions2.tdml with > compileAllTopLevel = true). Even if your sbt plugin can define only a > standard runner for each TDML file, it would be nice to remove most of the > Scala TDML test code except for the code using customized runners. I'm sure > there will be some edge cases where the sbt plugin must not run a TDML file > with a standard runner because it works only with a customized runner, but > the ability to specify configuration files in TDML or tell the sbt plugin to > skip some TDML files may help somewhat.
I suspect if we ever need the same TDML file to be run with different TDML Runner configurations, then we would just need two different TDML files. It's maybe a bit of a headache, but certainly much cleaner, and is a pretty uncommon occurrence. Having this plugin read the TDML file for attributes defined on the testSuite would be a good way to configure the runner. > We need to make sure we preserve the ability to skip some TDML test cases > with a string explaining why they are failing (usually a JIRA bug number). > If we comment out failing test cases directly in the TDML file then we take > the risk that these tests will bitrot faster once the TDML processor no > longer parses them. I looked at > daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd and I think > we could modify the TDML schema to support skipping failing test cases. Both > parserTestCase and unparserTestCase use the same attributeGroup > (testCaseAttribs). We could add a new optional string-type attribute > "reasonForSkipping" to testCaseAttribs and modify the sbt plugin or the TDML > processor to automatically skip such test cases (perhaps printing their names > and/or the reasons for skipping them, but we don't want to make the test > output too big). Agreed, parser/unparserTestCases definitely need a way to specify that this test is expected to fail and should be skipped. We sortof already have this capability with the "implementations" attribute. If that attribute doesn't contain "daffodil" that it is skipped when testing Daffodil. So you could do something like implementations="" or implementations="ibm" (if a test only works in IBM). Unfortunately, this doesn't required that bug information be included, which is pretty important. It would maybe be nice if a test would only be skipped if bug information were included. I wonder if maybe we need to rethink the "implementations" attribute? We might consider instead something like <implementation>-failure="details". So for example, a test that fails on Daffodil might include daffodil-failure="DAFFODIL-1234" This way, in order to have the TDML runner skip a test, you MUST include this attribute which can provide details. This detail can then be included in the output when a test is skipped. Additionally it makes it easy to grep for failed tests and related bug numbers. > Nice idea, it will be good to see how far it can reduce the size of the Scala > test boilerplate code! Oh, it occurs to me that sometimes it's useful to > debug a TDML test case directly from the IDE by running a Scala test method > under the debugger. How often do developers need to debug existing or new > TDML test cases, which would require them to write throw-away Scala test > methods by hand whenever necessary? Maybe we can code up some sort of > universal Scala entry point we can run from the IDE which can run any desired > TDML test case by editing (at most) one or two lines of code. Yeah, I think this is probably the the most important functionality to have, and probably isn't viable if IDE support isn't there. How does this integrate with IDEs? Maybe it "just works" for IDEs that support sbt? What about if an IDE supports BSP since our current version of SBT supports BSP now? But what about IDE's like eclipse. I'm not sure if any devs use Eclipse anymore since IntelliJ seems much better, but maybe something to consider? Can we claim we only support IDE's that support SBT/BSP? Also, SBT has the testOnly command to run a single testsuite/test. Is that supported with this? For example, currently we can run sbt testOnly org.apache.daffodil....TestSuite -- --tests=test_regex So it takes a test suite and an optional regex of tests in that suite to run. Having this similar functionality would be pretty critical. Presumably, it would be very similar to the above, but maybe just a path to the TDML file instead, e.g. sbt testOnly org/apache/daffodil/.../TestSuite.tdml -- --tests=test_regex Using the path to the tdml file as the test suite name avoids issues with duplicate test suite names in TDML files. Also makes for easily copy/pasting a path when you want to test a tdml file. Some other thoughts: Does anything output when a test fails? Or is that still to be implemented? I'm also curious how this works regarding Daffodil versions. Presumably this depends on a certain Daffodil version? What happens if a repo has a different Daffodil dependency? Does this override that? Or does this just expect that Daffodil is already a dependency and uses that version? And hopefully our TDMLRUnner API is stable enough that things just work? We definitely have cases where we need to run tests with different Daffodil versions, it would be nice if we didn't have to also change the plugin version to match. We also need to consider how this gets published. Do you plan to maintain and publish this separately as an outside plugin that Daffodil just depends on an uses? Or do you plant to have it merged into a Daffodil repo? If so, I wonder if it lives in the Daffodil repo so Daffodil always has the latest, and then this is published when there are Daffodil releases? Or maybe it's a separate Apache repo that follows the same Apache release process/requirements, but can be on a different schedule than Daffodil? If all the above issue can potentially be resolved, I think there's a good chance this could replace all of our boiler plate, which I'm in big favor of. > -----Original Message----- > From: John Wass <jwa...@gmail.com> > Sent: Friday, January 8, 2021 4:27 PM > To: dev@daffodil.apache.org > Subject: EXT: rfc: sbt-tdml > > https://github.com/jw3/sbt-tdml > I needed to get smarter on SBT plugins for a task and tinkered with something > in Daffodil to do it. > > The goal here was to remove the TDML test boilerplate. I threw this at the > daffodil-test project with surpsisingly sane results on the first try. See > the bottom of the readme. > > The learning process is pretty much complete, so not sure where this goes, > but wanted see if there are any comments. >
