I like the idea of removing the Scala TDML test boilerplate as much as possible. Does your sbt plugin handle the use case where some Scala test files define customized or multiple TDML runners in the same file? For example, see daffodil-test/src/test/scala/org/apache/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala where two runners are defined for the same TDML file (expressions.tdml, one standard runner and one customized runner with validateDFDLSchemas = false) and several other customized runners are defined (expressions2.tdml with compileAllTopLevel = true). Even if your sbt plugin can define only a standard runner for each TDML file, it would be nice to remove most of the Scala TDML test code except for the code using customized runners. I'm sure there will be some edge cases where the sbt plugin must not run a TDML file with a standard runner because it works only with a customized runner, but the ability to specify configuration files in TDML or tell the sbt plugin to skip some TDML files may help somewhat.
We need to make sure we preserve the ability to skip some TDML test cases with a string explaining why they are failing (usually a JIRA bug number). If we comment out failing test cases directly in the TDML file then we take the risk that these tests will bitrot faster once the TDML processor no longer parses them. I looked at daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd and I think we could modify the TDML schema to support skipping failing test cases. Both parserTestCase and unparserTestCase use the same attributeGroup (testCaseAttribs). We could add a new optional string-type attribute "reasonForSkipping" to testCaseAttribs and modify the sbt plugin or the TDML processor to automatically skip such test cases (perhaps printing their names and/or the reasons for skipping them, but we don't want to make the test output too big). Nice idea, it will be good to see how far it can reduce the size of the Scala test boilerplate code! Oh, it occurs to me that sometimes it's useful to debug a TDML test case directly from the IDE by running a Scala test method under the debugger. How often do developers need to debug existing or new TDML test cases, which would require them to write throw-away Scala test methods by hand whenever necessary? Maybe we can code up some sort of universal Scala entry point we can run from the IDE which can run any desired TDML test case by editing (at most) one or two lines of code. -----Original Message----- From: John Wass <jwa...@gmail.com> Sent: Friday, January 8, 2021 4:27 PM To: dev@daffodil.apache.org Subject: EXT: rfc: sbt-tdml https://github.com/jw3/sbt-tdml I needed to get smarter on SBT plugins for a task and tinkered with something in Daffodil to do it. The goal here was to remove the TDML test boilerplate. I threw this at the daffodil-test project with surpsisingly sane results on the first try. See the bottom of the readme. The learning process is pretty much complete, so not sure where this goes, but wanted see if there are any comments.
