Agreed about the need for testing multiple versions.
It might be convenient if we could come up with a solution that doesn't require
multiple branches. Multiple branches requires extra work to keep the branches in
sync, and also makes it difficult to share common code or tests.
I'm imagining a project structure where there could be multiple sub-projects,
each one designed for a specific Daffodil version. The projects would share the
src and test directories, so would only differ in things like dependencies to
Daffodil or plugins.
And for situations where code differences are needed between versions (e.g.
plugins) we could have daffodil version specific directories, e.g.
src/main/scala # code shared between all daffodil versions
src/main/scala-daffodil3110 # code only used for daffodil 3.11.0 sub project
src/main/scala-daffodil400 # code only used for daffodil 4.0.0 sub project
This would also work for src/main/resources and src/test.
This is kindof similar to how sbt supports projects building with Scala 2.x vs
3.x, and I think is not too difficult to do in SBT plugins.
To support Daffodil plugins, I think we could have a new sbt-daffodil setting
that defines daffodil plugin dependencies and each subproject would mutate that
to depend on the right version.
So an alternative SBT configuration might look something like this:
name := "myFormat"
version := "1.0.0"
libraryDependencies := Seq(...) // normal non-plugin dependencies
enablePlugin(DaffodilPlugin)
daffodlPluginDependencies := Seq(
"com.example.layers" %% "checksum" % "1.0.0"
)
daffodilProjectVersion("3.11.0")
daffodilProjectVersion("4.0.0")
That would create subprojects for 3.11.0 and 4.0.0, so you could do something
like:
sbt daffodil3110/test
sbt daffodil400/test
And something like "sbt publish" would publish the main schema jar and saved
parsers if configured.
Each of those subprojects would depend on the "checksum" plugin but they would
depend on a slightly modified names (e.g. "checksum_daffodil3110" vs
"checksum_daffodil400"). Additional features would be needed to publish plugins
with multiple versions and mutate the name to match. I imagine that would work
similar to the multiple subprojects.
I think there's still alot of details to work out, and it needs some testing to
figure out if it will actually work, but I think in theory it's possible to not
require branches so common code and tests can be easily shared and all tests run
without needing to change branches. And I think it could be done without too
much boilerplate.
- Steve
On 2025-09-03 08:17 AM, Mike Beckerle wrote:
+1 for creating a 4.0.0 release
To me the biggest thing needed is not anything to hold up the release, it's
documentation on how to evolve a DFDL schema project on github (or similar)
and package server (e.g., like Artifactory) in such a way that it can be
maintained to work with Daffodil 3.7.0, 3.10.0, 3.11.0, and 4.0.0 (and
subsequent) releases simultaneously, easily rebuilt and tested in
regression, etc. I pick those because the API (specifically packages)
changed after 3.7.0, 3.10.0 is the last of the Scala 2.12 releases, 3.11.0
is Scala 2.13, and 4.0.0 is Scala 3.
There are two scenarios - First is a pure schema - single component. The
DFDLSchemas FakeTDL is one such.
The second is a complex multi-component schema. The PCAP schema on
DFDLSchemas is such. It uses EthernetIP as a component and that has a
Daffodil layer extension to compute IPv4 checksums. This should illustrate
all the challenges.
I think this methodology is going to require use of multiple git branches,
since there are clearly code changes required for the layers. I think these
use only standard TDML tests however, so ad-hoc test rigs shouldn't muddy
the waters. But despite these git branches, much of the schema will be
independent of them, and an objective should be to share what can be shared
so as not to do too much cross-branch duplication of changes.
On Tue, Sep 2, 2025 at 3:03 PM Steve Lawrence <[email protected]> wrote:
Hi all,
Although Daffodil 3.11.0 was release fairly recently, the current main
branch of
Daffodil has a number of major improvements, including switching to Scala
3,
dropping official support for Java 8 and 11, a much improved API, and a
number
of important bug fixes. With these major changes complete, I think now
would be
a good time to release Daffodil 4.0.0.
For reference, here is a page that describes all the breaking changes and
how to
migrate to the new API:
https://daffodil.apache.org/migration-guides/4.0.0/
We should also plan to release the Daffodil SBT plugin 1.5.0 concurrently,
since
it has modifications needed to work with Daffodil 4.0.0.
There are a dozen or so open pull requests to update dependencies that I
think
we can merge in the coming days in time for 4.0.0. Please let us know if
there
are any other issues you think should be fixed.
If there are no additional changes or objections, I will volunteer as the
release manager and plan to start the vote on Monday, Sep 8.
Thanks,
- Steve
