Yeah, NiFi comes with slightly different issues. NiFi .nars usually embed
all
their dependencies, so when we build a NiFi .nar that uses Daffodil, that
version of the .nar is stuck with that version of Daffodil. This means
newer
nars with newer features cannot be used with older versions of Daffodil.
In
general this isn't an issue because we rarely add new features to the
nar--usually the only change in a new .nar version is bumping the Daffodil
version.
I believe NiFi has a way to deal with this, but it's a little bit more
complicated.
The idea is we would create a nar for every version of Daffodil we want to
support, and that nar literally just contains Dafodil jars and transitive
dependencies, no actual processor logic. We build a new one for every
version of
daffodil, for example
nifi-daffodil-impl-nar-4.0.0.nar
nifi-daffodil-impl-nar-3.11.0.nar
nifi-daffodil-impl-nar-3.10.0.nar
...
Then we have a single nar that provides the actually processor logic,
called
nifi-daffodil-nar. This version only changes when we add a new feature to
the
processor, nothing changes for new Daffodil releases. This new nar defines
a nar
dependency to "nifi-daffodil-impl" so that the nar can share the classpath
with
nifi-daffodil-impl-nar-XXX and find the daffodil nars. It would also need
to use
reflection to deal with any API differences between daffodil versions,
though
this is fairly minimal, and we already do something similar in our SBT
plugin.
From a user perspective, they now need to install two nars:
nifi-daffodil-nar
and a nifi-daffodil-impl-nar version of their choosing. So it's not too
much
effort, but does complicate things a bit.
We can build these nars so the work on any NiFi version (we actually
already do
that, so that's not an issue).
I think the question is how important is it be able to use newer
nifi-daffodil
features with older versions of Daffodil, and if all this extra
complication is
worth it. The last feature added to NiFi Daffodil was in v1.15 (added
support
for setting external variables), and that uses Daffodil 3.6.0 which is
pretty
old, so I'm not sure many users would need this, they just need to install
the
right version of the processor for the version of Daffodil they want. In
the
past this has been hard to actually know which version of Daffodil the
processor
uses, but I've added a matrix to the processor README which should help to
make
it more clear:
https://github.com/OwlCyberDefense/nifi-daffodil?tab=readme-ov-file#nifi-compatibility
So in most cases users can just find the version of Daffodil they want in
that
list and install the associated processor.
On 2025-09-03 09:07 AM, Sood, Harinder wrote:
We also need to consider how to provide releases for different Ni-FI
versions
Sincerely,
Harinder Sood
Senior Program Manager
[email protected]
240 805 4219
owlcyberdefense.com
The information contained in this transmission is for the personal and
confidential use of the individual or entity to which it is addressed. If
the reader is not the intended recipient, you are hereby notified that any
review, dissemination, or copying of this communication is strictly
prohibited. If you have received this transmission in error, please notify
the sender immediately.
-----Original Message-----
From: Steve Lawrence <[email protected]>
Sent: Wednesday, September 3, 2025 8:59 AM
To: [email protected]
Subject: Re: [DISCUSS] Release Apache Daffodil v4.0.0 and Apache
Daffodil SBT Plugin v1.5.0
Agreed about the need for testing multiple versions.
It might be convenient if we could come up with a solution that doesn't
require multiple branches. Multiple branches requires extra work to keep
the branches in sync, and also makes it difficult to share common code or
tests.
I'm imagining a project structure where there could be multiple
sub-projects, each one designed for a specific Daffodil version. The
projects would share the src and test directories, so would only differ in
things like dependencies to Daffodil or plugins.
And for situations where code differences are needed between versions
(e.g.
plugins) we could have daffodil version specific directories, e.g.
src/main/scala # code shared between all daffodil versions
src/main/scala-daffodil3110 # code only used for daffodil 3.11.0 sub
project
src/main/scala-daffodil400 # code only used for daffodil 4.0.0 sub
project
This would also work for src/main/resources and src/test.
This is kindof similar to how sbt supports projects building with Scala
2.x vs 3.x, and I think is not too difficult to do in SBT plugins.
To support Daffodil plugins, I think we could have a new sbt-daffodil
setting that defines daffodil plugin dependencies and each subproject would
mutate that to depend on the right version.
So an alternative SBT configuration might look something like this:
name := "myFormat"
version := "1.0.0"
libraryDependencies := Seq(...) // normal non-plugin dependencies
enablePlugin(DaffodilPlugin)
daffodlPluginDependencies := Seq(
"com.example.layers" %% "checksum" % "1.0.0"
)
daffodilProjectVersion("3.11.0")
daffodilProjectVersion("4.0.0")
That would create subprojects for 3.11.0 and 4.0.0, so you could do
something like:
sbt daffodil3110/test
sbt daffodil400/test
And something like "sbt publish" would publish the main schema jar and
saved parsers if configured.
Each of those subprojects would depend on the "checksum" plugin but they
would depend on a slightly modified names (e.g. "checksum_daffodil3110" vs
"checksum_daffodil400"). Additional features would be needed to publish
plugins with multiple versions and mutate the name to match. I imagine that
would work similar to the multiple subprojects.
I think there's still alot of details to work out, and it needs some
testing to figure out if it will actually work, but I think in theory it's
possible to not require branches so common code and tests can be easily
shared and all tests run without needing to change branches. And I think it
could be done without too much boilerplate.
- Steve
On 2025-09-03 08:17 AM, Mike Beckerle wrote:
+1 for creating a 4.0.0 release
To me the biggest thing needed is not anything to hold up the release,
it's documentation on how to evolve a DFDL schema project on github
(or similar) and package server (e.g., like Artifactory) in such a way
that it can be maintained to work with Daffodil 3.7.0, 3.10.0, 3.11.0,
and 4.0.0 (and
subsequent) releases simultaneously, easily rebuilt and tested in
regression, etc. I pick those because the API (specifically packages)
changed after 3.7.0, 3.10.0 is the last of the Scala 2.12 releases,
3.11.0 is Scala 2.13, and 4.0.0 is Scala 3.
There are two scenarios - First is a pure schema - single component.
The DFDLSchemas FakeTDL is one such.
The second is a complex multi-component schema. The PCAP schema on
DFDLSchemas is such. It uses EthernetIP as a component and that has a
Daffodil layer extension to compute IPv4 checksums. This should
illustrate all the challenges.
I think this methodology is going to require use of multiple git
branches, since there are clearly code changes required for the
layers. I think these use only standard TDML tests however, so ad-hoc
test rigs shouldn't muddy the waters. But despite these git branches,
much of the schema will be independent of them, and an objective
should be to share what can be shared so as not to do too much
cross-branch duplication of changes.
On Tue, Sep 2, 2025 at 3:03 PM Steve Lawrence <[email protected]>
wrote:
Hi all,
Although Daffodil 3.11.0 was release fairly recently, the current
main branch of Daffodil has a number of major improvements, including
switching to Scala 3, dropping official support for Java 8 and 11, a
much improved API, and a number of important bug fixes. With these
major changes complete, I think now would be a good time to release
Daffodil 4.0.0.
For reference, here is a page that describes all the breaking changes
and how to migrate to the new API:
https://daffodil.apache.org/migration-guides/4.0.0/
We should also plan to release the Daffodil SBT plugin 1.5.0
concurrently, since it has modifications needed to work with Daffodil
4.0.0.
There are a dozen or so open pull requests to update dependencies
that I think we can merge in the coming days in time for 4.0.0.
Please let us know if there are any other issues you think should be
fixed.
If there are no additional changes or objections, I will volunteer as
the release manager and plan to start the vote on Monday, Sep 8.
Thanks,
- Steve
