I think we need to get to the bottom of why ExtendedCharsets doesn't work
as easily as the zip-layer plugin across 3.11.0 and 3.10.0 (2.13 and 2.12).
I'm not sure why we would release 3.11.0 without understanding that. Do we
even know if this is due to the charset plugin code itself, daffodil 3.11.0
changes, or the sbt-daffodil plugin?
This is exactly the challenge I faced with evaluating 3.11.0-RC2, and your
instructions in the thread basically end with the conclusion that this
remains a mystery.
With that resolved, and sufficient instructions on a wiki page, and a link
to that to point people to it from the release notes page on the
daffodil-site, then I will revise my vote.
Until we have that, my vote has to be -1.
I've added some notes to your instructions in the thread below. Normally
user instructions go on the daffodil-site pages and developer instructions
on the wiki, but in this case I think a wiki page, reachable from the
release notes, is appropriate at least for now.
I agree in the future that sbt-daffodil and daffodil main library should be
released simultaneously or the sbt-daffodil first so that one can test more
easily. The plugin is now so essential to maintaining DFDL schema projects
that we can't really deal with them separately.
On Mon, Jun 2, 2025 at 1:19 PM Adams, Joshua
<jad...@owlcyberdefense.com.invalid> wrote:
> Mike,
>
> After re-reading your email I may have misinterpreted some things. Here
> is essentially the process I used to update all the schemas in our
> regression suite to work against Daffodil 3.11.0.
>
> For relatively simple schema projects (no Scala plugins):
>
> *
> Update all test rig code to be compatible with both Scala 2.12 and Scala
> 2.13
>
(It is not going to be immediately clear to people that this is even
possible.
We need to at least give people a link about how to modify scala code so
that it is 2.12 and 2.13 compatible, and then
if we assume that some of the test code requires conditionalization for
scala 2.12 vs. scala 2.13 - we need to provide a link to instructions on
how to cope with that.)
> *
> With updated version of sbt-daffodil, doing "sbt set
> daffodilPackageBinVersions := Seq("3.10.0", "3.11.0"); packageDaffodilBin"
> will build saved parsers for both Daffodil 3.10.0 and 3.11.0.
>
(How to get an updated version of sbt-daffodil? We need a githash here on
the main branch. This is a good reason for these instructions to be a wiki
page so this can be easily updated once the sbt-daffodil update is
released.)
>
> For a more complicated project containing Scala plugins:
>
> *
> Update all test rig code to be compatible with both Scala 2.12 and Scala
> 2.13
> *
> Update all plugin code to be compatible with both Scala 2.12 and Scala 2.13
> *
> Publish a 2.12 version of the plugins
> *
> Publish a 2.13 version of the plugins
> *
> With updated version of sbt-daffodil set daffodilVersion := "3.10.0" and
> daffodilPackageBinVersions := Seq("3.10.0") to build saved parsers against
> Daffodil 3.10.0.
>
(So this is important. You actually have to do two separate build
operations, not configure it to handle both at the same time? This
basically means two branches, or that's how I would expect to do it. )
> *
> With updated version of sbt-daffodil set daffodilVersion := "3.11.0" and
> daffodilPackageBinVersions := Seq("3.11.0") to build saved parsers against
> Daffodil 3.11.0.
>
> I believe that this is all I needed to do to get the schema projects
> working with Daffodil 3.11.0 while also being relatively compatible with
> previous versions. Note that for some Scala plugins like the zip layer
> transformer doing ""sbt set daffodilPackageBinVersions := Seq("3.10.0",
> "3.11.0"); packageDaffodilBin" was able to generate saved parsers for both
> 3.10.0 and 3.11.0. This did not work with projects that required the
> ExtendedCharsets plugin.
>
(- This issue - that some plugins work fine, but others don't. Needs a
release note.
- So what did you do for projects that require the ExtendedCharsets
plugin? Presumably some schema that uses it was tested. )
>
> I still think that right now it is inadvisable to attempt to target
> Daffodil 3.11.0 until the updated sbt-daffodil plugin is ready as currently
> it will not work at all if you attempt to target Daffodil 3.11.0.
It might also be possible to improve it's functionality regarding Scala
> pl;ugins, but we need to investigate why some plugins can work for both
> Scala 2.12 and 2.13, but not others.
>
(This mystery is a blocker. charset daffodil plugins are pretty darn
simple.)
>
> Should I add a wiki page detailing this process?
>
> Josh Adams
> ________________________________
> From: Adams, Joshua <jad...@owlcyberdefense.com.INVALID>
> Sent: Monday, June 2, 2025 11:53 AM
> To: dev@daffodil.apache.org <dev@daffodil.apache.org>
> Subject: Re: [VOTE] Release Apache Daffodil 3.11.0-rc2
>
> Mike,
>
> I think this might be a bit of a chicken and egg problem. Correct me if
> I'm wrong, but I believe we were waiting for the release of Daffodil 3.11.0
> before performing the next release of sbt-daffodil. Without the updates to
> sbt-daffodil, there a lot more manual steps that need to be taken. With
> the changes in the upcoming sbt-daffodil release the only change I've had
> to make for targeting older versions of Daffodil is to just set the
> corresponding daffodilVersion and daffodilPackageBinVersions in whatever
> schema project I'm in as well as it's dependencies.
>
> Maybe in the future it makes more sense to perform the release of
> sbt-daffodil (if changes are required) before releasing Daffodil so that we
> can provide better transition instrucitons?
>
> Josh
> ________________________________
> From: Mike Beckerle <mbecke...@apache.org>
> Sent: Monday, June 2, 2025 11:42 AM
> To: dev@daffodil.apache.org <dev@daffodil.apache.org>
> Subject: Re: [VOTE] Release Apache Daffodil 3.11.0-rc2
>
> As of right now I'm considering voting -1.
>
> I will change my vote if we provide sufficient documentation which may not
> require a new RC3. Rationale follows.
>
> I did not have time to thoroughly evaluate this release yet. But that's not
> for lack of trying. It is because it is entirely unclear to me how to
> evaluate it.
>
> What is the methodology going forward to maintain a DFDL schema that must
> work with say, Daffodil 3.5.0, Daffodil 3.9.0, and Daffodil 3.11.0, given
> plug-ins, and test-rig code. Test rig code is scala and must be ported to
> Scala 2.13.0. But, if the schema is to continue to work with Daffodil
> 3.10.0 or earlier then the code must remain Scala 2.12 linkable. Plugins
> must be compiled for scala 2.12 and 2.13 and maintained side by side like
> that.
>
> I'd like to see specific instructions on how to cope with this transition.
> What the sbt-daffodil plug in does for you, vs. what you must do yourself
> by branching your DFDL schema project is entirely unclear.
>
> Letting our users have to figure this out by trial-and-error is not
> reasonable, particularly if it is not even clear to PMC insiders like me
> how to best do this, or if it is even possible without maintaining a
> branch/fork of all one's DFDL schemas corresponding to each Daffodil
> release.
>
>
> On Mon, Jun 2, 2025 at 11:00 AM Kilo, Olabusayo
> <ok...@owlcyberdefense.com.invalid> wrote:
>
> > +1 (binding)
> >
> > I checked:
> >
> > [NOT OK] links in email are correct except
> > https://s.apache.org/daffodil-issues-3.11.0 which is giving a "can't
> view
> > this page" error. I'm not logged in to jira, but I assume users shouldn't
> > need a jira account to view the issues
> > [OK] verified Summary of Changes pages (maven, sbt, download and dnf)
> > [OK] verified download of binary and source
> > [OK] Successfully used apache-daffodil-3.11.0-bin.tgz
> > [OK] JavaDoc and ScalaDoc are correct
> >
> >
> > --
> > Lola Kilo
> > ________________________________
> > From: Adams, Joshua <jad...@owlcyberdefense.com.INVALID>
> > Sent: Friday, May 30, 2025 2:18 PM
> > To: dev@daffodil.apache.org <dev@daffodil.apache.org>
> > Subject: Re: [VOTE] Release Apache Daffodil 3.11.0-rc2
> >
> > [You don't often get email from jad...@owlcyberdefense.com.invalid.
> Learn
> > why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > +1
> >
> > I checked:
> >
> > [OK] hashes and signatures of source and helper binaries are correct
> > [OK] source and helper binaries are 100% reproducible
> > [OK] ~80 public and private DFDL schema projects pass tests
> >
> > Did notice one small bug, but not worth holding up a release - see
> > https://issues.apache.org/jira/browse/DAFFODIL-3001
> >
> > Also ran some preliminary performance measurements across some schema
> > projects. Seeing a consistent 5-10% improvement in throughput/latency
> > across both parsing and unparsing when compared to Daffodil 3.10.0.
> >
> > Josh Adams
> > ________________________________
> > From: Steve Lawrence <slawre...@apache.org>
> > Sent: Friday, May 30, 2025 7:36 AM
> > To: dev@daffodil.apache.org <dev@daffodil.apache.org>
> > Subject: Re: [VOTE] Release Apache Daffodil 3.11.0-rc2
> >
> > +1 (binding)
> >
> > I checked:
> >
> > [OK] hashes and signatures of source and helper binaries are correct
> > [OK] source and helper binaries are 100% reproducible
> > [OK] signature of git tag verifies
> > [OK] source release matches git tag (minus KEYS file)
> > [OK] source compiles and all tests pass (both en_US and de_DE)
> > [OK] helper tgz/zip/msi/rpm/maven have the same content except where
> > expected
> > [OK] src, binaries, and jars include correct LICENSE/NOTICE
> > [OK] RAT check passes
> > [OK] no unexpected binaries in source
> > [OK] rpm and msi install and run with basic usage
> > [OK] ~80 public and private DFDL schema projects pass tests
> > [OK] no issues found in JavaDoc and ScalaDoc
> > [OK] no open CVEs found using sbt dependencyCheck
> > [OK] Daffodil NiFi processor builds and tests pass
> > [OK] Daffodil SBT plugin builds and tests pass
> >
> >
> > On 2025-05-23 02:02 PM, Adams, Joshua wrote:
> > > Hi all,
> > >
> > > I'd like to call a vote to release Apache Daffodil 3.11.0-rc2.
> > >
> > > All distribution packages, including signatures, digests, etc. can be
> > found at:
> > >
> > > https://dist.apache.org/repos/dist/dev/daffodil/3.11.0-rc2/
> > >
> > > Staging artifacts can be found at:
> > >
> > >
> >
> https://repository.apache.org/content/repositories/orgapachedaffodil-1052/
> > >
> > > This release has been signed with PGP key 989D39300ACC1866,
> > corresponding to jad...@apache.org, which is included in the KEYS file
> > here:
> > >
> > > https://downloads.apache.org/daffodil/KEYS
> > >
> > > The release candidate has been tagged in git with v3.11.0-rc1.
> > >
> > > For reference, here is a list of all resolved JIRA issues tagged with
> > 3.11.0:
> > >
> > > https://s.apache.org/daffodil-issues-3.11.0
> > >
> > > For a summary of the changes in this release, see:
> > >
> > > https://daffodil.apache.org/releases/3.11.0/
> > >
> > > Please review and vote. The vote will be open for at least 72 hours
> > (Friday, 23 May 2025, 2:00 PM EST).
> > >
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove (and reason why)
> > >
> > > Thanks,
> > > - Josh
> >
> >
>
