Steve, thanks so much for going through this. I would go with your suggestion that it seems like we should stick with 3.5.0 for the release numbering.
There have been no objections to the creation of the release within the 72+ hours we've had this thread open, so we will proceed now to create RC1 and bring it for a vote. For our next feature release, we need to improve and firm up the API for all Daffodil features people are using (layering specifically) so that we can support it longer term. On Fri, Jun 9, 2023 at 8:33 AM Steve Lawrence <slawre...@apache.org> wrote: > I have pretty much completed updating the regression suite to work with > the latest 3.5.0 snapshot. Just a few to go, but they are variants of > schemas I have fixed, so I do no expect any additional regressions. > Below is a summary of the changes that were needed. Note that many of > the schema projecs have not been updated since much older versions of > Daffodil, like 2.6.0 and 3.1.0, so some changes were required that are > not specific to 3.5.0. Those are listed at towards the end. > > * DAFFODIL-1704: A number of schemas broke because we now ensure that > binary specified lengths are not larger than the associated types. I > think this is considered a DFDL schema bug and is a good regression that > schema authors should fix. > > * DAFFODIL-2683: Changes to OSGI broke a number of test suits. There are > a relatively small number of classes that affect the tests and the > needed changes are very simple. The relevant classes are: > > org.apache.daffodil.core.compiler.Compiler > org.apache.daffodil.lib.api.URISchemaSource > org.apache.daffodil.lib.api.WithDiagnostics > org.apache.daffodil.lib.exceptions.Abort > org.apache.daffodil.lib.exceptions.Assert > org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthKind > org.apache.daffodil.lib.schema.annotation.props.gen.LayerLengthUnits > org.apache.daffodil.lib.util.Misc > org.apache.daffodil.lib.xml.XMLUtils > org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType > org.apache.daffodil.runtime1.layers.ByteBufferExplicitLengthLayerTransform > org.apache.daffodil.runtime1.layers.LayerCompileInfo > org.apache.daffodil.runtime1.layers.LayerCompiler > org.apache.daffodil.runtime1.layers.LayerRuntimeInfo > org.apache.daffodil.runtime1.layers.LayerTransformerFactory > org.apache.daffodil.runtime1.processors.ParseOrUnparseState > org.apache.daffodil.runtime1.processors.VariableRuntimeData > org.apache.daffodil.runtime1.processors.parsers.ParseError > > Note that no changes are needed for the TDML runner or Java/Scala API > classes, which I sort of consider the "public" API which tries harder to > maintain backwards compatibility. > > Some of these, like the Compiler/Diagnostics/etc should probably just be > updated to the the more stable Scala/Java API. Others are related to > custom layers, but I consider that an experimental API that is still > subject to change. Considering this and the ease of changes (only import > lines needed to change), while it did cause regressions, I'm not sure it > rises to the level of a 4.0.0 release. If we required actual code > changes, or broke the TDMLRunner or Scala/Java API's which are more > "public", I think that would be a stronger argument. > > * DAFFODIL-2158: We changed behavior of textNumberCheckPolicy="strict" > so that the pattern is used to determine if a decimal point must exist > or not. For example, if text numbers are integers without a decimal, the > pattern should not contain a decimal. I believe this was just a bug in > Daffodil, and schema authors should update their schemas to either use a > lax policy or use a pattern that correctly matches their data. > > * DAFFODIL-2818: A regression related to parsing negative zero was found > and fixed. This bug was created and fixed in 3.5.0. > > * DAFFODIL-2607: Diagnostics were modified/improved when an enumeration > key was not found, which some tests checked for. The tests either need > to be less specific or check for the new error message. I don't think we > make a guarantee that diagnostics will not change. > > ==== > > The following changes were required due to Scala/Java API changes, but > those went through a deprecation process and were removed prior to > 3.5.0. Because it was prior to 3.5.0, I have not looked into exactly > which prior release the changes occurred: > > * If validation errors are created by a parse but no validation errors > are listed in a TDML test cases, the TDML test fails. I am actually not > sure if this is a change to Daffodil behavior or if depedent schema > update just included new validation checks that now create errors. This > may be a schema regression and not a Daffodil regression. > > * Some formats had custom tests that use an old Scala/Java API before > the introduction of InputSourceInputDataStream. This must be updated to > use the new API. > > * The setValidateDFDLSchemas function was removed from the Compiler > class, which a few classes used to enable schema validation. Schemas are > now always validated and the Java/Scala API does not have a way to > disable this. > > * The setDistinguishedRootNode function was removed from the Compiler > class. Instead the root node should be passed as a parameter to the > compileSource/File function, or withDistinguishedRootNode should be used > on the ProcessorFactory. > > * The ParseResult.isAtEnd function was removed from the ParseResult > class. Instead, the InputSourceDataInputStream.hasData function should > be used. > > * A number of test suites still use the old tdml.DFDLTestSuite instead > of the new tdml.Runner. > > > > > On 2023-06-06 07:42 PM, Mike Beckerle wrote: > > Topic: Should this be version 4.0.0, not 3.5.0? > > > > The OGSI-package-name-changes have broken many schemas because the test > > rigs used Daffodil internals, and some deprecated things were finally > > removed in the current 3.5.0 snapshot. > > > > But as a result, these schemas, many of which are non-public, but > > nevertheless can be counted, a long list, which used to work, don't work > > any more. > > It's possible there are false-negatives here. Nevertheless it is a long > > list. > > > > [fail] (04/62) dfdl-bmp > > [fail] (11/62) dfdl-examples-hexwords > > [fail] (12/62) dfdl-examples-self-descriptive-data > > [fail] (13/62) dfdl-examples-xslt-csv > > [fail] (22/62) dfdl-imf-icalendar-cse > > [fail] (23/62) dfdl-ipf-demo > > [fail] (29/62) dfdl-jpeg2000 > > [fail] (30/62) dfdl-jreap > > [fail] (34/62) dfdl-mil-std-2045 > > [fail] (38/62) dfdl-nitf > > [fail] (42/62) dfdl-plc4x-s7 > > [fail] (43/62) dfdl-png > > [fail] (46/62) dfdl-raster > > [fail] (49/62) dfdl-stanag-5516 > > [fail] (50/62) dfdl-stanag-5516-with-nact > > [fail] (53/62) dfdl-usmtf > > [fail] (54/62) dfdl-usmtf-generic > > [fail] (57/62) dfdl-vmf > > [fail] (59/62) dfdl-vmf-spock > > [fail] (61/62) ibm-dfdl-crosstester > > > > This suggests to me that user-created schemas have some fairly high > > likelihood of being clobbered. > > > > We changed the packages of many things. We can claim these were internal > > things, but there are many schemas that depend on Daffodil internal > things > > now. This needs to get fixed so we can torque the internals more without > > breaking things, but right now... things are broken. > > > > Every schema using layers, for example as we never hardened the API for > > layers, so as it has evolved, schemas have broken. > > > > > > > > On Mon, Jun 5, 2023 at 6:07 PM Mike Beckerle <mbecke...@apache.org> > wrote: > > > >> > >> All critical and blocker type bugs have been fixed including a couple > >> quite challenging ones in the last few weeks. > >> > >> I think it is time to create an RC for 3.5.0. > >> > >> Please discuss through 6pm EDT.US on Thursday June 8. (72 hours) and if > >> favorable we'll figure out who gets to create the RC. > >> > >> -mikeb > >> > >> Mike Beckerle > >> Apache Daffodil PMC | daffodil.apache.org > >> OGF DFDL Workgroup Co-Chair | > www.ogf.org/ogf/doku.php/standards/dfdl/dfdl > >> Owl Cyber Defense | www.owlcyberdefense.com > >> > >> > >> > > > >
