Yes, I expect something will be needed. Not to debug the layer code itself, but to help a DFDL schema author cope with the layer behavior.
Some layers, like a decompressor or encoding change (eg base64, or even line-folding), the correspondence between fields and positions in the original data is 100% gone. The correspondence is between the fields and the output from the layer decoding process. So some sort of stack needs to set aside the current input stream, start a new nested data-stream concept for the layer-contained part of the schema, and then when that is over, resume the other data stream. For parsing a stack-discipline makes sense. Unparsing not so much, but unparsing is a whole separate can of worms. Other layers don't modify the data stream. at all, so there is no loss of correspondence. They just compute something from it like a checksum/CRC. Those show up to the DFDL parser as just some DFDL variable gets assigned a value. That sort of layer really has very little impact on the vscode environment - so long as it shows you variables, and when their values change. Once this stuff gets merged it will be useful to try to use the VSCode extension on the layer examples in order to try to figure out what features may be needed. One important detail is that layers apply to sequences, not elements. On Mon, Mar 25, 2024 at 11:26 AM Larry Barber <larry.bar...@nteligen.com> wrote: > Mike, > Are there any changes to support debugging the new layers stuff? > Something that could/should be added to enhance the VS debugger extension? > Larry > > -----Original Message----- > From: Mike Beckerle <mbecke...@apache.org> > Sent: Monday, March 18, 2024 11:22 AM > To: dev@daffodil.apache.org > Subject: Re: Discuss: Layer changes - 3.7.0 or 4.0.0 ? > > I think the compelling argument to get the layers stuff into 3.7.0 is that > we don't want any more layers built using 3.6.0-and-prior APIs. > > I tend to agree we should not make the next release 4.0.0 because we have > other things to get into that release, and the layers stuff is not an > official part of DFDL even if it is a practical necessity for many schemas. > > The easiest way to make-whole the current layer users is just to convert > the layers that exist, making new versions of them for Daffodil 3.7.0. I am > certainly more than happy to help people do that. > > I know Owl Cyber has perhaps 6 project-specific layers written. It's quite > easy to convert them all, and that's easier than dealing with co-existence. > > Truly nothing has come up about layers on our mailing lists that I recall, > excepting a long time ago someone outside of our core project team created > the first version of the byte-swap layer. > But that's now a Daffodil built-in layer, and so other than converting the > dfdlx layer-related properties from old-to-new, should still work the same. > > The risk is only that users can't upgrade as easily to 3.7.0. But > arguably this is no different than the upgrade where the package names all > changes to make OGSI components possible. Perhaps easier. > > > On Mon, Mar 18, 2024 at 4:48 AM Steve Lawrence <slawre...@apache.org> > wrote: > > > I don't think we should make our next release 4.0.0. There's a handful > > of changes that have been discussed that probably require a major > > version bump (e.g. drop Java 8 support, bump to Scala 2.13, merge sapi > > and japi into api, change directory structures/jar names, remove > > deprecated functions/properties), I imagine we want to do all that > > kind of stuff in 4.0.0. And we are about due for another release and I > > don't think we can get all of this in a short amount of time to turn > > 3.7.0 into 4.0.0. > > > > I'm not sure if it's worth trying to support both APIs. Sounds like it > > will be pretty complicated since the two APIs are so different. > > > > I think it comes down to whether or not we can comfortably claim the > > old layer API was experimental and if we can claim transitioning to > > the new API is a relatively minor change. Personally, I think we can, > > but if we want to play it safe and go strictly by semantic versioning > > we should delay until 4.0.0 and plan for that to be the next release. > > Maybe we can only focus on those changes for the next release and get > > 4.0.0 out relatively soon after 3.7.0. > > > > > > > > > > On 2024-03-16 07:40 PM, Mike Beckerle wrote: > > > Separate from the pull request review, I want to discuss whether > > > these layer changes, which are really a more or less rewrite of the > > > system go > > > > > > * in 3.7.0, > > > * delay them to 4.0.0 > > > > > > or come up with some co-existence strategy so they can be added to > > > 3.7.0 without displacing the existing stuff. > > > > > > I thought about trying to recast the existing layer stuff in terms > > > of the new stuff, and it's hard work. Functional equivalence, bug > > > for bug, would be impossible. > > > > > > Another possible trick is to just add the new stuff... using new > > > package names, and separate grammar classes, parser, unparser, etc. > > > In theory the two implementations, old and this new one, could > co-exist for a release. > > > > > > Another thought is to just punt on 3.7.0, and renumber so our next > > release > > > is 4.0.0, where we're allowed more to break things and not be > > > backward compatible. > > > > > > Thoughts on this? > > > > > > Mike Beckerle > > > Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup > > > Co-Chair | > > https://usg02.safelinks.protection.office365.us/?url=http%3A%2F%2Fwww. > > ogf.org%2Fogf%2Fdoku.php%2Fstandards%2Fdfdl%2Fdfdl&data=05%7C02%7Clarr > > y.barber%40nteligen.com%7C95d086e0b966480189af08dc475f8e65%7C379c214c5 > > c944e86a6062d047675f02a%7C0%7C0%7C638463722963030745%7CUnknown%7CTWFpb > > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > > %3D%7C0%7C%7C%7C&sdata=7f%2BOo7KR61dcLRmd%2Bdy6akkI8lV2eQZ6DygmM44OQog > > %3D&reserved=0 > > > Owl Cyber Defense | > > > https://usg02.safelinks.protection.office365.us/?url=http%3A%2F%2Fww > > > w.owlcyberdefense.com%2F&data=05%7C02%7Clarry.barber%40nteligen.com% > > > 7C95d086e0b966480189af08dc475f8e65%7C379c214c5c944e86a6062d047675f02 > > > a%7C0%7C0%7C638463722963036320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL > > > jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&s > > > data=CyeVzuCD5AD5uhIvPLFajNSZzgHyRPJrLPPlBWRjgvQ%3D&reserved=0 > > > > > > > >
