Latest push to the PR adds true ServiceLoader feature to dynamically discover and load the layer transformers.
This should now work with CLI, vscode, etc. There is still no Java API for writing a layer transformer in Java. I don't know that this is actually a requirement. On Wed, Oct 6, 2021 at 5:40 PM Larry Barber <larry.bar...@nteligen.com> wrote: > It seems like it would need to work with the CLI, debugger, as well as the > NiFi version > > -----Original Message----- > From: Mike Beckerle <mbecke...@apache.org> > Sent: Wednesday, October 6, 2021 4:38 PM > To: dev@daffodil.apache.org > Subject: Re: Please review PR for - Re: Design discussion - dynamically > loadable layers - DAFFODIL-1927 > > Good point. Won't work from the new debugger. I guess I have to figure out > the service loader stuff. > > On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence <slawre...@apache.org> > wrote: > > > Ah, right, I didn't think about registering it from within the test > > suite object. > > > > Though, that isn't possible for the CLI, and I'd guess it also isn't > > possible from the new VSCode debugger. > > > > > > On 10/6/21 4:09 PM, Mike Beckerle wrote: > > > Much easier than that. > > > > > > Look at TestAIS.scala: > > > > > > > > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgit > > hub.com%2Fapache%2Fdaffodil%2Fpull%2F651%2Ffiles%23diff-59f6bd4163627a > > a9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91&data=04%7C01%7C > > larry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c2 > > 14c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CT > > WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI > > 6Mn0%3D%7C3000&sdata=BRpeY%2FZZaW6n3hyZ8HJec%2BFv%2B1nFN5mBTgy6aHS > > gLpc%3D&reserved=0 > > > > > > You just call LayerCompilerRegistry.Register( > > > AISPayloadArmoringLayerCompiler) > > > From the test object. > > > > > > That's all an application using Daffodil with a custom layer has to > > > do also. > > > > > > Honestly, I almost prefer this to dynamic loading. > > > > > > > > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence <slawre...@apache.org> > > wrote: > > > > > >> Don't those only work because they are built in to Daffodil and > > >> pre-registered? If someone wants to keep their layer outside of > > >> Daffodil, it needs to be manually registered, but that can't be > > >> done by the TDML Runner or CLI? > > >> > > >> On 10/6/21 3:38 PM, Mike Beckerle wrote: > > >>> The TDML runner invoked from JUnit works fine. > > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 > > >>> examples in daffodil-test work this way, and anyone's DFDL schema > > >>> could do the same thing. > > >>> > > >>> I plan to update the ethernetIP and PCAP schemas to compute their > > >> checksums > > >>> this way. > > >>> > > >>> Only the CLI is left out. > > >>> > > >>> > > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > > >>> <slawre...@apache.org> > > >> wrote: > > >>> > > >>>> If we don't support dynamic loading and instead require that > > >>>> users manually register the layer, that means that externally > > >>>> defined layers cannot be tested using the TDML runner, right? It > > >>>> also means there's > > no > > >>>> way to run schemas with external layers using the CLI? > > >>>> > > >>>> I guess that's not an issue until there are external layers, so > > >>>> maybe not critical, but still feels pretty important. Maybe > > >>>> adding it to the next release is reasonable, since that's > > >>>> probably when external layers might start showing up? > > >>>> > > >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote: > > >>>>> I have done a bunch of improvements to the code for this layer > > loading > > >>>>> based on the prior PR. > > >>>>> > > >>>>> It is now stabilized and worth it to examine/reexamine the PR at > > >>>>> this > > >>>> point. > > >>>>> > > >>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F > > >>>>> %2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F651&data=04%7C01% > > >>>>> 7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc > > >>>>> %7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839 > > >>>>> %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ > > >>>>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7YEJPwd2M%2BaGjrgtu > > >>>>> 0z3bQL%2BN%2Fi1fCSFnu4PMeKH9MY%3D&reserved=0 > > >>>>> > > >>>>> I have also changed the goals here somewhat. I split out the > > >>>>> defining/creation of a Java API for layers, and true dynamic > > >>>>> loading > > >> via > > >>>>> service loaders into a separate JIRA ticket DAFFODIL-2567. > > >>>>> > > >>>>> I think that experience with what we have so far makes sense > > >>>>> before defining any Java API for doing this. I also think the > > >>>>> need for true dynamic detect and load behavior is far less > > >>>>> critical than the basic ability to define these in external jars > > >>>>> that aren't part of > > daffodil. > > >>>>> > > >>>>> This PR now includes just the modularization of layers, so they > > >>>>> can > > be > > >>>>> written in scala and defined in a jar outside of Daffodil, An > > >> application > > >>>>> must register them for use by calling > > >>>> LayerCompilerRegistry.register(...). > > >>>>> > > >>>>> All the built-in layers are in daffodil-runtime1-layers which is > > >>>>> a > > new > > >>>>> module. > > >>>>> > > >>>>> Other layers are defined in daffodil-test for testing that > > >>>> checksums/check > > >>>>> digits work. The example AIS payload encoding layer is also now > > defined > > >>>> and > > >>>>> tested in daffodil-test. > > >>>>> > > >>>>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle > > >>>>> <mbecke...@apache.org> > > >>>> wrote: > > >>>>> > > >>>>>> Please see: > > >>>>>> > > >>>> > > >> > > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fcwi > > ki.apache.org%2Fconfluence%2Fdisplay%2FDAFFODIL%2FProposal%253A%2BDyna > > mically%2Bloading%2BLayer%2BTransformations&data=04%7C01%7Clarry.b > > arber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c214c5c94 > > 4e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CTWFpbGZs > > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > > %7C3000&sdata=VG0y4Wrh3rkI2Ou8DzMu0rEgSHA7Oy4yeE%2FD1Vu7RbQ%3D& > > ;reserved=0 > > >>>>>> > > >>>>>> Also: > > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2 > > >>>>>> F%2Fissues.apache.org%2Fjira%2Fbrowse%2FDAFFODIL-1927&data= > > >>>>>> 04%7C01%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d > > >>>>>> 9890929fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149 > > >>>>>> 6871894833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi > > >>>>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Dm9bfd0 > > >>>>>> yosgB22nvGRMsbYvpAhx1LqsnobSsZQ9DR%2Fg%3D&reserved=0 > > >>>>>> > > >>>>>> See related PR: > > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2 > > >>>>>> F%2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F643&data=04%7C0 > > >>>>>> 1%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d989092 > > >>>>>> 9fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149687189 > > >>>>>> 4833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz > > >>>>>> IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=zD%2FWa3zLDGU > > >>>>>> XIohNzN6n6Ro2umRyKlfdUCT0lsXxyns%3D&reserved=0 which > > is > > >>>> for > > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2 > > >>>>>> F%2Fissues.apache.org%2Fjira%2Fbrowse%2FDAFFODIL-2221&data= > > >>>>>> 04%7C01%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d > > >>>>>> 9890929fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149 > > >>>>>> 6871894833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi > > >>>>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FPn%2 > > >>>>>> B76%2FJ4MdpkHE%2BEBRwaqyasOK%2B33TBpNW96l%2F7FeA%3D&reserve > > >>>>>> d=0 > > >>>>>> > > >>>>>> Please add comments or directly edit that wiki page, or discuss > > >>>>>> in > > >> this > > >>>>>> email thread if you prefer. > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>> > > >> > > >> > > > > > > > >
