It seems like it would need to work with the CLI, debugger, as well as the NiFi version
-----Original Message----- From: Mike Beckerle <mbecke...@apache.org> Sent: Wednesday, October 6, 2021 4:38 PM To: dev@daffodil.apache.org Subject: Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927 Good point. Won't work from the new debugger. I guess I have to figure out the service loader stuff. On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence <slawre...@apache.org> wrote: > Ah, right, I didn't think about registering it from within the test > suite object. > > Though, that isn't possible for the CLI, and I'd guess it also isn't > possible from the new VSCode debugger. > > > On 10/6/21 4:09 PM, Mike Beckerle wrote: > > Much easier than that. > > > > Look at TestAIS.scala: > > > > > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgit > hub.com%2Fapache%2Fdaffodil%2Fpull%2F651%2Ffiles%23diff-59f6bd4163627a > a9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91&data=04%7C01%7C > larry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c2 > 14c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CT > WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI > 6Mn0%3D%7C3000&sdata=BRpeY%2FZZaW6n3hyZ8HJec%2BFv%2B1nFN5mBTgy6aHS > gLpc%3D&reserved=0 > > > > You just call LayerCompilerRegistry.Register( > > AISPayloadArmoringLayerCompiler) > > From the test object. > > > > That's all an application using Daffodil with a custom layer has to > > do also. > > > > Honestly, I almost prefer this to dynamic loading. > > > > > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence <slawre...@apache.org> > wrote: > > > >> Don't those only work because they are built in to Daffodil and > >> pre-registered? If someone wants to keep their layer outside of > >> Daffodil, it needs to be manually registered, but that can't be > >> done by the TDML Runner or CLI? > >> > >> On 10/6/21 3:38 PM, Mike Beckerle wrote: > >>> The TDML runner invoked from JUnit works fine. > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 > >>> examples in daffodil-test work this way, and anyone's DFDL schema > >>> could do the same thing. > >>> > >>> I plan to update the ethernetIP and PCAP schemas to compute their > >> checksums > >>> this way. > >>> > >>> Only the CLI is left out. > >>> > >>> > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > >>> <slawre...@apache.org> > >> wrote: > >>> > >>>> If we don't support dynamic loading and instead require that > >>>> users manually register the layer, that means that externally > >>>> defined layers cannot be tested using the TDML runner, right? It > >>>> also means there's > no > >>>> way to run schemas with external layers using the CLI? > >>>> > >>>> I guess that's not an issue until there are external layers, so > >>>> maybe not critical, but still feels pretty important. Maybe > >>>> adding it to the next release is reasonable, since that's > >>>> probably when external layers might start showing up? > >>>> > >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote: > >>>>> I have done a bunch of improvements to the code for this layer > loading > >>>>> based on the prior PR. > >>>>> > >>>>> It is now stabilized and worth it to examine/reexamine the PR at > >>>>> this > >>>> point. > >>>>> > >>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F > >>>>> %2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F651&data=04%7C01% > >>>>> 7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc > >>>>> %7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839 > >>>>> %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ > >>>>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7YEJPwd2M%2BaGjrgtu > >>>>> 0z3bQL%2BN%2Fi1fCSFnu4PMeKH9MY%3D&reserved=0 > >>>>> > >>>>> I have also changed the goals here somewhat. I split out the > >>>>> defining/creation of a Java API for layers, and true dynamic > >>>>> loading > >> via > >>>>> service loaders into a separate JIRA ticket DAFFODIL-2567. > >>>>> > >>>>> I think that experience with what we have so far makes sense > >>>>> before defining any Java API for doing this. I also think the > >>>>> need for true dynamic detect and load behavior is far less > >>>>> critical than the basic ability to define these in external jars > >>>>> that aren't part of > daffodil. > >>>>> > >>>>> This PR now includes just the modularization of layers, so they > >>>>> can > be > >>>>> written in scala and defined in a jar outside of Daffodil, An > >> application > >>>>> must register them for use by calling > >>>> LayerCompilerRegistry.register(...). > >>>>> > >>>>> All the built-in layers are in daffodil-runtime1-layers which is > >>>>> a > new > >>>>> module. > >>>>> > >>>>> Other layers are defined in daffodil-test for testing that > >>>> checksums/check > >>>>> digits work. The example AIS payload encoding layer is also now > defined > >>>> and > >>>>> tested in daffodil-test. > >>>>> > >>>>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle > >>>>> <mbecke...@apache.org> > >>>> wrote: > >>>>> > >>>>>> Please see: > >>>>>> > >>>> > >> > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fcwi > ki.apache.org%2Fconfluence%2Fdisplay%2FDAFFODIL%2FProposal%253A%2BDyna > mically%2Bloading%2BLayer%2BTransformations&data=04%7C01%7Clarry.b > arber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c214c5c94 > 4e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CTWFpbGZs > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > %7C3000&sdata=VG0y4Wrh3rkI2Ou8DzMu0rEgSHA7Oy4yeE%2FD1Vu7RbQ%3D& > ;reserved=0 > >>>>>> > >>>>>> Also: > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2 > >>>>>> F%2Fissues.apache.org%2Fjira%2Fbrowse%2FDAFFODIL-1927&data= > >>>>>> 04%7C01%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d > >>>>>> 9890929fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149 > >>>>>> 6871894833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi > >>>>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Dm9bfd0 > >>>>>> yosgB22nvGRMsbYvpAhx1LqsnobSsZQ9DR%2Fg%3D&reserved=0 > >>>>>> > >>>>>> See related PR: > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2 > >>>>>> F%2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F643&data=04%7C0 > >>>>>> 1%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d989092 > >>>>>> 9fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149687189 > >>>>>> 4833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz > >>>>>> IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=zD%2FWa3zLDGU > >>>>>> XIohNzN6n6Ro2umRyKlfdUCT0lsXxyns%3D&reserved=0 which > is > >>>> for > >>>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2 > >>>>>> F%2Fissues.apache.org%2Fjira%2Fbrowse%2FDAFFODIL-2221&data= > >>>>>> 04%7C01%7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d > >>>>>> 9890929fc%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C63769149 > >>>>>> 6871894833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi > >>>>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FPn%2 > >>>>>> B76%2FJ4MdpkHE%2BEBRwaqyasOK%2B33TBpNW96l%2F7FeA%3D&reserve > >>>>>> d=0 > >>>>>> > >>>>>> Please add comments or directly edit that wiki page, or discuss > >>>>>> in > >> this > >>>>>> email thread if you prefer. > >>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>> > >> > >> > > > >
