Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Latest push to the PR adds true ServiceLoader feature to dynamically discover and load the layer transformers. This should now work with CLI, vscode, etc. There is still no Java API for writing a layer transformer in Java. I don't know that this is actually a requirement. On Wed, Oct 6, 2021 at 5:40 PM Larry Barber wrote: > It seems like it would need to work with the CLI, debugger, as well as the > NiFi version > > -Original Message- > From: Mike Beckerle > Sent: Wednesday, October 6, 2021 4:38 PM > To: dev@daffodil.apache.org > Subject: Re: Please review PR for - Re: Design discussion - dynamically > loadable layers - DAFFODIL-1927 > > Good point. Won't work from the new debugger. I guess I have to figure out > the service loader stuff. > > On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence > wrote: > > > Ah, right, I didn't think about registering it from within the test > > suite object. > > > > Though, that isn't possible for the CLI, and I'd guess it also isn't > > possible from the new VSCode debugger. > > > > > > On 10/6/21 4:09 PM, Mike Beckerle wrote: > > > Much easier than that. > > > > > > Look at TestAIS.scala: > > > > > > > > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgit > > hub.com%2Fapache%2Fdaffodil%2Fpull%2F651%2Ffiles%23diff-59f6bd4163627a > > a9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91data=04%7C01%7C > > larry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c2 > > 14c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CT > > WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI > > 6Mn0%3D%7C3000sdata=BRpeY%2FZZaW6n3hyZ8HJec%2BFv%2B1nFN5mBTgy6aHS > > gLpc%3Dreserved=0 > > > > > > You just call LayerCompilerRegistry.Register( > > > AISPayloadArmoringLayerCompiler) > > > From the test object. > > > > > > That's all an application using Daffodil with a custom layer has to > > > do also. > > > > > > Honestly, I almost prefer this to dynamic loading. > > > > > > > > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence > > wrote: > > > > > >> Don't those only work because they are built in to Daffodil and > > >> pre-registered? If someone wants to keep their layer outside of > > >> Daffodil, it needs to be manually registered, but that can't be > > >> done by the TDML Runner or CLI? > > >> > > >> On 10/6/21 3:38 PM, Mike Beckerle wrote: > > >>> The TDML runner invoked from JUnit works fine. > > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 > > >>> examples in daffodil-test work this way, and anyone's DFDL schema > > >>> could do the same thing. > > >>> > > >>> I plan to update the ethernetIP and PCAP schemas to compute their > > >> checksums > > >>> this way. > > >>> > > >>> Only the CLI is left out. > > >>> > > >>> > > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > > >>> > > >> wrote: > > >>> > > >>>> If we don't support dynamic loading and instead require that > > >>>> users manually register the layer, that means that externally > > >>>> defined layers cannot be tested using the TDML runner, right? It > > >>>> also means there's > > no > > >>>> way to run schemas with external layers using the CLI? > > >>>> > > >>>> I guess that's not an issue until there are external layers, so > > >>>> maybe not critical, but still feels pretty important. Maybe > > >>>> adding it to the next release is reasonable, since that's > > >>>> probably when external layers might start showing up? > > >>>> > > >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote: > > >>>>> I have done a bunch of improvements to the code for this layer > > loading > > >>>>> based on the prior PR. > > >>>>> > > >>>>> It is now stabilized and worth it to examine/reexamine the PR at > > >>>>> this > > >>>> point. > > >>>>> > > >>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F > > >>>>> %2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F651data=04%7C01% > > >>>>> 7Clarry.barber%40nte
RE: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
It seems like it would need to work with the CLI, debugger, as well as the NiFi version -Original Message- From: Mike Beckerle Sent: Wednesday, October 6, 2021 4:38 PM To: dev@daffodil.apache.org Subject: Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927 Good point. Won't work from the new debugger. I guess I have to figure out the service loader stuff. On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence wrote: > Ah, right, I didn't think about registering it from within the test > suite object. > > Though, that isn't possible for the CLI, and I'd guess it also isn't > possible from the new VSCode debugger. > > > On 10/6/21 4:09 PM, Mike Beckerle wrote: > > Much easier than that. > > > > Look at TestAIS.scala: > > > > > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgit > hub.com%2Fapache%2Fdaffodil%2Fpull%2F651%2Ffiles%23diff-59f6bd4163627a > a9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91data=04%7C01%7C > larry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c2 > 14c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CT > WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI > 6Mn0%3D%7C3000sdata=BRpeY%2FZZaW6n3hyZ8HJec%2BFv%2B1nFN5mBTgy6aHS > gLpc%3Dreserved=0 > > > > You just call LayerCompilerRegistry.Register( > > AISPayloadArmoringLayerCompiler) > > From the test object. > > > > That's all an application using Daffodil with a custom layer has to > > do also. > > > > Honestly, I almost prefer this to dynamic loading. > > > > > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence > wrote: > > > >> Don't those only work because they are built in to Daffodil and > >> pre-registered? If someone wants to keep their layer outside of > >> Daffodil, it needs to be manually registered, but that can't be > >> done by the TDML Runner or CLI? > >> > >> On 10/6/21 3:38 PM, Mike Beckerle wrote: > >>> The TDML runner invoked from JUnit works fine. > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 > >>> examples in daffodil-test work this way, and anyone's DFDL schema > >>> could do the same thing. > >>> > >>> I plan to update the ethernetIP and PCAP schemas to compute their > >> checksums > >>> this way. > >>> > >>> Only the CLI is left out. > >>> > >>> > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > >>> > >> wrote: > >>> > >>>> If we don't support dynamic loading and instead require that > >>>> users manually register the layer, that means that externally > >>>> defined layers cannot be tested using the TDML runner, right? It > >>>> also means there's > no > >>>> way to run schemas with external layers using the CLI? > >>>> > >>>> I guess that's not an issue until there are external layers, so > >>>> maybe not critical, but still feels pretty important. Maybe > >>>> adding it to the next release is reasonable, since that's > >>>> probably when external layers might start showing up? > >>>> > >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote: > >>>>> I have done a bunch of improvements to the code for this layer > loading > >>>>> based on the prior PR. > >>>>> > >>>>> It is now stabilized and worth it to examine/reexamine the PR at > >>>>> this > >>>> point. > >>>>> > >>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F > >>>>> %2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F651data=04%7C01% > >>>>> 7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc > >>>>> %7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839 > >>>>> %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ > >>>>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=7YEJPwd2M%2BaGjrgtu > >>>>> 0z3bQL%2BN%2Fi1fCSFnu4PMeKH9MY%3Dreserved=0 > >>>>> > >>>>> I have also changed the goals here somewhat. I split out the > >>>>> defining/creation of a Java API for layers, and true dynamic > >>>>> loading > >> via > >>>>> service loaders into a separate JIRA ticket DAFFODIL-2567. > >>>>> > >>>>> I think that experie
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Good point. Won't work from the new debugger. I guess I have to figure out the service loader stuff. On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence wrote: > Ah, right, I didn't think about registering it from within the test > suite object. > > Though, that isn't possible for the CLI, and I'd guess it also isn't > possible from the new VSCode debugger. > > > On 10/6/21 4:09 PM, Mike Beckerle wrote: > > Much easier than that. > > > > Look at TestAIS.scala: > > > > > https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91 > > > > You just call LayerCompilerRegistry.Register( > > AISPayloadArmoringLayerCompiler) > > From the test object. > > > > That's all an application using Daffodil with a custom layer has to do > > also. > > > > Honestly, I almost prefer this to dynamic loading. > > > > > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence > wrote: > > > >> Don't those only work because they are built in to Daffodil and > >> pre-registered? If someone wants to keep their layer outside of > >> Daffodil, it needs to be manually registered, but that can't be done by > >> the TDML Runner or CLI? > >> > >> On 10/6/21 3:38 PM, Mike Beckerle wrote: > >>> The TDML runner invoked from JUnit works fine. > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in > >>> daffodil-test work this way, and anyone's DFDL schema could do the same > >>> thing. > >>> > >>> I plan to update the ethernetIP and PCAP schemas to compute their > >> checksums > >>> this way. > >>> > >>> Only the CLI is left out. > >>> > >>> > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > >> wrote: > >>> > >>>> If we don't support dynamic loading and instead require that users > >>>> manually register the layer, that means that externally defined layers > >>>> cannot be tested using the TDML runner, right? It also means there's > no > >>>> way to run schemas with external layers using the CLI? > >>>> > >>>> I guess that's not an issue until there are external layers, so maybe > >>>> not critical, but still feels pretty important. Maybe adding it to the > >>>> next release is reasonable, since that's probably when external layers > >>>> might start showing up? > >>>> > >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote: > >>>>> I have done a bunch of improvements to the code for this layer > loading > >>>>> based on the prior PR. > >>>>> > >>>>> It is now stabilized and worth it to examine/reexamine the PR at this > >>>> point. > >>>>> > >>>>> https://github.com/apache/daffodil/pull/651 > >>>>> > >>>>> I have also changed the goals here somewhat. I split out the > >>>>> defining/creation of a Java API for layers, and true dynamic loading > >> via > >>>>> service loaders into a separate JIRA ticket DAFFODIL-2567. > >>>>> > >>>>> I think that experience with what we have so far makes sense before > >>>>> defining any Java API for doing this. I also think the need for true > >>>>> dynamic detect and load behavior is far less critical than the basic > >>>>> ability to define these in external jars that aren't part of > daffodil. > >>>>> > >>>>> This PR now includes just the modularization of layers, so they can > be > >>>>> written in scala and defined in a jar outside of Daffodil, An > >> application > >>>>> must register them for use by calling > >>>> LayerCompilerRegistry.register(...). > >>>>> > >>>>> All the built-in layers are in daffodil-runtime1-layers which is a > new > >>>>> module. > >>>>> > >>>>> Other layers are defined in daffodil-test for testing that > >>>> checksums/check > >>>>> digits work. The example AIS payload encoding layer is also now > defined > >>>> and > >>>>> tested in daffodil-test. > >>>>> > >>>>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle > >>>> wrote: > >>>>> > >>>>>> Please see: > >>>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations > >>>>>> > >>>>>> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 > >>>>>> > >>>>>> See related PR: https://github.com/apache/daffodil/pull/643 which > is > >>>> for > >>>>>> https://issues.apache.org/jira/browse/DAFFODIL-2221 > >>>>>> > >>>>>> Please add comments or directly edit that wiki page, or discuss in > >> this > >>>>>> email thread if you prefer. > >>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>> > >> > >> > > > >
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Ah, right, I didn't think about registering it from within the test suite object. Though, that isn't possible for the CLI, and I'd guess it also isn't possible from the new VSCode debugger. On 10/6/21 4:09 PM, Mike Beckerle wrote: Much easier than that. Look at TestAIS.scala: https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91 You just call LayerCompilerRegistry.Register( AISPayloadArmoringLayerCompiler) From the test object. That's all an application using Daffodil with a custom layer has to do also. Honestly, I almost prefer this to dynamic loading. On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence wrote: Don't those only work because they are built in to Daffodil and pre-registered? If someone wants to keep their layer outside of Daffodil, it needs to be manually registered, but that can't be done by the TDML Runner or CLI? On 10/6/21 3:38 PM, Mike Beckerle wrote: The TDML runner invoked from JUnit works fine. The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in daffodil-test work this way, and anyone's DFDL schema could do the same thing. I plan to update the ethernetIP and PCAP schemas to compute their checksums this way. Only the CLI is left out. On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence wrote: If we don't support dynamic loading and instead require that users manually register the layer, that means that externally defined layers cannot be tested using the TDML runner, right? It also means there's no way to run schemas with external layers using the CLI? I guess that's not an issue until there are external layers, so maybe not critical, but still feels pretty important. Maybe adding it to the next release is reasonable, since that's probably when external layers might start showing up? On 10/6/21 1:47 PM, Mike Beckerle wrote: I have done a bunch of improvements to the code for this layer loading based on the prior PR. It is now stabilized and worth it to examine/reexamine the PR at this point. https://github.com/apache/daffodil/pull/651 I have also changed the goals here somewhat. I split out the defining/creation of a Java API for layers, and true dynamic loading via service loaders into a separate JIRA ticket DAFFODIL-2567. I think that experience with what we have so far makes sense before defining any Java API for doing this. I also think the need for true dynamic detect and load behavior is far less critical than the basic ability to define these in external jars that aren't part of daffodil. This PR now includes just the modularization of layers, so they can be written in scala and defined in a jar outside of Daffodil, An application must register them for use by calling LayerCompilerRegistry.register(...). All the built-in layers are in daffodil-runtime1-layers which is a new module. Other layers are defined in daffodil-test for testing that checksums/check digits work. The example AIS payload encoding layer is also now defined and tested in daffodil-test. On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote: Please see: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 See related PR: https://github.com/apache/daffodil/pull/643 which is for https://issues.apache.org/jira/browse/DAFFODIL-2221 Please add comments or directly edit that wiki page, or discuss in this email thread if you prefer.
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Much easier than that. Look at TestAIS.scala: https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91 You just call LayerCompilerRegistry.Register( AISPayloadArmoringLayerCompiler) >From the test object. That's all an application using Daffodil with a custom layer has to do also. Honestly, I almost prefer this to dynamic loading. On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence wrote: > Don't those only work because they are built in to Daffodil and > pre-registered? If someone wants to keep their layer outside of > Daffodil, it needs to be manually registered, but that can't be done by > the TDML Runner or CLI? > > On 10/6/21 3:38 PM, Mike Beckerle wrote: > > The TDML runner invoked from JUnit works fine. > > The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in > > daffodil-test work this way, and anyone's DFDL schema could do the same > > thing. > > > > I plan to update the ethernetIP and PCAP schemas to compute their > checksums > > this way. > > > > Only the CLI is left out. > > > > > > On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > wrote: > > > >> If we don't support dynamic loading and instead require that users > >> manually register the layer, that means that externally defined layers > >> cannot be tested using the TDML runner, right? It also means there's no > >> way to run schemas with external layers using the CLI? > >> > >> I guess that's not an issue until there are external layers, so maybe > >> not critical, but still feels pretty important. Maybe adding it to the > >> next release is reasonable, since that's probably when external layers > >> might start showing up? > >> > >> On 10/6/21 1:47 PM, Mike Beckerle wrote: > >>> I have done a bunch of improvements to the code for this layer loading > >>> based on the prior PR. > >>> > >>> It is now stabilized and worth it to examine/reexamine the PR at this > >> point. > >>> > >>> https://github.com/apache/daffodil/pull/651 > >>> > >>> I have also changed the goals here somewhat. I split out the > >>> defining/creation of a Java API for layers, and true dynamic loading > via > >>> service loaders into a separate JIRA ticket DAFFODIL-2567. > >>> > >>> I think that experience with what we have so far makes sense before > >>> defining any Java API for doing this. I also think the need for true > >>> dynamic detect and load behavior is far less critical than the basic > >>> ability to define these in external jars that aren't part of daffodil. > >>> > >>> This PR now includes just the modularization of layers, so they can be > >>> written in scala and defined in a jar outside of Daffodil, An > application > >>> must register them for use by calling > >> LayerCompilerRegistry.register(...). > >>> > >>> All the built-in layers are in daffodil-runtime1-layers which is a new > >>> module. > >>> > >>> Other layers are defined in daffodil-test for testing that > >> checksums/check > >>> digits work. The example AIS payload encoding layer is also now defined > >> and > >>> tested in daffodil-test. > >>> > >>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle > >> wrote: > >>> > >>>> Please see: > >>>> > >> > https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations > >>>> > >>>> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 > >>>> > >>>> See related PR: https://github.com/apache/daffodil/pull/643 which is > >> for > >>>> https://issues.apache.org/jira/browse/DAFFODIL-2221 > >>>> > >>>> Please add comments or directly edit that wiki page, or discuss in > this > >>>> email thread if you prefer. > >>>> > >>>> > >>> > >> > >> > > > >
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Don't those only work because they are built in to Daffodil and pre-registered? If someone wants to keep their layer outside of Daffodil, it needs to be manually registered, but that can't be done by the TDML Runner or CLI? On 10/6/21 3:38 PM, Mike Beckerle wrote: The TDML runner invoked from JUnit works fine. The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in daffodil-test work this way, and anyone's DFDL schema could do the same thing. I plan to update the ethernetIP and PCAP schemas to compute their checksums this way. Only the CLI is left out. On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence wrote: If we don't support dynamic loading and instead require that users manually register the layer, that means that externally defined layers cannot be tested using the TDML runner, right? It also means there's no way to run schemas with external layers using the CLI? I guess that's not an issue until there are external layers, so maybe not critical, but still feels pretty important. Maybe adding it to the next release is reasonable, since that's probably when external layers might start showing up? On 10/6/21 1:47 PM, Mike Beckerle wrote: I have done a bunch of improvements to the code for this layer loading based on the prior PR. It is now stabilized and worth it to examine/reexamine the PR at this point. https://github.com/apache/daffodil/pull/651 I have also changed the goals here somewhat. I split out the defining/creation of a Java API for layers, and true dynamic loading via service loaders into a separate JIRA ticket DAFFODIL-2567. I think that experience with what we have so far makes sense before defining any Java API for doing this. I also think the need for true dynamic detect and load behavior is far less critical than the basic ability to define these in external jars that aren't part of daffodil. This PR now includes just the modularization of layers, so they can be written in scala and defined in a jar outside of Daffodil, An application must register them for use by calling LayerCompilerRegistry.register(...). All the built-in layers are in daffodil-runtime1-layers which is a new module. Other layers are defined in daffodil-test for testing that checksums/check digits work. The example AIS payload encoding layer is also now defined and tested in daffodil-test. On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote: Please see: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 See related PR: https://github.com/apache/daffodil/pull/643 which is for https://issues.apache.org/jira/browse/DAFFODIL-2221 Please add comments or directly edit that wiki page, or discuss in this email thread if you prefer.
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
The TDML runner invoked from JUnit works fine. The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in daffodil-test work this way, and anyone's DFDL schema could do the same thing. I plan to update the ethernetIP and PCAP schemas to compute their checksums this way. Only the CLI is left out. On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence wrote: > If we don't support dynamic loading and instead require that users > manually register the layer, that means that externally defined layers > cannot be tested using the TDML runner, right? It also means there's no > way to run schemas with external layers using the CLI? > > I guess that's not an issue until there are external layers, so maybe > not critical, but still feels pretty important. Maybe adding it to the > next release is reasonable, since that's probably when external layers > might start showing up? > > On 10/6/21 1:47 PM, Mike Beckerle wrote: > > I have done a bunch of improvements to the code for this layer loading > > based on the prior PR. > > > > It is now stabilized and worth it to examine/reexamine the PR at this > point. > > > > https://github.com/apache/daffodil/pull/651 > > > > I have also changed the goals here somewhat. I split out the > > defining/creation of a Java API for layers, and true dynamic loading via > > service loaders into a separate JIRA ticket DAFFODIL-2567. > > > > I think that experience with what we have so far makes sense before > > defining any Java API for doing this. I also think the need for true > > dynamic detect and load behavior is far less critical than the basic > > ability to define these in external jars that aren't part of daffodil. > > > > This PR now includes just the modularization of layers, so they can be > > written in scala and defined in a jar outside of Daffodil, An application > > must register them for use by calling > LayerCompilerRegistry.register(...). > > > > All the built-in layers are in daffodil-runtime1-layers which is a new > > module. > > > > Other layers are defined in daffodil-test for testing that > checksums/check > > digits work. The example AIS payload encoding layer is also now defined > and > > tested in daffodil-test. > > > > On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle > wrote: > > > >> Please see: > >> > https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations > >> > >> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 > >> > >> See related PR: https://github.com/apache/daffodil/pull/643 which is > for > >> https://issues.apache.org/jira/browse/DAFFODIL-2221 > >> > >> Please add comments or directly edit that wiki page, or discuss in this > >> email thread if you prefer. > >> > >> > > > >
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
If we don't support dynamic loading and instead require that users manually register the layer, that means that externally defined layers cannot be tested using the TDML runner, right? It also means there's no way to run schemas with external layers using the CLI? I guess that's not an issue until there are external layers, so maybe not critical, but still feels pretty important. Maybe adding it to the next release is reasonable, since that's probably when external layers might start showing up? On 10/6/21 1:47 PM, Mike Beckerle wrote: I have done a bunch of improvements to the code for this layer loading based on the prior PR. It is now stabilized and worth it to examine/reexamine the PR at this point. https://github.com/apache/daffodil/pull/651 I have also changed the goals here somewhat. I split out the defining/creation of a Java API for layers, and true dynamic loading via service loaders into a separate JIRA ticket DAFFODIL-2567. I think that experience with what we have so far makes sense before defining any Java API for doing this. I also think the need for true dynamic detect and load behavior is far less critical than the basic ability to define these in external jars that aren't part of daffodil. This PR now includes just the modularization of layers, so they can be written in scala and defined in a jar outside of Daffodil, An application must register them for use by calling LayerCompilerRegistry.register(...). All the built-in layers are in daffodil-runtime1-layers which is a new module. Other layers are defined in daffodil-test for testing that checksums/check digits work. The example AIS payload encoding layer is also now defined and tested in daffodil-test. On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote: Please see: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 See related PR: https://github.com/apache/daffodil/pull/643 which is for https://issues.apache.org/jira/browse/DAFFODIL-2221 Please add comments or directly edit that wiki page, or discuss in this email thread if you prefer.
Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
I have done a bunch of improvements to the code for this layer loading based on the prior PR. It is now stabilized and worth it to examine/reexamine the PR at this point. https://github.com/apache/daffodil/pull/651 I have also changed the goals here somewhat. I split out the defining/creation of a Java API for layers, and true dynamic loading via service loaders into a separate JIRA ticket DAFFODIL-2567. I think that experience with what we have so far makes sense before defining any Java API for doing this. I also think the need for true dynamic detect and load behavior is far less critical than the basic ability to define these in external jars that aren't part of daffodil. This PR now includes just the modularization of layers, so they can be written in scala and defined in a jar outside of Daffodil, An application must register them for use by calling LayerCompilerRegistry.register(...). All the built-in layers are in daffodil-runtime1-layers which is a new module. Other layers are defined in daffodil-test for testing that checksums/check digits work. The example AIS payload encoding layer is also now defined and tested in daffodil-test. On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote: > Please see: > https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations > > Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 > > See related PR: https://github.com/apache/daffodil/pull/643 which is for > https://issues.apache.org/jira/browse/DAFFODIL-2221 > > Please add comments or directly edit that wiki page, or discuss in this > email thread if you prefer. > >
Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
In the PR review of the changes for DAFFODIL-2221, a key observation is
that the ipv4 and checkdigits layers defined there are in need of far too
many Daffodil internal data structures. We don't expose PState/UState, and
especially don't want to expose the unparser suspensable operation stuff as
a supported API.
So those examples need to be refactored so that the layer definition is
highly isolated from almost all aspects of Daffodil internals.
I believe a layer should be able to be defined using only:
0) the name of the layer transform
1) what are the QNames of any DFDL variables the layer needs to read.
(Could be none)
2) what is the QName of the DFDL variable the layer needs to write.
3) a method that given a mutable ByteBuffer, and a parse/unparse flag,
computes
(a) the checksum/CRC/parity output (and returns it) - both for parse and
unparse.
(b) when unparsing, modifies the byte buffer with the actual bytes to be
output, if it is different from what is provided to the method.
Everything else should be daffodil-internal framework code.
e.g.,
final class MyExampleLayerTransform()
extends ByteBufferExplicitLengthLayerTransform("myExample",
Seq("pre:nameOfVarRead"), "pre:nameOfVarWritten") {
override def doTransform(isUnparse: Boolean, bytes: ByteBuffer): Int = {
???
}
}
Or something along those lines. This is only for small-message cases. A
data format with large blobs to be checksummed, or CRCs over large
file-sized data, needs a different abstraction than a byte buffer, and of
course there are more length kinds than just 'explicit'.
On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote:
> Please see:
> https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations
>
> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927
>
> See related PR: https://github.com/apache/daffodil/pull/643 which is for
> https://issues.apache.org/jira/browse/DAFFODIL-2221
>
> Please add comments or directly edit that wiki page, or discuss in this
> email thread if you prefer.
>
>
Design discussion - dynamically loadable layers - DAFFODIL-1927
Please see: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 See related PR: https://github.com/apache/daffodil/pull/643 which is for https://issues.apache.org/jira/browse/DAFFODIL-2221 Please add comments or directly edit that wiki page, or discuss in this email thread if you prefer.
Re: DAFFODIL-1927
This is an important change. There is design work to do here for how it should work. There are two features in Daffodil that dynamically load extensions already: User-defined functions, and Validators. So you should be able to look at those to figure out how to dynamically load layer implementations from the classpath. One possible approach is to take one of the existing layering transforms, like the one for AIS payload armoring, and make that into a dynamically loaded one. There are already tests for that, so you would be able to tell easily if your dynamically loaded version works the same without having to invent a lot of tests. And AIS is an obscure and special-purpose thing that really should​ be outside of daffodil as a loadable layer transform. Arguably, all of the layer transforms should be in external libraries that live in separate jars. See daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/AISTransformer.scala You'll see that these are "registered" in the file LayerTransformer.scala, and that mechanism is what must change to get the transformer from an external classpath/jar. I am happy if the layer transformers have to be written in Scala for now. A later enhancement can be to make a polished Java API for these extensions. The UDF feature allows writing UDFs in either Java or Scala, but I don't think we need to do a Java API as yet for this layering stuff, or I'd start by just doing scala for now. If a layer definition moves out to a separate jar, the testing for it also needs to be expressed outside as well. The unit tests this is straightforward to to. But there are TDML-based tests in daffodil-test/src/test/resources/org/apache/daffodil/layers/ais.tdml daffodil-test/src/test/scala/org/apache/daffodil/layers/TestAIS.scala The dynamic loading aspect of the user-defined function (UDF) feature is in daffodil-runtime1/src/main/scala/org/apache/daffodil/udf/UserDefinedFunctionService.scala It has TDML tests as well in: daffodil-test/src/test/scala/org/apache/daffodil/udf/TestUdfsInSchemas.scala but the UDFs used to test, are themselves defined in daffodil-udf/src/test/scala/org/sgoodudfs/example/StringFunctions/StringFunctionsProvider.scala Which is a separate module of daffodil, Note these are for testing, and so live under src/test in that module. So they're not delivered as part of Daffodil jars. The primary developer of UDF was Olabusayo Kilo, and the primary developer of the dynamically loadable validations was John Wass (jw3) From: Sandeep Kumar Sent: Thursday, May 20, 2021 7:47 AM To: dev@daffodil.apache.org Subject: DAFFODIL-1927 Hi Team, I'd like to work on the following task : https://issues.apache.org/jira/browse/DAFFODIL-1927 Can you please help with the initial steps for this? Regards, Sandeep
DAFFODIL-1927
Hi Team, I'd like to work on the following task : https://issues.apache.org/jira/browse/DAFFODIL-1927 Can you please help with the initial steps for this? Regards, Sandeep
[jira] [Created] (DAFFODIL-1927) Ability to add new layering transformers on the fly
Michael Beckerle created DAFFODIL-1927: -- Summary: Ability to add new layering transformers on the fly Key: DAFFODIL-1927 URL: https://issues.apache.org/jira/browse/DAFFODIL-1927 Project: Daffodil Issue Type: New Feature Components: Back End Affects Versions: 2.2.0 Reporter: Michael Beckerle Fix For: deferred Pluggable layering transforms, rather than registering them in a singleton object. We might want to use the java Services API for this kindof thing. Would prevent users from having to call the register function--just need to put a special file in the META-INF directory in the jar file. See the file LayerTransformer.scala. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
