Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Latest push to the PR adds true ServiceLoader feature to dynamically discover and load the layer transformers. This should now work with CLI, vscode, etc. There is still no Java API for writing a layer transformer in Java. I don't know that this is actually a requirement. On Wed, Oct 6, 2021 at 5:40 PM Larry Barber wrote: > It seems like it would need to work with the CLI, debugger, as well as the > NiFi version > > -Original Message- > From: Mike Beckerle > Sent: Wednesday, October 6, 2021 4:38 PM > To: dev@daffodil.apache.org > Subject: Re: Please review PR for - Re: Design discussion - dynamically > loadable layers - DAFFODIL-1927 > > Good point. Won't work from the new debugger. I guess I have to figure out > the service loader stuff. > > On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence > wrote: > > > Ah, right, I didn't think about registering it from within the test > > suite object. > > > > Though, that isn't possible for the CLI, and I'd guess it also isn't > > possible from the new VSCode debugger. > > > > > > On 10/6/21 4:09 PM, Mike Beckerle wrote: > > > Much easier than that. > > > > > > Look at TestAIS.scala: > > > > > > > > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgit > > hub.com%2Fapache%2Fdaffodil%2Fpull%2F651%2Ffiles%23diff-59f6bd4163627a > > a9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91&data=04%7C01%7C > > larry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c2 > > 14c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CT > > WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI > > 6Mn0%3D%7C3000&sdata=BRpeY%2FZZaW6n3hyZ8HJec%2BFv%2B1nFN5mBTgy6aHS > > gLpc%3D&reserved=0 > > > > > > You just call LayerCompilerRegistry.Register( > > > AISPayloadArmoringLayerCompiler) > > > From the test object. > > > > > > That's all an application using Daffodil with a custom layer has to > > > do also. > > > > > > Honestly, I almost prefer this to dynamic loading. > > > > > > > > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence > > wrote: > > > > > >> Don't those only work because they are built in to Daffodil and > > >> pre-registered? If someone wants to keep their layer outside of > > >> Daffodil, it needs to be manually registered, but that can't be > > >> done by the TDML Runner or CLI? > > >> > > >> On 10/6/21 3:38 PM, Mike Beckerle wrote: > > >>> The TDML runner invoked from JUnit works fine. > > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 > > >>> examples in daffodil-test work this way, and anyone's DFDL schema > > >>> could do the same thing. > > >>> > > >>> I plan to update the ethernetIP and PCAP schemas to compute their > > >> checksums > > >>> this way. > > >>> > > >>> Only the CLI is left out. > > >>> > > >>> > > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > > >>> > > >> wrote: > > >>> > > >>>> If we don't support dynamic loading and instead require that > > >>>> users manually register the layer, that means that externally > > >>>> defined layers cannot be tested using the TDML runner, right? It > > >>>> also means there's > > no > > >>>> way to run schemas with external layers using the CLI? > > >>>> > > >>>> I guess that's not an issue until there are external layers, so > > >>>> maybe not critical, but still feels pretty important. Maybe > > >>>> adding it to the next release is reasonable, since that's > > >>>> probably when external layers might start showing up? > > >>>> > > >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote: > > >>>>> I have done a bunch of improvements to the code for this layer > > loading > > >>>>> based on the prior PR. > > >>>>> > > >>>>> It is now stabilized and worth it to examine/reexamine the PR at > > >>>>> this > > >>>> point. > > >>>>> > > >>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F > > >>>>> %2Fgithub.com%2Fapache%2Fdaffodi
RE: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
It seems like it would need to work with the CLI, debugger, as well as the NiFi version -Original Message- From: Mike Beckerle Sent: Wednesday, October 6, 2021 4:38 PM To: dev@daffodil.apache.org Subject: Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927 Good point. Won't work from the new debugger. I guess I have to figure out the service loader stuff. On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence wrote: > Ah, right, I didn't think about registering it from within the test > suite object. > > Though, that isn't possible for the CLI, and I'd guess it also isn't > possible from the new VSCode debugger. > > > On 10/6/21 4:09 PM, Mike Beckerle wrote: > > Much easier than that. > > > > Look at TestAIS.scala: > > > > > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgit > hub.com%2Fapache%2Fdaffodil%2Fpull%2F651%2Ffiles%23diff-59f6bd4163627a > a9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91&data=04%7C01%7C > larry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc%7C379c2 > 14c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839%7CUnknown%7CT > WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI > 6Mn0%3D%7C3000&sdata=BRpeY%2FZZaW6n3hyZ8HJec%2BFv%2B1nFN5mBTgy6aHS > gLpc%3D&reserved=0 > > > > You just call LayerCompilerRegistry.Register( > > AISPayloadArmoringLayerCompiler) > > From the test object. > > > > That's all an application using Daffodil with a custom layer has to > > do also. > > > > Honestly, I almost prefer this to dynamic loading. > > > > > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence > wrote: > > > >> Don't those only work because they are built in to Daffodil and > >> pre-registered? If someone wants to keep their layer outside of > >> Daffodil, it needs to be manually registered, but that can't be > >> done by the TDML Runner or CLI? > >> > >> On 10/6/21 3:38 PM, Mike Beckerle wrote: > >>> The TDML runner invoked from JUnit works fine. > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 > >>> examples in daffodil-test work this way, and anyone's DFDL schema > >>> could do the same thing. > >>> > >>> I plan to update the ethernetIP and PCAP schemas to compute their > >> checksums > >>> this way. > >>> > >>> Only the CLI is left out. > >>> > >>> > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > >>> > >> wrote: > >>> > >>>> If we don't support dynamic loading and instead require that > >>>> users manually register the layer, that means that externally > >>>> defined layers cannot be tested using the TDML runner, right? It > >>>> also means there's > no > >>>> way to run schemas with external layers using the CLI? > >>>> > >>>> I guess that's not an issue until there are external layers, so > >>>> maybe not critical, but still feels pretty important. Maybe > >>>> adding it to the next release is reasonable, since that's > >>>> probably when external layers might start showing up? > >>>> > >>>> On 10/6/21 1:47 PM, Mike Beckerle wrote: > >>>>> I have done a bunch of improvements to the code for this layer > loading > >>>>> based on the prior PR. > >>>>> > >>>>> It is now stabilized and worth it to examine/reexamine the PR at > >>>>> this > >>>> point. > >>>>> > >>>>> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F > >>>>> %2Fgithub.com%2Fapache%2Fdaffodil%2Fpull%2F651&data=04%7C01% > >>>>> 7Clarry.barber%40nteligen.com%7Caa7e020cc7ce47f2f62d08d9890929fc > >>>>> %7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C637691496871884839 > >>>>> %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ > >>>>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7YEJPwd2M%2BaGjrgtu > >>>>> 0z3bQL%2BN%2Fi1fCSFnu4PMeKH9MY%3D&reserved=0 > >>>>> > >>>>> I have also changed the goals here somewhat. I split out the > >>>>> defining/creation of a Java API for layers, and true dynamic > >>>>> loading > >> via > >>>>> service loaders into a separate J
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Good point. Won't work from the new debugger. I guess I have to figure out the service loader stuff. On Wed, Oct 6, 2021 at 4:25 PM Steve Lawrence wrote: > Ah, right, I didn't think about registering it from within the test > suite object. > > Though, that isn't possible for the CLI, and I'd guess it also isn't > possible from the new VSCode debugger. > > > On 10/6/21 4:09 PM, Mike Beckerle wrote: > > Much easier than that. > > > > Look at TestAIS.scala: > > > > > https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91 > > > > You just call LayerCompilerRegistry.Register( > > AISPayloadArmoringLayerCompiler) > > From the test object. > > > > That's all an application using Daffodil with a custom layer has to do > > also. > > > > Honestly, I almost prefer this to dynamic loading. > > > > > > On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence > wrote: > > > >> Don't those only work because they are built in to Daffodil and > >> pre-registered? If someone wants to keep their layer outside of > >> Daffodil, it needs to be manually registered, but that can't be done by > >> the TDML Runner or CLI? > >> > >> On 10/6/21 3:38 PM, Mike Beckerle wrote: > >>> The TDML runner invoked from JUnit works fine. > >>> The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in > >>> daffodil-test work this way, and anyone's DFDL schema could do the same > >>> thing. > >>> > >>> I plan to update the ethernetIP and PCAP schemas to compute their > >> checksums > >>> this way. > >>> > >>> Only the CLI is left out. > >>> > >>> > >>> On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > >> wrote: > >>> > If we don't support dynamic loading and instead require that users > manually register the layer, that means that externally defined layers > cannot be tested using the TDML runner, right? It also means there's > no > way to run schemas with external layers using the CLI? > > I guess that's not an issue until there are external layers, so maybe > not critical, but still feels pretty important. Maybe adding it to the > next release is reasonable, since that's probably when external layers > might start showing up? > > On 10/6/21 1:47 PM, Mike Beckerle wrote: > > I have done a bunch of improvements to the code for this layer > loading > > based on the prior PR. > > > > It is now stabilized and worth it to examine/reexamine the PR at this > point. > > > > https://github.com/apache/daffodil/pull/651 > > > > I have also changed the goals here somewhat. I split out the > > defining/creation of a Java API for layers, and true dynamic loading > >> via > > service loaders into a separate JIRA ticket DAFFODIL-2567. > > > > I think that experience with what we have so far makes sense before > > defining any Java API for doing this. I also think the need for true > > dynamic detect and load behavior is far less critical than the basic > > ability to define these in external jars that aren't part of > daffodil. > > > > This PR now includes just the modularization of layers, so they can > be > > written in scala and defined in a jar outside of Daffodil, An > >> application > > must register them for use by calling > LayerCompilerRegistry.register(...). > > > > All the built-in layers are in daffodil-runtime1-layers which is a > new > > module. > > > > Other layers are defined in daffodil-test for testing that > checksums/check > > digits work. The example AIS payload encoding layer is also now > defined > and > > tested in daffodil-test. > > > > On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle > wrote: > > > >> Please see: > >> > > >> > https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations > >> > >> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 > >> > >> See related PR: https://github.com/apache/daffodil/pull/643 which > is > for > >> https://issues.apache.org/jira/browse/DAFFODIL-2221 > >> > >> Please add comments or directly edit that wiki page, or discuss in > >> this > >> email thread if you prefer. > >> > >> > > > > > >>> > >> > >> > > > >
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Ah, right, I didn't think about registering it from within the test suite object. Though, that isn't possible for the CLI, and I'd guess it also isn't possible from the new VSCode debugger. On 10/6/21 4:09 PM, Mike Beckerle wrote: Much easier than that. Look at TestAIS.scala: https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91 You just call LayerCompilerRegistry.Register( AISPayloadArmoringLayerCompiler) From the test object. That's all an application using Daffodil with a custom layer has to do also. Honestly, I almost prefer this to dynamic loading. On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence wrote: Don't those only work because they are built in to Daffodil and pre-registered? If someone wants to keep their layer outside of Daffodil, it needs to be manually registered, but that can't be done by the TDML Runner or CLI? On 10/6/21 3:38 PM, Mike Beckerle wrote: The TDML runner invoked from JUnit works fine. The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in daffodil-test work this way, and anyone's DFDL schema could do the same thing. I plan to update the ethernetIP and PCAP schemas to compute their checksums this way. Only the CLI is left out. On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence wrote: If we don't support dynamic loading and instead require that users manually register the layer, that means that externally defined layers cannot be tested using the TDML runner, right? It also means there's no way to run schemas with external layers using the CLI? I guess that's not an issue until there are external layers, so maybe not critical, but still feels pretty important. Maybe adding it to the next release is reasonable, since that's probably when external layers might start showing up? On 10/6/21 1:47 PM, Mike Beckerle wrote: I have done a bunch of improvements to the code for this layer loading based on the prior PR. It is now stabilized and worth it to examine/reexamine the PR at this point. https://github.com/apache/daffodil/pull/651 I have also changed the goals here somewhat. I split out the defining/creation of a Java API for layers, and true dynamic loading via service loaders into a separate JIRA ticket DAFFODIL-2567. I think that experience with what we have so far makes sense before defining any Java API for doing this. I also think the need for true dynamic detect and load behavior is far less critical than the basic ability to define these in external jars that aren't part of daffodil. This PR now includes just the modularization of layers, so they can be written in scala and defined in a jar outside of Daffodil, An application must register them for use by calling LayerCompilerRegistry.register(...). All the built-in layers are in daffodil-runtime1-layers which is a new module. Other layers are defined in daffodil-test for testing that checksums/check digits work. The example AIS payload encoding layer is also now defined and tested in daffodil-test. On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote: Please see: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 See related PR: https://github.com/apache/daffodil/pull/643 which is for https://issues.apache.org/jira/browse/DAFFODIL-2221 Please add comments or directly edit that wiki page, or discuss in this email thread if you prefer.
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Much easier than that. Look at TestAIS.scala: https://github.com/apache/daffodil/pull/651/files#diff-59f6bd4163627aa9de0ed8b2ff04ba8c60c7c45fc99e6abed430d0a6a46c4c91 You just call LayerCompilerRegistry.Register( AISPayloadArmoringLayerCompiler) >From the test object. That's all an application using Daffodil with a custom layer has to do also. Honestly, I almost prefer this to dynamic loading. On Wed, Oct 6, 2021 at 3:50 PM Steve Lawrence wrote: > Don't those only work because they are built in to Daffodil and > pre-registered? If someone wants to keep their layer outside of > Daffodil, it needs to be manually registered, but that can't be done by > the TDML Runner or CLI? > > On 10/6/21 3:38 PM, Mike Beckerle wrote: > > The TDML runner invoked from JUnit works fine. > > The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in > > daffodil-test work this way, and anyone's DFDL schema could do the same > > thing. > > > > I plan to update the ethernetIP and PCAP schemas to compute their > checksums > > this way. > > > > Only the CLI is left out. > > > > > > On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence > wrote: > > > >> If we don't support dynamic loading and instead require that users > >> manually register the layer, that means that externally defined layers > >> cannot be tested using the TDML runner, right? It also means there's no > >> way to run schemas with external layers using the CLI? > >> > >> I guess that's not an issue until there are external layers, so maybe > >> not critical, but still feels pretty important. Maybe adding it to the > >> next release is reasonable, since that's probably when external layers > >> might start showing up? > >> > >> On 10/6/21 1:47 PM, Mike Beckerle wrote: > >>> I have done a bunch of improvements to the code for this layer loading > >>> based on the prior PR. > >>> > >>> It is now stabilized and worth it to examine/reexamine the PR at this > >> point. > >>> > >>> https://github.com/apache/daffodil/pull/651 > >>> > >>> I have also changed the goals here somewhat. I split out the > >>> defining/creation of a Java API for layers, and true dynamic loading > via > >>> service loaders into a separate JIRA ticket DAFFODIL-2567. > >>> > >>> I think that experience with what we have so far makes sense before > >>> defining any Java API for doing this. I also think the need for true > >>> dynamic detect and load behavior is far less critical than the basic > >>> ability to define these in external jars that aren't part of daffodil. > >>> > >>> This PR now includes just the modularization of layers, so they can be > >>> written in scala and defined in a jar outside of Daffodil, An > application > >>> must register them for use by calling > >> LayerCompilerRegistry.register(...). > >>> > >>> All the built-in layers are in daffodil-runtime1-layers which is a new > >>> module. > >>> > >>> Other layers are defined in daffodil-test for testing that > >> checksums/check > >>> digits work. The example AIS payload encoding layer is also now defined > >> and > >>> tested in daffodil-test. > >>> > >>> On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle > >> wrote: > >>> > Please see: > > >> > https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations > > Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 > > See related PR: https://github.com/apache/daffodil/pull/643 which is > >> for > https://issues.apache.org/jira/browse/DAFFODIL-2221 > > Please add comments or directly edit that wiki page, or discuss in > this > email thread if you prefer. > > > >>> > >> > >> > > > >
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
Don't those only work because they are built in to Daffodil and pre-registered? If someone wants to keep their layer outside of Daffodil, it needs to be manually registered, but that can't be done by the TDML Runner or CLI? On 10/6/21 3:38 PM, Mike Beckerle wrote: The TDML runner invoked from JUnit works fine. The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in daffodil-test work this way, and anyone's DFDL schema could do the same thing. I plan to update the ethernetIP and PCAP schemas to compute their checksums this way. Only the CLI is left out. On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence wrote: If we don't support dynamic loading and instead require that users manually register the layer, that means that externally defined layers cannot be tested using the TDML runner, right? It also means there's no way to run schemas with external layers using the CLI? I guess that's not an issue until there are external layers, so maybe not critical, but still feels pretty important. Maybe adding it to the next release is reasonable, since that's probably when external layers might start showing up? On 10/6/21 1:47 PM, Mike Beckerle wrote: I have done a bunch of improvements to the code for this layer loading based on the prior PR. It is now stabilized and worth it to examine/reexamine the PR at this point. https://github.com/apache/daffodil/pull/651 I have also changed the goals here somewhat. I split out the defining/creation of a Java API for layers, and true dynamic loading via service loaders into a separate JIRA ticket DAFFODIL-2567. I think that experience with what we have so far makes sense before defining any Java API for doing this. I also think the need for true dynamic detect and load behavior is far less critical than the basic ability to define these in external jars that aren't part of daffodil. This PR now includes just the modularization of layers, so they can be written in scala and defined in a jar outside of Daffodil, An application must register them for use by calling LayerCompilerRegistry.register(...). All the built-in layers are in daffodil-runtime1-layers which is a new module. Other layers are defined in daffodil-test for testing that checksums/check digits work. The example AIS payload encoding layer is also now defined and tested in daffodil-test. On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote: Please see: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 See related PR: https://github.com/apache/daffodil/pull/643 which is for https://issues.apache.org/jira/browse/DAFFODIL-2221 Please add comments or directly edit that wiki page, or discuss in this email thread if you prefer.
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
The TDML runner invoked from JUnit works fine. The AIS Payload Armoring, and the CheckDigits and the IPv4 examples in daffodil-test work this way, and anyone's DFDL schema could do the same thing. I plan to update the ethernetIP and PCAP schemas to compute their checksums this way. Only the CLI is left out. On Wed, Oct 6, 2021 at 2:26 PM Steve Lawrence wrote: > If we don't support dynamic loading and instead require that users > manually register the layer, that means that externally defined layers > cannot be tested using the TDML runner, right? It also means there's no > way to run schemas with external layers using the CLI? > > I guess that's not an issue until there are external layers, so maybe > not critical, but still feels pretty important. Maybe adding it to the > next release is reasonable, since that's probably when external layers > might start showing up? > > On 10/6/21 1:47 PM, Mike Beckerle wrote: > > I have done a bunch of improvements to the code for this layer loading > > based on the prior PR. > > > > It is now stabilized and worth it to examine/reexamine the PR at this > point. > > > > https://github.com/apache/daffodil/pull/651 > > > > I have also changed the goals here somewhat. I split out the > > defining/creation of a Java API for layers, and true dynamic loading via > > service loaders into a separate JIRA ticket DAFFODIL-2567. > > > > I think that experience with what we have so far makes sense before > > defining any Java API for doing this. I also think the need for true > > dynamic detect and load behavior is far less critical than the basic > > ability to define these in external jars that aren't part of daffodil. > > > > This PR now includes just the modularization of layers, so they can be > > written in scala and defined in a jar outside of Daffodil, An application > > must register them for use by calling > LayerCompilerRegistry.register(...). > > > > All the built-in layers are in daffodil-runtime1-layers which is a new > > module. > > > > Other layers are defined in daffodil-test for testing that > checksums/check > > digits work. The example AIS payload encoding layer is also now defined > and > > tested in daffodil-test. > > > > On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle > wrote: > > > >> Please see: > >> > https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations > >> > >> Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 > >> > >> See related PR: https://github.com/apache/daffodil/pull/643 which is > for > >> https://issues.apache.org/jira/browse/DAFFODIL-2221 > >> > >> Please add comments or directly edit that wiki page, or discuss in this > >> email thread if you prefer. > >> > >> > > > >
Re: Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
If we don't support dynamic loading and instead require that users manually register the layer, that means that externally defined layers cannot be tested using the TDML runner, right? It also means there's no way to run schemas with external layers using the CLI? I guess that's not an issue until there are external layers, so maybe not critical, but still feels pretty important. Maybe adding it to the next release is reasonable, since that's probably when external layers might start showing up? On 10/6/21 1:47 PM, Mike Beckerle wrote: I have done a bunch of improvements to the code for this layer loading based on the prior PR. It is now stabilized and worth it to examine/reexamine the PR at this point. https://github.com/apache/daffodil/pull/651 I have also changed the goals here somewhat. I split out the defining/creation of a Java API for layers, and true dynamic loading via service loaders into a separate JIRA ticket DAFFODIL-2567. I think that experience with what we have so far makes sense before defining any Java API for doing this. I also think the need for true dynamic detect and load behavior is far less critical than the basic ability to define these in external jars that aren't part of daffodil. This PR now includes just the modularization of layers, so they can be written in scala and defined in a jar outside of Daffodil, An application must register them for use by calling LayerCompilerRegistry.register(...). All the built-in layers are in daffodil-runtime1-layers which is a new module. Other layers are defined in daffodil-test for testing that checksums/check digits work. The example AIS payload encoding layer is also now defined and tested in daffodil-test. On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote: Please see: https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 See related PR: https://github.com/apache/daffodil/pull/643 which is for https://issues.apache.org/jira/browse/DAFFODIL-2221 Please add comments or directly edit that wiki page, or discuss in this email thread if you prefer.
Please review PR for - Re: Design discussion - dynamically loadable layers - DAFFODIL-1927
I have done a bunch of improvements to the code for this layer loading based on the prior PR. It is now stabilized and worth it to examine/reexamine the PR at this point. https://github.com/apache/daffodil/pull/651 I have also changed the goals here somewhat. I split out the defining/creation of a Java API for layers, and true dynamic loading via service loaders into a separate JIRA ticket DAFFODIL-2567. I think that experience with what we have so far makes sense before defining any Java API for doing this. I also think the need for true dynamic detect and load behavior is far less critical than the basic ability to define these in external jars that aren't part of daffodil. This PR now includes just the modularization of layers, so they can be written in scala and defined in a jar outside of Daffodil, An application must register them for use by calling LayerCompilerRegistry.register(...). All the built-in layers are in daffodil-runtime1-layers which is a new module. Other layers are defined in daffodil-test for testing that checksums/check digits work. The example AIS payload encoding layer is also now defined and tested in daffodil-test. On Sat, Sep 25, 2021 at 8:31 PM Mike Beckerle wrote: > Please see: > https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Dynamically+loading+Layer+Transformations > > Also: https://issues.apache.org/jira/browse/DAFFODIL-1927 > > See related PR: https://github.com/apache/daffodil/pull/643 which is for > https://issues.apache.org/jira/browse/DAFFODIL-2221 > > Please add comments or directly edit that wiki page, or discuss in this > email thread if you prefer. > >
