Re: broke master 3.2.0-SNAPSHOT branch with latest commit

[Beckerle, Mike]

Note only broken for Java 8 JVM.

From: Beckerle, Mike 
Sent: Thursday, May 20, 2021 3:08 PM
To: dev@daffodil.apache.org 
Subject: broke master 3.2.0-SNAPSHOT branch with latest commit

Will fix shortly.


Mike Beckerle | Principal Engineer

[cid:233aca18-26a6-4e2f-9ce6-557022deda7e]

mbecke...@owlcyberdefense.com

P +1-781-330-0412

broke master 3.2.0-SNAPSHOT branch with latest commit

[Beckerle, Mike]

Will fix shortly.


Mike Beckerle | Principal Engineer

[cid:233aca18-26a6-4e2f-9ce6-557022deda7e]

mbecke...@owlcyberdefense.com

P +1-781-330-0412

Re: DAFFODIL-1927

[Beckerle, Mike]

This is an important change. There is design work to do here for how it should 
work.

There are two features in Daffodil that dynamically load extensions already: 
User-defined functions, and Validators.

So you should be able to look at those to figure out how to dynamically load 
layer implementations from the classpath.

One possible approach is to take one of the existing layering transforms, like 
the one for AIS payload armoring, and make that into a dynamically loaded one.

There are already tests for that, so you would be able to tell easily if your 
dynamically loaded version works the same without having to invent a lot of 
tests.

And AIS is an obscure and special-purpose thing that really should​ be outside 
of daffodil as a loadable layer transform.

Arguably, all of the layer transforms should be in external libraries that live 
in separate jars.

See 
daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/AISTransformer.scala

You'll see that these are "registered" in the file LayerTransformer.scala, and 
that mechanism is what must change to get the transformer from an external 
classpath/jar.

I am happy if the layer transformers have to be written in Scala for now. A 
later enhancement can be to make a polished Java API for these extensions. The 
UDF feature allows writing UDFs in either Java or Scala, but I don't think we 
need to do a Java API as yet for this layering stuff, or I'd start by just 
doing scala for now.

If a layer definition moves out to a separate jar, the testing for it also 
needs to be expressed outside as well.
The unit tests this is straightforward to to. But there are TDML-based tests in

daffodil-test/src/test/resources/org/apache/daffodil/layers/ais.tdml
daffodil-test/src/test/scala/org/apache/daffodil/layers/TestAIS.scala

The dynamic loading aspect of the user-defined function (UDF) feature is in

daffodil-runtime1/src/main/scala/org/apache/daffodil/udf/UserDefinedFunctionService.scala

It has TDML tests as well in:

daffodil-test/src/test/scala/org/apache/daffodil/udf/TestUdfsInSchemas.scala

but the UDFs used to test, are themselves defined in

daffodil-udf/src/test/scala/org/sgoodudfs/example/StringFunctions/StringFunctionsProvider.scala

Which is a separate module of daffodil, Note these are for testing, and so live 
under src/test in that module. So they're not delivered as part of Daffodil 
jars.

The primary developer of UDF was Olabusayo Kilo, and the primary developer of 
the dynamically loadable validations was John Wass (jw3)


From: Sandeep Kumar 
Sent: Thursday, May 20, 2021 7:47 AM
To: dev@daffodil.apache.org 
Subject: DAFFODIL-1927

Hi Team,

I'd like to work on the following task :
https://issues.apache.org/jira/browse/DAFFODIL-1927
Can you please help with the initial steps for this?

Regards,
Sandeep

DAFFODIL-1927

[Sandeep Kumar]

Hi Team,

I'd like to work on the following task :
https://issues.apache.org/jira/browse/DAFFODIL-1927
Can you please help with the initial steps for this?

Regards,
Sandeep