> All that said, I think what we really need is a way to move about from 
> requiring
> DAFFODIL_CLASSPATH at all. For example, if we could embed dependencies in the
> actual saved parser file, then using that saved parser wouldn't need any
> classpath modifications, it's just all already there and Daffodil internals
> would uses those embedded dependencies for class lookups. I'm unsure of 
> exactly
> how to do that, but it's definitely possible--NiFi does something very similar
> with it's "nar" format

Changing the saved parser format to include classpath JAR's sounds like 
something perfect for a 4.0.0 release of Daffodil.

Josh
________________________________
From: Steve Lawrence <slawre...@apache.org>
Sent: Thursday, October 24, 2024 9:48 AM
To: dev@daffodil.apache.org <dev@daffodil.apache.org>
Subject: Re: How to get DAFFODIL_CLASSPATH for a complex DFDL schema?

I've also just opened a PR to daffodil-sbt to fix the bug that causes "show
fullClasspath" and "export fullClasspath" to include daffodil dependencies:

https://github.com/apache/daffodil-sbt/pull/64


On 2024-10-24 08:24 AM, Steve Lawrence wrote:
> Instead of `sbt dependencyTree`, you can run `sbt "show fullClasspath"` to
> output all the dependencies that `packageDaffodilBin` uses. You can also run
> `sbt "export fullClasspath"` to get an actual classpath string that you can 
> drop
> into DAFFODIL_CLASSPATH. In one line, I think you could do:
>
> export DAFFODIL_CLASSPATH=$(sbt -batch -error "export fullClasspath")
>
> Note that the -batch and -error are needed to disable [info] and other output
> messages.
>
> Also note that this includes the scala dependency and I think we might have a
> bug in daffodil-sbt that causes it to also include Daffodil dependencies if 
> any
> schemas are layers/charsets/etc. I *think* the way the Daffodil CLI builds up
> the classpath those extra dependencies will all be ignored, but if not you 
> might
> have to manually build up the classpath with just the paths you want.
>
> If we want we could add a special sbt task that essentially mimics this
> behavior, but I'd rather we just document this magic export command somewhere 
> so
> we don't have to maintain it.
>
>
> The Daffodil synonm is an interesting idea. I guess it would just set fork the
> daffodil script with DAFFODIL_CLASSPATH already set, and just pass any task
> arguments to the script? I think that's possible in SBT, but I think the 
> export
> magic above is a bit more flexible and efficient since you don't need to keep
> sbt running to run a Daffodil command.
>
>
> All that said, I think what we really need is a way to move about from 
> requiring
> DAFFODIL_CLASSPATH at all. For example, if we could embed dependencies in the
> actual saved parser file, then using that saved parser wouldn't need any
> classpath modifications, it's just all already there and Daffodil internals
> would uses those embedded dependencies for class lookups. I'm unsure of 
> exactly
> how to do that, but it's definitely possible--NiFi does something very similar
> with it's "nar" format.
>
>
> On 2024-10-23 06:00 PM, Mike Beckerle wrote:
>> I am trying to go from 'sbt test' to a schema I can play with from the
>> daffodil CLI.
>>
>> The schema of interest is the DFDLSchemas envelope-payload example.
>>
>> This schema depends on
>> * tcpMessage
>> * mil-std-2045
>> * pcap
>>
>> The pcap schema in turn depends on
>> * ethernetIP
>>
>> The ethernetIP schema defines a Daffodil layer plugin that exists in its
>> jar.
>>
>> So far if I clone these all, and 'sbt publishLocal' all of the components,
>> then I can 'sbt test' in envelope-payload and it passes all tests.
>>
>> So now I'd like to do 'daffodil save-parser
>> -s src/main/resources/io/github/dfdlschemas/envelopepayload/xsd/
>> envelopePayload.dfdl.xsd
>> -o /tmp/envPay.bin'
>>
>> By adding these to the build.sbt
>>
>> daffodilPackageBinVersions := Seq(daffodilVersion.value)
>> daffodilPackageBinInfos := Seq(
>>    ("/io/github/dfdlschemas/tcpMessage/xsd/tcpMessage.dfdl.xsd",
>> Some("message"), None)
>> )
>>
>> Then 'sbt packageDaffodilBin' will create a compiled schema under the
>> target/ directory named:
>>
>> dfdl-envelope-payload-1.1.0-daffodil390.bin
>>
>> So far so good.
>>
>> Now the challenge.
>>
>> But this can't be used with the daffodil CLI without also setting up
>> DAFFODIL_CLASSPATH to have at least the ethernetIP jar file.
>>
>> Which is where? (yes I know in the ~/.ivy2/local cache, but there is tons
>> of stuff in there.)
>>
>> If I want to use xerces (aka full) validation then I also have to have all
>> the other component schema jar files on the DAFFODIL_CLASSPATH as well.
>>
>> I tried issuing 'sbt dependencyTree', but the dependency tree is not just
>> the schemas, but all the dependencies on daffodil and everything
>> transitively it uses.
>>
>> This is much too hard.
>>
>> Why not have the daffodil-sbt plugin output a shell script that appends to
>> DAFFODIL_CLASSPATH with all the necessary component schema jars.
>>
>> Then users could just run that script to establish the DAFFODIL_CLASSPATH
>> once, and then they could use the daffodil CLI normally.
>>
>> In principle, a file could also be written intended to inform the daffodil
>> VSCode extension of the classpath to construct for the schema.
>>
>> Thoughts?
>>
>> One possible alternative is to add a daffodil command to sbt via the plugin
>> so that one can run daffodil command lines directly from the sbt prompt:
>>
>> sbt
>>> daffodil parse -p target/...bin foo.dat
>>
>> This is really just a synonym for issuing sbt run for the Main class of the
>> daffodil-cli module, (after doing packageDaffodilBin) so might be very easy
>> to do.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Mike Beckerle
>> Apache Daffodil PMC | daffodil.apache.org
>> OGF DFDL Workgroup Co-Chair | 
>> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl<http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl>
>> Owl Cyber Defense | www.owlcyberdefense.com<http://www.owlcyberdefense.com>
>>
>

Reply via email to