> All that said, I think what we really need is a way to move about from > requiring > DAFFODIL_CLASSPATH at all. For example, if we could embed dependencies in the > actual saved parser file, then using that saved parser wouldn't need any > classpath modifications, it's just all already there and Daffodil internals > would uses those embedded dependencies for class lookups. I'm unsure of > exactly > how to do that, but it's definitely possible--NiFi does something very similar > with it's "nar" format
Changing the saved parser format to include classpath JAR's sounds like something perfect for a 4.0.0 release of Daffodil. Josh ________________________________ From: Steve Lawrence <slawre...@apache.org> Sent: Thursday, October 24, 2024 9:48 AM To: dev@daffodil.apache.org <dev@daffodil.apache.org> Subject: Re: How to get DAFFODIL_CLASSPATH for a complex DFDL schema? I've also just opened a PR to daffodil-sbt to fix the bug that causes "show fullClasspath" and "export fullClasspath" to include daffodil dependencies: https://github.com/apache/daffodil-sbt/pull/64 On 2024-10-24 08:24 AM, Steve Lawrence wrote: > Instead of `sbt dependencyTree`, you can run `sbt "show fullClasspath"` to > output all the dependencies that `packageDaffodilBin` uses. You can also run > `sbt "export fullClasspath"` to get an actual classpath string that you can > drop > into DAFFODIL_CLASSPATH. In one line, I think you could do: > > export DAFFODIL_CLASSPATH=$(sbt -batch -error "export fullClasspath") > > Note that the -batch and -error are needed to disable [info] and other output > messages. > > Also note that this includes the scala dependency and I think we might have a > bug in daffodil-sbt that causes it to also include Daffodil dependencies if > any > schemas are layers/charsets/etc. I *think* the way the Daffodil CLI builds up > the classpath those extra dependencies will all be ignored, but if not you > might > have to manually build up the classpath with just the paths you want. > > If we want we could add a special sbt task that essentially mimics this > behavior, but I'd rather we just document this magic export command somewhere > so > we don't have to maintain it. > > > The Daffodil synonm is an interesting idea. I guess it would just set fork the > daffodil script with DAFFODIL_CLASSPATH already set, and just pass any task > arguments to the script? I think that's possible in SBT, but I think the > export > magic above is a bit more flexible and efficient since you don't need to keep > sbt running to run a Daffodil command. > > > All that said, I think what we really need is a way to move about from > requiring > DAFFODIL_CLASSPATH at all. For example, if we could embed dependencies in the > actual saved parser file, then using that saved parser wouldn't need any > classpath modifications, it's just all already there and Daffodil internals > would uses those embedded dependencies for class lookups. I'm unsure of > exactly > how to do that, but it's definitely possible--NiFi does something very similar > with it's "nar" format. > > > On 2024-10-23 06:00 PM, Mike Beckerle wrote: >> I am trying to go from 'sbt test' to a schema I can play with from the >> daffodil CLI. >> >> The schema of interest is the DFDLSchemas envelope-payload example. >> >> This schema depends on >> * tcpMessage >> * mil-std-2045 >> * pcap >> >> The pcap schema in turn depends on >> * ethernetIP >> >> The ethernetIP schema defines a Daffodil layer plugin that exists in its >> jar. >> >> So far if I clone these all, and 'sbt publishLocal' all of the components, >> then I can 'sbt test' in envelope-payload and it passes all tests. >> >> So now I'd like to do 'daffodil save-parser >> -s src/main/resources/io/github/dfdlschemas/envelopepayload/xsd/ >> envelopePayload.dfdl.xsd >> -o /tmp/envPay.bin' >> >> By adding these to the build.sbt >> >> daffodilPackageBinVersions := Seq(daffodilVersion.value) >> daffodilPackageBinInfos := Seq( >> ("/io/github/dfdlschemas/tcpMessage/xsd/tcpMessage.dfdl.xsd", >> Some("message"), None) >> ) >> >> Then 'sbt packageDaffodilBin' will create a compiled schema under the >> target/ directory named: >> >> dfdl-envelope-payload-1.1.0-daffodil390.bin >> >> So far so good. >> >> Now the challenge. >> >> But this can't be used with the daffodil CLI without also setting up >> DAFFODIL_CLASSPATH to have at least the ethernetIP jar file. >> >> Which is where? (yes I know in the ~/.ivy2/local cache, but there is tons >> of stuff in there.) >> >> If I want to use xerces (aka full) validation then I also have to have all >> the other component schema jar files on the DAFFODIL_CLASSPATH as well. >> >> I tried issuing 'sbt dependencyTree', but the dependency tree is not just >> the schemas, but all the dependencies on daffodil and everything >> transitively it uses. >> >> This is much too hard. >> >> Why not have the daffodil-sbt plugin output a shell script that appends to >> DAFFODIL_CLASSPATH with all the necessary component schema jars. >> >> Then users could just run that script to establish the DAFFODIL_CLASSPATH >> once, and then they could use the daffodil CLI normally. >> >> In principle, a file could also be written intended to inform the daffodil >> VSCode extension of the classpath to construct for the schema. >> >> Thoughts? >> >> One possible alternative is to add a daffodil command to sbt via the plugin >> so that one can run daffodil command lines directly from the sbt prompt: >> >> sbt >>> daffodil parse -p target/...bin foo.dat >> >> This is really just a synonym for issuing sbt run for the Main class of the >> daffodil-cli module, (after doing packageDaffodilBin) so might be very easy >> to do. >> >> >> >> >> >> >> >> >> >> >> Mike Beckerle >> Apache Daffodil PMC | daffodil.apache.org >> OGF DFDL Workgroup Co-Chair | >> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl<http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl> >> Owl Cyber Defense | www.owlcyberdefense.com<http://www.owlcyberdefense.com> >> >
