Fat jars would need to implement Java-Modules/OGSI-style isolation, otherwise a depencency jar contained in the fat jar could conflict with some other dependency.
Do fat jars do that, or are they just a means of packaging these things all together so you don't have to chase them down and add to the classpath individually? Alternatively both fat and regular (non-fat) jars would be needed so you can use the fat jar, but if you get conflicts, go back to the non-fat and doing things the hard way. I definitely agree I don't want to add maven/ivy support to the CLI. But in some sense the CLI's "parse --schema schemaFilePath" option is not particularly helpful. As currently formulated it suggests that's all that is needed to specify the schema to use. The reality is that when that schema has dependencies, much more is needed. We're finding that most "real" schemas have these dependencies. Perhaps we start by just saying that plug-ins (of any kind, layers, UDFs) should document that they have dependencies, and how to use sbt to list them (in case the doc gets out of date), and how to add them to the classpath via DAFFODIL_CLASSPATH. On Wed, Jan 11, 2023 at 9:53 AM Steve Lawrence <[email protected]> wrote: > Ultimately all jars need to end up on the classpath somehow. > > Maybe one option is to add maven/ivy support to the Daffodil CLI and add > way to specify a dependency, e.g > > daffodil parse --dependency com.example.transforms/[email protected] ... > > Then Daffodil could download any dependencies (excluding Daffodil deps > that it already has) and update the classpath, maybe with a custom > classloader or maybe reexecing itself with a new claspath or something. > > > The feels a bit extreme though--I'm not sure I want the CLI to have > maven/ivy support. > > I think maybe a more reasonable option is to document how to create a > fat jar for these kinds of things (with a way to exclude the Daffodil > dependencies), and then users only need to add a single jar to their > classpath. It still requires a little user intervention but it's not > terrible. > > And it looks like the sbt-assembly plugin can already create a fat jar > and exclude dependencies using the "provided" keyword which just a > couple of sbt lines: > > https://www.baeldung.com/scala/sbt > > This is nice too because this will include schemas, transformers, and > dependencies (both Java and schema dependencies) all in a single jar, so > it's a all in one bundle that can easily be put on the class path that > contains everything you could need. > > We've also discussed a Daffodil/DFDL plugin in the past to simplify > schema development. Having that plugin also configure these fat jars for > releases might make a lot of sense. This way, all you would do is add > the DFDL plugin to your project, define your dependencies the normal > way, put your schemas and transforms in the right place, and run "sbt > assembly", and you'll get a fat jar to put on the classpath. > > > On 2023-01-10 05:52 PM, Mike Beckerle wrote: > > So when I use daffodil by way of sbt, all the managed dependencies of the > > zip-layer-transformer (really just on apache commons compress) are taken > > care of for me. > > > > My main schema just declares a managed dependency on a specific version > of > > the zip-layer-transformer, and voila, the transitive dependency on the > > apache commons compress takes care of itself. > > > > When I use the CLI, I have to do this stuff manually. It's amazingly hard > > to get DAFFODIL_CLASSPATH complete and correct. > > > > 'sbt show dependencyClasspath' shows me the dependency on apache commons > > compress library, of course it also has all the daffodil jars and all > their > > dependencies. > > > > How do we make the CLI or any environment for running Daffodil, as easy > as > > sbt? > > > > > > On Tue, Jan 10, 2023 at 3:36 PM Steve Lawrence <[email protected]> > wrote: > > > >> I think so. Daffodil can't know what libraries a layer transformer might > >> need, so we can't depend on them or include them in releases. > >> > >> So if a transformer plugin requires one or more libraries that don't > >> already ship with Daffodil, then they should be marked as a dependency > >> of the transformer and their jars downloaded and included in the > >> classpath along with the transformer. > >> > >> You could maybe get around his by making your transformer a fat jar, but > >> it might include all the Daffodil jars as well, which might cause some > >> problems. > >> > >> > >> On 2023-01-10 03:24 PM, Mike Beckerle wrote: > >>> So I wrote a layer transformer that unzips zip files in data. > >>> > >>> It depends on apache commons compress library. > >>> > >>> Daffodil recognizes that a schema requires this, and dynamically loads > >> it. > >>> > >>> But then things fail because apache commons compress library is not > >> found. > >>> > >>> We have to manually add apache commons compress jar to the daffodil > >>> classpath. > >>> > >>> Is this expected behavior? > >>> > >>> Mike Beckerle > >>> Apache Daffodil PMC | daffodil.apache.org > >>> OGF DFDL Workgroup Co-Chair | > >> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl > >>> Owl Cyber Defense | www.owlcyberdefense.com > >>> > >> > >> > > > >
