Hello Hops, I've been struggling with a few classpath related issues:
* Plugin data types are only accessible from the plugin they were introduced with (Avro, Graph) * It's not possible in a safe way for another plugin to add plugins (Beam) to the root class loader This has been causing all sorts of class loader problems which are typically resolved by either shoving everything in the root classloader (Avro data type) or by having large blobs for a plugin (engines/beam). In the ideal scenario we'd have for example all the Kafka plugins in one plugin with all the dependencies nicely grouped together in one plugins/transforms/kafka folder and this would include all the Beam related code as well. The caveats being that we can't ship the Beam libs in every plugin and that it should be easy to get rid of functionality. What I would love to do is come up with an alternative way of assembling and building our software. For this to happen I think it should be possible for any external "plugin" project to register classes in the root class loader. To make this happen there are various options like for example having an extra folder like "libroot" in the plugin folder. It would have to act as if the libraries in there belonged to the root classloader and our scripts would need to be able to pick this up. I would also love to see some extra metadata around the libraries that we have assembled in folders. For example we'll want to create a smarter "fat jar" builder which knows that Spark, Flink and Dataflow are different platforms and that we don't need all libraries from all 3 platforms to run something on either of those. Perhaps by splitting up libraries in a more fine-grained manner we can also add a small JSON file like "library-metadata.json" containing some metadata that can then be picked up by the fat jar builder? In the plugins/engines/beam case you'd have folders: lib, libroot, libspark, libflink, libdataflow, ... The Kafka transforms code could move to plugins/transforms/kafka and so on. Let's brainstorm around the possibilities and the possible problems to come up with the next architecture for Hop. Cheers, Matt
