mattcasters commented on issue #5707:
URL: https://github.com/apache/hop/issues/5707#issuecomment-3304252801
```
I'm not sure to understand why we need this on hand class loading
33 replies
Matt Casters
[22 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135068612859?thread_ts=1758094721.051119&cid=C07T7TK66E8)
In theory you can have classpaths per plugin. In practice you can't have
conflicting libraries anyway. I'm all for ideas to have a single classpath
like we have in the Beam runtimes (one fat jar does that). We'd need some
wrapper to collects the libraries from all the plugins but that's about it.
Francois Papon
[16 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135424937969?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Yes I can imagine that if 2 plugins want to use 2 differents versions of the
same 3rd party lib it will not work with a single classloader but as you said
if it will not happen, no need to have multiple classloaders.
Matt Casters
[15 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135456450639?thread_ts=1758094721.051119&cid=C07T7TK66E8)
This stuff dates back to Kettle version 2.0 from 2004, go figure.
Francois Papon
[14 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135503191949?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I eard from old Kettle users that it was runing on Apache Karaf which is OSGi
Matt Casters
[14 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135505151579?thread_ts=1758094721.051119&cid=C07T7TK66E8)
There was an older attempt which was really slow but that got solved with
the annotation indexing.
[8:58](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135534504659?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Karaf was introduced in Kettle 5-6 I think
Francois Papon
[14 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135548952859?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Ha yes 5
Matt Casters
[14 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135550187299?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I think they removed it in 9.x
Francois Papon
[13 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135556640859?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Yes
Matt Casters
[13 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135611880679?thread_ts=1758094721.051119&cid=C07T7TK66E8)
But we could build a new collector for the classpath, even build a new janex
file on first load to have faster caching and loading, ... I'm game
:upvote:
1
Francois Papon
[12 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135645648319?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Another thing is that I don't know yet if it's easily possible to load a
dynamic classpath only based on the plugins used in a workflow
Matt Casters
[12 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135658210929?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I would make it easier to call other "plugins" since they'd be simply in the
same classpath
Francois Papon
[11 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135703969679?thread_ts=1758094721.051119&cid=C07T7TK66E8)
As a Karaf PMC, I know well the OSGi/multiple classloader stuffs, it can be
a nightmare....
Matt Casters
[10 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135760530629?thread_ts=1758094721.051119&cid=C07T7TK66E8)
You need to serialize/de-serialize metadata to make sure you're loading in
the right classloader. Not ideal.
[9:03](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135786444529?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Not the end of the world either but still...
Francois Papon
[9 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135794694869?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Having to much unused lib in the classpath can use a lot of cpu on startup
Matt Casters
[9 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135832021219?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Well, all plugins are touched at startup so they're in at least one
classpath now as well.
[9:04](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135864675439?thread_ts=1758094721.051119&cid=C07T7TK66E8)
We actually look in the jar files to find plugins.
[9:04](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135879616759?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I don't think one classpath would be worse.
Francois Papon
[8 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135893720609?thread_ts=1758094721.051119&cid=C07T7TK66E8)
All? Even ones that are not used in the workflow/pipelines?
Matt Casters
[8 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135905685819?thread_ts=1758094721.051119&cid=C07T7TK66E8)
We'd have one Jandex index to go through, not hundreds.
[9:05](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135922691399?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I suspect all, yes.
[9:05](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135939849289?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Perhaps not those jar files that are not used to run.
[9:05](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135949627429?thread_ts=1758094721.051119&cid=C07T7TK66E8)
If the JVM is smart enough.
[9:06](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135964947989?thread_ts=1758094721.051119&cid=C07T7TK66E8)
But that could still be the case in the new scenario.
[9:06](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135984360409?thread_ts=1758094721.051119&cid=C07T7TK66E8)
"In the classpath but not yet read" that is.
Francois Papon
[6 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136018112029?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Yep
[9:08](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136121907039?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I can imagine that if a user has a short list of the plugins that he usually
use, he can also make a custom docker image or assembly with only his plugins
embedded
[9:09](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136144648319?thread_ts=1758094721.051119&cid=C07T7TK66E8)
To have a lightweight image
Matt Casters
[3 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136159009949?thread_ts=1758094721.051119&cid=C07T7TK66E8)
That wouldn't change either way.
[9:09](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136177623509?thread_ts=1758094721.051119&cid=C07T7TK66E8)
We can trim Hop down to below 100MB I think
Francois Papon
[3 minutes
ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136196257379?thread_ts=1758094721.051119&cid=C07T7TK66E8)
vs 700MB yes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
