Looks like the ModuleLayer.defineModules is the thing I needed. With it, I can use my current classloader to load all the modules and it can find the classes from both module and unnamed module.
Note that class loaders for modules behave differently then original class loaders. For example they will favor local class definitions over class definitions of the parent loader. Not sure if there are other important differences. The loader is implemented in jdk.internal.loader.Loader [1].
There is some trick to get a class from an upper layer to bind to the a class in the boot layer?
You can pass the created module layer as first argument to ServiceLoader.load​(ModuleLayer, Class) [2].
[1] https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/jdk/internal/loader/Loader.java
[2] https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/util/ServiceLoader.html#load(java.lang.ModuleLayer,java.lang.Class)
