Osma-- The short answer is that yes, given the right tools you _can_ have different versions of code accessible in different ways. The longer answer is that it's probably not a viable alternative for Jena for this problem, at least not without a lot of other change.
You are right to point to the classloader mechanism as being at the heart of this question, but I must alter your remark just slightly. From "the Java classloader only sees a single, flat package/class namespace and a set of compiled classes" to "ANY GIVEN Java classloader only sees a single, flat package/class namespace and a set of compiled classes". This is the fact that OSGi uses to make it possible to maintain strict module boundaries (and even dynamic module relationships at run-time). Each OSGi bundle sees its own classloader, and the framework is responsible for connecting bundles up to ensure that every bundle has what it needs in the way of types to function, based on metadata that the bundles provide to the framework. It's an incredibly powerful system (I use it every day and enjoy it enormously) but it's also very "heavy" and requires a good deal of investment to use. In particular, it's probably too large to put _inside_ Jena. (I frequently put Jena inside an OSGi instance, on the other hand.) Java 9 Jigsaw [1] offers some possibility for strong modularization of this kind, but it's really meant for the JDK itself, not application libraries. In theory, we could "roll our own" classloader management for this problem. That sounds like more than a bit of a rabbit hole to me. There might be another, more lightweight, toolkit out there to this purpose, but I'm not aware of any myself. Otherwise, yes, you get into shading and the like. We have to do that for Guava for now because of HADOOP-10101 (grumble grumble) but it's hardly a thing we want to do any more of than needed, I don't think. --- A. Soroka The University of Virginia Library [1] http://openjdk.java.net/projects/jigsaw/ > On Mar 1, 2017, at 9:03 AM, Osma Suominen <osma.suomi...@helsinki.fi> wrote: > > Hi Anuj! > > Thanks for the clarification. > > However, I'm still not sure I understand the situation completely. I know > Maven can perform a lot of tricks, but Maven modules are just convenient ways > to structure a Java project. Maven cannot change the fact that at runtime, > module divisions don't really matter (except that they usually correspond to > package sub-namespaces) and the Java classloader only sees a single, flat > package/class namespace and a set of compiled classes (usually within JARs) > in the classpath that it needs to check to find the right classes, and if > there are two versions of the same library (eg Lucene) with overlapping class > names, that's going to cause trouble. The only way around that is to shade > some of the libraries, i.e. rename them so that they end up in another, > non-conflicting namespace. Apparently Elasticsearch also did some of that in > the past [1] but nowadays tries to avoid it. > > Does your assumption 1 ("At a given point in time, only a single Indexing > Technology is used") imply that in the assembler configuration, you cannot > have ja:loadClass declarations for both Lucene and ES backends? Or how do you > run something like Fuseki that contains (in a single big JAR) both the > jena-text and jena-text-es modules with all their dependencies, one of which > requires the Lucene 4.x classes and the other one the Lucene 6.4.1 classes? > How do you ensure that only one of them is used at a time, and that the Java > classloader, even though it has access to both versions of Lucene, only loads > classes from the single, correct one and not the other? Or do you need to > have separate "Fuseki-Lucene" and "Fuseki-ES" packages, so that you don't end > up with two Lucene versions within the same Fuseki JAR? > > -Osma > > [1] https://www.elastic.co/blog/to-shade-or-not-to-shade > > 01.03.2017, 11:03, anuj kumar kirjoitti: >> Hi Osma, >> >> I understand what you are saying. There are ways to mitigate risks and >> balance the refactoring without affecting the existing modules. But I will >> not delve into those now. I am not an expert in Jena to convincingly say >> that it is possible, without any hiccups. But I can take a guess and say >> that it is indeed possible :) >> >> For the question: "is it even possible to mix modules that depend on >> different versions of the Lucene libraries within the same project?" >> >> I actually do not understand what you mean by mixing modules. I assume you >> mean having jena-text and jena-text-es as dependencies in a build without >> causing the build to conflict. If that is what you mean than the answer is >> yes it is possible and quite simple as well. Let me explain how it is >> possible. But before that some assumption which I want to call out >> explicitly. >> >> *Assumption:* >> 1. At a given point in time, only a single Indexing Technology is used for >> text based indexing and searching via Jean. What this means is that we will >> either use Lucene Implementation OR Solr Implementation OR ES >> Implementation at any given point in time. >> 2. Fuseki build does not depend on any Lucene 4.9.1 specific classes but >> only on jena-text classes, if at all. >> >> Based on these assumptions it is possible to create a build that contains >> jena-text based common classes + ES specific classes without any >> compatibility issues. And it is infact quite simple. I did it in the >> current jena-text-es module and ran the entire build which succeeded. >> The key is to include the latest Lucene dependencies at the very beginning >> in the pom and then include jena-text dependency. Maven will then >> automatically resolve the dependency issues by including the Lucene >> librarires that we included in our es specific pom. Have a look the pom of >> jena-text-es module here to see how it can be done : >> https://github.com/EaseTech/jena/blob/master/jena-text-es/pom.xml >> >> >> Thanks, >> Anuj Kumar >> >> >> On Wed, Mar 1, 2017 at 7:27 AM, Osma Suominen <osma.suomi...@helsinki.fi> >> wrote: >> >>> Hi Anuj, >>> >>> I understand your concerns. However, we also need to balance between the >>> needs of individual modules/features and the whole codebase. I'm willing to >>> put in the effort to keep the other modules up to date with newer Lucene >>> versions. Lucene upgrade requirements are well documented, the only hitches >>> seen in JENA-1250 were related to how jena-text (ab)used some Lucene >>> features that were dropped from newer versions. >>> >>> A perhaps stupid question to more experienced Java developers: is it even >>> possible to mix modules that depend on different versions of the Lucene >>> libraries within the same project? In my (quite limited) understanding of >>> Java projects and libraries, this requires special arrangements (e.g. >>> shading) as the Java package/class namespace is shared by all the code >>> running within the same JVM. >>> >>> So can you create, say, a Fuseki build that contains the current jena-text >>> module (depending on Lucene 4.x) and the new jena-text-es module (depending >>> on Lucene 6.4.1) without any compatibility issues? >>> >>> -Osma >>> >>> >>> >>> >>> 01.03.2017, 00:47, anuj kumar kirjoitti: >>> >>>> Hi, >>>> >>>> My 2 Cents : >>>> >>>> The reason I proposed to have separate modules for Lucene, Solr and ES is >>>> exactly for avoiding the "All or Nothing" approach we need to take if we >>>> club them all together. If they stay together and if in the near future I >>>> want to upgrade ES to another version, I also need to again upgrade Lucene >>>> and Solr and possibly another implementation that may have been added >>>> during the time. As we all know, this means weeks of work if not months to >>>> get the changes released. This will personally de-motivate me to do >>>> anything and I will probably start maintaining my version of Jena-Text as >>>> that would be much simpler to do than to upgrade and test and in the >>>> process own(read fix bugs) the upgrade for each and every technology. >>>> >>>> If they are developed as separate modules, they can evolve independently >>>> of >>>> each other and we can avoid situations where we cant upgrade to latest >>>> version of Lucene because we do not know what effect it will have on Solr >>>> Implementation. >>>> >>>> We can start with having a separate Module for Jena Text ES and see how >>>> things go. If they go well, we could extract out Solr and Lucene out of >>>> Jena Text. >>>> >>>> Again this is just a suggestion based on my limited industry experience. >>>> >>>> Thanks, >>>> Anuj Kumar >>>> >>>> >>>> >>>> On Tue, Feb 28, 2017 at 5:23 PM, Osma Suominen <osma.suomi...@helsinki.fi >>>>> >>>> wrote: >>>> >>>> 28.02.2017, 17:12, A. Soroka kirjoitti: >>>>> >>>>> https://lists.apache.org/thread.html/dce0d502b11891c28e57bbc >>>>>> bb0cdef27d8374d58d9634076b8ef4cd7@1431107516@%3Cdev.jena.apache.org%3E >>>>>> ? In other words, might it be better to factor out between -text and >>>>>> -spatial and _then_ try to upgrade the Lucene version? >>>>>> >>>>>> >>>>> I certainly wouldn't object to that, but somebody has to volunteer to do >>>>> the actual work! >>>>> >>>>> I don't use the Solr component now, but I could easily see so doing... >>>>> >>>>>> that's pretty vague, I know, and I'm not in a position to do any work to >>>>>> maintain it, so consider that just a very small and blurry data point. >>>>>> :) >>>>>> >>>>>> >>>>> Last time I tried it (it was a while ago) I couldn't figure out how to >>>>> get >>>>> it running... If you could just try that with some toy data, then your >>>>> data >>>>> point would be a lot less blurry :) I haven't used Solr for anything, so >>>>> I'm not very familiar with how to set it up, and the jena-text >>>>> instructions >>>>> are pretty vague unfortunately. >>>>> >>>>> >>>>> -Osma >>>>> >>>>> >>>>> -- >>>>> Osma Suominen >>>>> D.Sc. (Tech), Information Systems Specialist >>>>> National Library of Finland >>>>> P.O. Box 26 (Kaikukatu 4) >>>>> 00014 HELSINGIN YLIOPISTO >>>>> Tel. +358 50 3199529 >>>>> osma.suomi...@helsinki.fi >>>>> http://www.nationallibrary.fi >>>>> >>>>> >>>> >>>> >>>> >>> >>> -- >>> Osma Suominen >>> D.Sc. (Tech), Information Systems Specialist >>> National Library of Finland >>> P.O. Box 26 (Kaikukatu 4) >>> 00014 HELSINGIN YLIOPISTO >>> Tel. +358 50 3199529 >>> osma.suomi...@helsinki.fi >>> http://www.nationallibrary.fi >>> >> >> >> > > > -- > Osma Suominen > D.Sc. (Tech), Information Systems Specialist > National Library of Finland > P.O. Box 26 (Kaikukatu 4) > 00014 HELSINGIN YLIOPISTO > Tel. +358 50 3199529 > osma.suomi...@helsinki.fi > http://www.nationallibrary.fi
