Hi graph/document storage could be convenient (but not possible with neo4j as it's GPL license [1]) well we can add solr as an additional webapp with our jetty distribution but this will be a pain for users who want to use tomcat or any other servlet container... we still need to investigate a new storage model :-)
Olivier [1] https://neo4j.com/licensing/ On 25 June 2017 at 06:26, Martin <marti...@apache.org> wrote: > Yes, you are right. The lucene dependency causes a lot of trouble and will > cause headaches with each version change of one of the dependencies. > What are the requirements for a replacement? > - We want to store hierarchical data? > - We want to store metadata for nodes ? > - Fulltext search (only metadata or for artifacts too?) > - Blob / Artifact storage (I don't think so, but not so familiar with the > archiva artifact model)? > > Maybe some graph database may be an alternative. Don't know if the license > of > neo4j is compatible to the apache license, and I think it brings lucene as > dependency too. I will have a look. > Problem is, if there is fulltext search needed, I think, for most of the > frameworks we get a lucene dependency, if it's embedded. > > Other alternatives: > - Implement fulltext search by our own (index of the metadata stored via > the > archiva api) and use the lucene dependency that comes from the > maven-indexer > - Jcr Oak with Solr. Solr is not embedded, must run as its own application > (war). > > Greetings > > Martin > > > > Am Samstag, 24. Juni 2017, 14:05:26 CEST schrieb Olivier Lamy: > > well this gonna be a pain. > > IMHO we need to find a new alternative to jcr oak. > > And something not using Lucene as it's a real pain to have different > > librairies using lucene as they do not update in the same time (and > Lucene > > break backward compat so quickly...) > > Any ideas? I'd like to have something embedded (but with a possible > > external server configuration). > > There is currently a Cassandra implementation. I was not satisfied about > > performance but I guess I did that 4yo ago so can be improved for sure > :-) > > Maybe orientdb? > > What else? > > > > On 24 June 2017 at 09:50, Olivier Lamy <ol...@apache.org> wrote: > > > well the issue is non compatible version of Lucene for Maven Indexer > and > > > Oak (well I can try push a patch to Oak for upgrading...) > > > > > > On 24 June 2017 at 08:41, Olivier Lamy <ol...@apache.org> wrote: > > >> Hi > > >> Maven Indexer 6.0-SNAPSHOT doesn't need anymore plexus bridge. > > >> I'm working on it in the branch ( feature/jcr_oak ) > > >> Not sure why but I have intermittent failure with store-jcr module. > > >> I definitely agree on the upgrade. > > >> Well we can simply detect it's not oak compatible and schedule a full > > >> reindex (maybe with a message in logs and ui?) > > >> But we need to be sure we can still read central index and not sure > about > > >> possible lucene conflict with oak and maven indexer. > > >> We can work on this branch? (I created a Jenkins job for it > > >> https://builds.apache.org/view/A-D/view/Archiva/job/archi > > >> va-jcr-oak-branch/) > > >> If you prefer master I would say no worries neither. > > >> Something else to look at is upgrading maven-core etc... > > >> Anyway > > >> Cheers > > >> Olivier > > >> > > >> On 22 June 2017 at 19:16, Martin <marti...@apache.org> wrote: > > >>> Hi, > > >>> > > >>> upgrading the maven indexer leads to some major changes. > > >>> Lucene is used by maven-indexer and also by jackrabbit. Jackrabbit > > >>> sticks to > > >>> the old 3.x version and, as I see it, they will not move to a newer > > >>> version. > > >>> There is Jackrabbit Oak as alternative. > > >>> I tried a proof of concept and could replace the jackrabbit > > >>> implementation of > > >>> metadata-store-jcr with a oak implementation. At least I got the unit > > >>> tests of > > >>> this module all to pass. > > >>> But switching to Oak has some drawbacks: > > >>> - The repository format changed and we must provide a way to migrate > > >>> (either > > >>> migrate the existing repository or create a new one by reindexing) > > >>> - The lucene version used is newer but does not match to the version > > >>> from the > > >>> maven-indexer dependencies. There may come up some incompatibilities > > >>> that are > > >>> not solvable without using a modified version of one of the both. Or > > >>> there may > > >>> be the possibility to switch to solr (as separate component) and get > rid > > >>> of > > >>> the lucene dependencies for jcr inside the archiva project. > > >>> > > >>> Switching to maven-indexer 6.0-SNAPSHOT means some changes too: > > >>> - The Plexus-Sisu-Bridge does not work as before. > > >>> - We must migrate from the NexusIndexer to the indexer API. > > >>> > > >>> So switching to the new indexer and oak means more work as expected > and > > >>> some > > >>> risks regarding new incompatibility problems. And I think this > cannot be > > >>> done > > >>> without broken master builds for some time period. > > >>> > > >>> So, what should we do? I think maven indexer is one of the core > > >>> components of > > >>> archiva, and we should utilize the 3.x-version to migrate to the new > > >>> indexer > > >>> version, even if this means switching to jcr oak. Otherwise it would > > >>> mean to > > >>> stick to the old version for the next years. > > >>> @Olivier, regarding the maven-indexer / sisu-Bridge API changes, I > hope > > >>> you > > >>> can provide useful help. > > >>> > > >>> I committed the PoC to the branch feature/jcr_oak. There are some > > >>> modules > > >>> where the tests do not pass (mainly because of the indexer API > changes). > > >>> > > >>> Any comments? > > >>> > > >>> Cheers > > >>> > > >>> Martin > > >>> > > >>> Am Dienstag, 13. Juni 2017, 09:07:35 CEST schrieb Olivier Lamy: > > >>> > forget it but we need to ensure we can read maven index files.... > > >>> > > > >>> > On 13 June 2017 at 17:06, Olivier Lamy <ol...@apache.org> wrote: > > >>> > > Hi, > > >>> > > Remember jackrabbit depends on Lucene as well so upgrading Lucene > > >>> > > >>> can be a > > >>> > > >>> > > problem here. > > >>> > > Regarding maven-indexer yes we can depend on a snapshot until the > > >>> > > >>> release. > > >>> > > >>> > > I can release it ;-) > > >>> > > > > >>> > > On 13 June 2017 at 06:06, Martin <marti...@apache.org> wrote: > > >>> > >> Hi, > > >>> > >> > > >>> > >> the lucene version depends on the maven indexer. But I'm not > sure > > >>> > > >>> about > > >>> > > >>> > >> the > > >>> > >> current state of maven-indexer. The version has not changed > since > > >>> > > >>> some > > >>> > > >>> > >> 2013. > > >>> > >> > > >>> > >> There are commits on the master branch since then, and the > lucene > > >>> > > >>> version > > >>> > > >>> > >> has > > >>> > >> been changed too, but no releases were tagged. > > >>> > >> Does it make sense to switch to the maven-indexer 6.0-SNAPSHOT? > > >>> > >> > > >>> > >> As I know there are new compact index formats with new lucene > > >>> > > >>> versions > > >>> > > >>> > >> but I'm > > >>> > >> not sure if this is relevant for the maven indexes. > > >>> > >> > > >>> > >> Cheers > > >>> > >> > > >>> > >> Martin > > >>> > > > > >>> > > -- > > >>> > > Olivier Lamy > > >>> > > http://twitter.com/olamy | http://linkedin.com/in/olamy > > >> > > >> -- > > >> Olivier Lamy > > >> http://twitter.com/olamy | http://linkedin.com/in/olamy > > > > > > -- > > > Olivier Lamy > > > http://twitter.com/olamy | http://linkedin.com/in/olamy > > > -- Olivier Lamy http://twitter.com/olamy | http://linkedin.com/in/olamy
