Hi So I have merged to master :-) On 18 August 2017 at 01:22, Martin Stockhammer <marti...@apache.org> wrote:
> Hi Olivier, > > great! I will look at it. I will give you feedback the next days. > And yes I have to optimize the jcr oak part and stabilize it. I will work > on it. > > Greetings > > Martin > > > > > Am 15. August 2017 11:30:04 MESZ schrieb Olivier Lamy <ol...@apache.org>: > >Hi > >Took a bit of time but I finally get the branch working :-) > >branch: feature/jcr_oak > >Let me know what do you think of? > >Well I guess there are still some optimisations to do for jcr oak > >I can see some logs: > >21:02:39.559 [1071] [main] WARN oak.query.QueryImpl - Traversal query > >(query without index): SELECT * FROM [nt:base] WHERE [jcr:uuid] = $id > >/* > >oak-internal */; consider creating an index > >21:02:39.563 [328] [main] WARN plugins.index.Cursors$TraversingCursor > >- > >Traversed 1000 nodes with filter Filter(query=SELECT * FROM [nt:base] > >WHERE > >[jcr:uuid] = $id /* oak-internal */, path=*, > >property=[jcr:uuid=[21232f29-7a57-35a7-8389-4a0e4a801fc3]]); consider > >creating an index or changing the query > > > > > > > > > > > >On 8 July 2017 at 06:22, Martin <marti...@apache.org> wrote: > > > >> Hi Olivier, > >> > >> great! > >> For my understanding: The dependency to lucene in the pom of > >indexer-core > >> is > >> still there, but the lucene packages are moved to the > >> ...maven.index.shaded... > >> package? You develop indexer-core with the standard lucene packages > >and the > >> shading is executed during the build of the indexer package? > >> > >> I think that may solve our dependency problem. > >> > >> I still got errors in the maven-indexer module, but I think the > >status is > >> still "work in progress". I don't want to interfere too much with > >your > >> changes. > >> > >> I'm not sure, if we should keep the JCR Oak as metadata > >implementation. I > >> think OrientDB may be a feasible alternative: Embeddable, Graph > >database, > >> Lucene index optional and may be omitted, Apache License. And with > >JCR Oak > >> we > >> also have to convert the existing metadata index. > >> > >> But one step after the other. If we agree that the shaded indexer > >works, we > >> should merge only the maven indexer changes to the master branch > >without > >> the > >> JCR/lucene update and change the JCR and or lucene afterwards. > >> > >> Greetings > >> > >> Martin > >> > >> Am Freitag, 7. Juli 2017, 09:23:24 CEST schrieb Olivier Lamy: > >> > So the repo contains a branch feature/jar_shaded_lucene here > >> > > >https://git1-us-west.apache.org/repos/asf?p=maven-indexer.git;a=summary > >> > and I pushed what I started for Archiva in the branch called > >> feature/jcr_oak > >> > So in order to test it you need to build first maven-indexer from > >the > >> > branch feature/jar_shaded_lucene > >> > > >> > On 6 July 2017 at 22:31, Olivier Lamy <ol...@apache.org> wrote: > >> > > I will try to share the work I did tomorrow in a branch > >> > > > >> > > On Thu, 6 Jul 2017 at 7:48 pm, Martin Stockhammer > ><marti...@apache.org > >> > > >> > > > >> > > wrote: > >> > >> We have different lucene (incompatible) dependencies that > >prevents us > >> to > >> > >> update the maven indexer and/or jackrabbit. And this will happen > >again > >> > >> with > >> > >> each upgrade from one of these two packages in the future. > >> > >> So would be really good if we can find a solution that removes > >one of > >> the > >> > >> lucene dependencies. > >> > >> > >> > >> Greetings > >> > >> > >> > >> Martin > >> > >> > >> > >> > >> > >> Am 6. Juli 2017 09:36:06 MESZ schrieb Chris Graham < > >> chrisgw...@gmail.com > >> > >> > >> > >> >Can I please an obvious/stupid question? > >> > >> > > >> > >> >What is driving this need for change? > >> > >> > > >> > >> >From a quick read of the thread above, all of the options > >appear to > >> > >> >introduce a lot of breaking changes, and a whole lot more > >> uncertainty. > >> > >> > > >> > >> >So, what is so broken that it is driving these changes? > >> > >> > > >> > >> >Sent from my iPhone > >> > >> > > >> > >> >> On 6 Jul 2017, at 12:39 pm, Olivier Lamy <ol...@apache.org> > >wrote: > >> > >> >> > >> > >> >> Yup. > >> > >> >> The idea is to have an extra jar produced by the > >maven-indexer with > >> > >> > > >> > >> >shaded > >> > >> > > >> > >> >> lucene version. > >> > >> >> So the lucene classes (version used by Maven indexer) will be > >> > >> > > >> > >> >relocated in > >> > >> > > >> > >> >> a package called org.apache.maven.index.shaded.lucene (such > >> > >> >> org.apache.maven.index.shaded.lucene.search.BooleanClause ) > >> > >> >> Then you exclude lucene dependencies used by maven indexer > >and > >> voila. > >> > >> >> The voila is a bit optimistic and not so ezy but anyway > >working on > >> it > >> > >> > > >> > >> >ATM. > >> > >> > > >> > >> >>> On 6 July 2017 at 07:08, Martin <marti...@apache.org> wrote: > >> > >> >>> > >> > >> >>> What do you mean exactly by shading? Moving to another > >package > >> name? > >> > >> >>> > >> > >> >>> Am Mittwoch, 5. Juli 2017, 01:19:17 CEST schrieb Olivier > >Lamy: > >> > >> >>>> maybe an option is to use some shading? > >> > >> >>>> I'm thinking of shading lucene packages used by maven > >indexer. I > >> > >> > > >> > >> >can > >> > >> > > >> > >> >>> easily > >> > >> >>> > >> > >> >>>> provide a build for that. > >> > >> >>>> WDYT? > >> > >> >>>> > >> > >> >>>>> On 26 June 2017 at 11:49, Olivier Lamy <ol...@apache.org> > >> wrote: > >> > >> >>>>> Hi > >> > >> >>>>> graph/document storage could be convenient (but not > >possible > >> with > >> > >> >>> > >> > >> >>> neo4j as > >> > >> >>> > >> > >> >>>>> it's GPL license [1]) > >> > >> >>>>> well we can add solr as an additional webapp with our > >jetty > >> > >> >>> > >> > >> >>> distribution > >> > >> >>> > >> > >> >>>>> but this will be a pain for users who want to use tomcat > >or any > >> > >> > > >> > >> >other > >> > >> > > >> > >> >>>>> servlet container... > >> > >> >>>>> we still need to investigate a new storage model :-) > >> > >> >>>>> > >> > >> >>>>> Olivier > >> > >> >>>>> [1] https://neo4j.com/licensing/ > >> > >> >>>>> > >> > >> >>>>>> On 25 June 2017 at 06:26, Martin <marti...@apache.org> > >wrote: > >> > >> >>>>>> Yes, you are right. The lucene dependency causes a lot of > >> trouble > >> > >> > > >> > >> >and > >> > >> > > >> > >> >>>>>> will > >> > >> >>>>>> cause headaches with each version change of one of the > >> > >> > > >> > >> >dependencies. > >> > >> > > >> > >> >>>>>> What are the requirements for a replacement? > >> > >> >>>>>> - We want to store hierarchical data? > >> > >> >>>>>> - We want to store metadata for nodes ? > >> > >> >>>>>> - Fulltext search (only metadata or for artifacts too?) > >> > >> >>>>>> - Blob / Artifact storage (I don't think so, but not so > >> familiar > >> > >> > > >> > >> >with > >> > >> > > >> > >> >>> the > >> > >> >>> > >> > >> >>>>>> archiva artifact model)? > >> > >> >>>>>> > >> > >> >>>>>> Maybe some graph database may be an alternative. Don't > >know if > >> > >> > > >> > >> >the > >> > >> > > >> > >> >>>>>> license of > >> > >> >>>>>> neo4j is compatible to the apache license, and I think it > >> brings > >> > >> >>> > >> > >> >>> lucene > >> > >> >>> > >> > >> >>>>>> as > >> > >> >>>>>> dependency too. I will have a look. > >> > >> >>>>>> Problem is, if there is fulltext search needed, I think, > >for > >> most > >> > >> > > >> > >> >of > >> > >> > > >> > >> >>> the > >> > >> >>> > >> > >> >>>>>> frameworks we get a lucene dependency, if it's embedded. > >> > >> >>>>>> > >> > >> >>>>>> Other alternatives: > >> > >> >>>>>> - Implement fulltext search by our own (index of the > >metadata > >> > >> > > >> > >> >stored > >> > >> > > >> > >> >>> via > >> > >> >>> > >> > >> >>>>>> the > >> > >> >>>>>> archiva api) and use the lucene dependency that comes > >from the > >> > >> >>>>>> maven-indexer > >> > >> >>>>>> - Jcr Oak with Solr. Solr is not embedded, must run as > >its own > >> > >> >>>>>> application > >> > >> >>>>>> (war). > >> > >> >>>>>> > >> > >> >>>>>> Greetings > >> > >> >>>>>> > >> > >> >>>>>> Martin > >> > >> >>>>>> > >> > >> >>>>>> Am Samstag, 24. Juni 2017, 14:05:26 CEST schrieb Olivier > >Lamy: > >> > >> >>>>>>> well this gonna be a pain. > >> > >> >>>>>>> IMHO we need to find a new alternative to jcr oak. > >> > >> >>>>>>> And something not using Lucene as it's a real pain to > >have > >> > >> > > >> > >> >different > >> > >> > > >> > >> >>>>>>> librairies using lucene as they do not update in the > >same time > >> > >> > > >> > >> >(and > >> > >> > > >> > >> >>>>>> Lucene > >> > >> >>>>>> > >> > >> >>>>>>> break backward compat so quickly...) > >> > >> >>>>>>> Any ideas? I'd like to have something embedded (but with > >a > >> > >> > > >> > >> >possible > >> > >> > > >> > >> >>>>>>> external server configuration). > >> > >> >>>>>>> There is currently a Cassandra implementation. I was not > >> > >> > > >> > >> >satisfied > >> > >> > > >> > >> >>>>>>> about > >> > >> >>>>>>> performance but I guess I did that 4yo ago so can be > >improved > >> > >> > > >> > >> >for > >> > >> > > >> > >> >>> sure > >> > >> >>> > >> > >> >>>>>> :-) > >> > >> >>>>>> : > >> > >> >>>>>>> Maybe orientdb? > >> > >> >>>>>>> What else? > >> > >> >>>>>>> > >> > >> >>>>>>>> On 24 June 2017 at 09:50, Olivier Lamy > ><ol...@apache.org> > >> > >> > > >> > >> >wrote: > >> > >> >>>>>>>> well the issue is non compatible version of Lucene for > >Maven > >> > >> >>> > >> > >> >>> Indexer > >> > >> >>> > >> > >> >>>>>> and > >> > >> >>>>>> > >> > >> >>>>>>>> Oak (well I can try push a patch to Oak for > >upgrading...) > >> > >> >>>>>>>> > >> > >> >>>>>>>>> On 24 June 2017 at 08:41, Olivier Lamy > ><ol...@apache.org> > >> > >> > > >> > >> >wrote: > >> > >> >>>>>>>>> Hi > >> > >> >>>>>>>>> Maven Indexer 6.0-SNAPSHOT doesn't need anymore plexus > >> bridge. > >> > >> >>>>>>>>> I'm working on it in the branch ( feature/jcr_oak ) > >> > >> >>>>>>>>> Not sure why but I have intermittent failure with > >store-jcr > >> > >> >>> > >> > >> >>> module. > >> > >> >>> > >> > >> >>>>>>>>> I definitely agree on the upgrade. > >> > >> >>>>>>>>> Well we can simply detect it's not oak compatible and > >> schedule > >> > >> > > >> > >> >a > >> > >> > > >> > >> >>>>>>>>> full > >> > >> >>>>>>>>> reindex (maybe with a message in logs and ui?) > >> > >> >>>>>>>>> But we need to be sure we can still read central index > >and > >> not > >> > >> >>> > >> > >> >>> sure > >> > >> >>> > >> > >> >>>>>> about > >> > >> >>>>>> > >> > >> >>>>>>>>> possible lucene conflict with oak and maven indexer. > >> > >> >>>>>>>>> We can work on this branch? (I created a Jenkins job > >for it > >> > >> >>>>>>>>> > >https://builds.apache.org/view/A-D/view/Archiva/job/archi > >> > >> >>>>>>>>> va-jcr-oak-branch/) > >> > >> >>>>>>>>> If you prefer master I would say no worries neither. > >> > >> >>>>>>>>> Something else to look at is upgrading maven-core > >etc... > >> > >> >>>>>>>>> Anyway > >> > >> >>>>>>>>> Cheers > >> > >> >>>>>>>>> Olivier > >> > >> >>>>>>>>> > >> > >> >>>>>>>>>> On 22 June 2017 at 19:16, Martin > ><marti...@apache.org> > >> wrote: > >> > >> >>>>>>>>>> Hi, > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>> upgrading the maven indexer leads to some major > >changes. > >> > >> >>>>>>>>>> Lucene is used by maven-indexer and also by > >jackrabbit. > >> > >> >>> > >> > >> >>> Jackrabbit > >> > >> >>> > >> > >> >>>>>>>>>> sticks to > >> > >> >>>>>>>>>> the old 3.x version and, as I see it, they will not > >move > >> to a > >> > >> >>> > >> > >> >>> newer > >> > >> >>> > >> > >> >>>>>>>>>> version. > >> > >> >>>>>>>>>> There is Jackrabbit Oak as alternative. > >> > >> >>>>>>>>>> I tried a proof of concept and could replace the > >jackrabbit > >> > >> >>>>>>>>>> implementation of > >> > >> >>>>>>>>>> metadata-store-jcr with a oak implementation. At > >least I > >> got > >> > >> > > >> > >> >the > >> > >> > > >> > >> >>>>>> unit > >> > >> >>>>>> > >> > >> >>>>>>>>>> tests of > >> > >> >>>>>>>>>> this module all to pass. > >> > >> >>>>>>>>>> But switching to Oak has some drawbacks: > >> > >> >>>>>>>>>> - The repository format changed and we must provide a > >way > >> to > >> > >> >>>>>>>>>> migrate > >> > >> >>>>>>>>>> (either > >> > >> >>>>>>>>>> migrate the existing repository or create a new one > >by > >> > >> >>> > >> > >> >>> reindexing) > >> > >> >>> > >> > >> >>>>>>>>>> - The lucene version used is newer but does not match > >to > >> the > >> > >> >>>>>>>>>> version > >> > >> >>>>>>>>>> from the > >> > >> >>>>>>>>>> maven-indexer dependencies. There may come up some > >> > >> >>>>>>>>>> incompatibilities > >> > >> >>>>>>>>>> that are > >> > >> >>>>>>>>>> not solvable without using a modified version of one > >of the > >> > >> >>> > >> > >> >>> both. > >> > >> >>> > >> > >> >>>>>>>>>> Or > >> > >> >>>>>>>>>> there may > >> > >> >>>>>>>>>> be the possibility to switch to solr (as separate > >> component) > >> > >> > > >> > >> >and > >> > >> > > >> > >> >>>>>> get rid > >> > >> >>>>>> > >> > >> >>>>>>>>>> of > >> > >> >>>>>>>>>> the lucene dependencies for jcr inside the archiva > >project. > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>> Switching to maven-indexer 6.0-SNAPSHOT means some > >changes > >> > >> > > >> > >> >too: > >> > >> >>>>>>>>>> - The Plexus-Sisu-Bridge does not work as before. > >> > >> >>>>>>>>>> - We must migrate from the NexusIndexer to the > >indexer API. > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>> So switching to the new indexer and oak means more > >work as > >> > >> >>> > >> > >> >>> expected > >> > >> >>> > >> > >> >>>>>> and > >> > >> >>>>>> > >> > >> >>>>>>>>>> some > >> > >> >>>>>>>>>> risks regarding new incompatibility problems. And I > >think > >> > >> > > >> > >> >this > >> > >> > > >> > >> >>>>>> cannot be > >> > >> >>>>>> > >> > >> >>>>>>>>>> done > >> > >> >>>>>>>>>> without broken master builds for some time period. > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>> So, what should we do? I think maven indexer is one > >of the > >> > >> > > >> > >> >core > >> > >> > > >> > >> >>>>>>>>>> components of > >> > >> >>>>>>>>>> archiva, and we should utilize the 3.x-version to > >migrate > >> to > >> > >> >>> > >> > >> >>> the > >> > >> >>> > >> > >> >>>>>> new > >> > >> >>>>>> > >> > >> >>>>>>>>>> indexer > >> > >> >>>>>>>>>> version, even if this means switching to jcr oak. > >Otherwise > >> > >> > > >> > >> >it > >> > >> > > >> > >> >>>>>>>>>> would > >> > >> >>>>>>>>>> mean to > >> > >> >>>>>>>>>> stick to the old version for the next years. > >> > >> >>>>>>>>>> @Olivier, regarding the maven-indexer / sisu-Bridge > >API > >> > >> >>> > >> > >> >>> changes, I > >> > >> >>> > >> > >> >>>>>> hope > >> > >> >>>>>> > >> > >> >>>>>>>>>> you > >> > >> >>>>>>>>>> can provide useful help. > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>> I committed the PoC to the branch feature/jcr_oak. > >There > >> are > >> > >> >>> > >> > >> >>> some > >> > >> >>> > >> > >> >>>>>>>>>> modules > >> > >> >>>>>>>>>> where the tests do not pass (mainly because of the > >indexer > >> > >> > > >> > >> >API > >> > >> > > >> > >> >>>>>> changes). > >> > >> >>>>>> > >> > >> >>>>>>>>>> Any comments? > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>> Cheers > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>> Martin > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>> Am Dienstag, 13. Juni 2017, 09:07:35 CEST schrieb > >Olivier > >> > >> > > >> > >> >Lamy: > >> > >> >>>>>>>>>>> forget it but we need to ensure we can read maven > >index > >> > >> >>> > >> > >> >>> files.... > >> > >> >>> > >> > >> >>>>>>>>>>> On 13 June 2017 at 17:06, Olivier Lamy > ><ol...@apache.org> > >> > >> >>> > >> > >> >>> wrote: > >> > >> >>>>>>>>>>>> Hi, > >> > >> >>>>>>>>>>>> Remember jackrabbit depends on Lucene as well so > >> upgrading > >> > >> >>>>>> > >> > >> >>>>>> Lucene > >> > >> >>>>>> > >> > >> >>>>>>>>>> can be a > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>>>> problem here. > >> > >> >>>>>>>>>>>> Regarding maven-indexer yes we can depend on a > >snapshot > >> > >> >>> > >> > >> >>> until > >> > >> >>> > >> > >> >>>>>> the > >> > >> >>>>>> > >> > >> >>>>>>>>>> release. > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>>>> I can release it ;-) > >> > >> >>>>>>>>>>>> > >> > >> >>>>>>>>>>>> On 13 June 2017 at 06:06, Martin > ><marti...@apache.org> > >> > >> >>> > >> > >> >>> wrote: > >> > >> >>>>>>>>>>>>> Hi, > >> > >> >>>>>>>>>>>>> > >> > >> >>>>>>>>>>>>> the lucene version depends on the maven indexer. > >But I'm > >> > >> >>> > >> > >> >>> not > >> > >> >>> > >> > >> >>>>>> sure > >> > >> >>>>>> > >> > >> >>>>>>>>>> about > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>>>>> the > >> > >> >>>>>>>>>>>>> current state of maven-indexer. The version has > >not > >> > >> > > >> > >> >changed > >> > >> > > >> > >> >>>>>> since > >> > >> >>>>>> > >> > >> >>>>>>>>>> some > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>>>>> 2013. > >> > >> >>>>>>>>>>>>> > >> > >> >>>>>>>>>>>>> There are commits on the master branch since then, > >and > >> the > >> > >> >>>>>> > >> > >> >>>>>> lucene > >> > >> >>>>>> > >> > >> >>>>>>>>>> version > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>>>>> has > >> > >> >>>>>>>>>>>>> been changed too, but no releases were tagged. > >> > >> >>>>>>>>>>>>> Does it make sense to switch to the maven-indexer > >> > >> >>>>>>>>>>>>> 6.0-SNAPSHOT? > >> > >> >>>>>>>>>>>>> > >> > >> >>>>>>>>>>>>> As I know there are new compact index formats with > >new > >> > >> >>> > >> > >> >>> lucene > >> > >> >>> > >> > >> >>>>>>>>>> versions > >> > >> >>>>>>>>>> > >> > >> >>>>>>>>>>>>> but I'm > >> > >> >>>>>>>>>>>>> not sure if this is relevant for the maven > >indexes. > >> > >> >>>>>>>>>>>>> > >> > >> >>>>>>>>>>>>> Cheers > >> > >> >>>>>>>>>>>>> > >> > >> >>>>>>>>>>>>> Martin > >> > >> >>>>>>>>>>>> > >> > >> >>>>>>>>>>>> -- > >> > >> >>>>>>>>>>>> Olivier Lamy > >> > >> >>>>>>>>>>>> http://twitter.com/olamy | > >http://linkedin.com/in/olamy > >> > >> >>>>>>>>> > >> > >> >>>>>>>>> -- > >> > >> >>>>>>>>> Olivier Lamy > >> > >> >>>>>>>>> http://twitter.com/olamy | > >http://linkedin.com/in/olamy > >> > >> >>>>>>>> > >> > >> >>>>>>>> -- > >> > >> >>>>>>>> Olivier Lamy > >> > >> >>>>>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy > >> > >> >>>>> > >> > >> >>>>> -- > >> > >> >>>>> Olivier Lamy > >> > >> >>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy > >> > >> >> > >> > >> >> -- > >> > >> >> Olivier Lamy > >> > >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy > >> > >> > >> > >> -- > >> > >> Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail > >gesendet. > >> > > > >> > > -- > >> > > Olivier Lamy > >> > > http://twitter.com/olamy | http://linkedin.com/in/olamy > >> > >> > >> > > > > > >-- > >Olivier Lamy > >http://twitter.com/olamy | http://linkedin.com/in/olamy > > -- > Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet. > -- Olivier Lamy http://twitter.com/olamy | http://linkedin.com/in/olamy