2011/3/22 Massimo Lusetti <mluse...@gmail.com> > On Tue, Mar 22, 2011 at 6:40 PM, Rick Bullotta > <rick.bullo...@thingworx.com> wrote: > > > Hi, Massimo. > > > > When you say you are using an externally managed Lucene index, does that > imply that you are not using the Neo index framework and interacting with > Lucene directly? > > Summarizing, yes. > > My use case is that I have a lot of log row each of which need to be > processed one and only once, so I calculate the hash for each row, > store it in the Index and check for presence before process the log > row. This way I can skip the already processed logs. > > My logs are already 220.000.000 (last December) and are growing fast. > > My tests shows that using the internal Index framework for such a big > Index has very poor performance which using Lucene "externally" it > doesn't. > If you need numbers I think there'are some threads of mine in the list. > > For the records there's only one difference in the two solutions: > using the internal index makes me index a Node while using the > external index I simply put "1" in the indexed field cause I simply > need to know if the key is present or not, I didn't study the internal > of the Neo4j's Index framework but this difference probably could > cause the difference performance... >
Having index modifications transactional comes with a cost, I wouldn't expect this much though. Another thing is that I'm planning to a an index "type" which is one-to-one where you can index/query the lucene index more performant in scenarios like this. Your solutions is even more specialized for your case, so it's probably hard to match that :) > > Cheers > -- > Massimo > http://meridio.blogspot.com > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
