Hi, this sounds very interesting. Do you know which versions of Lucene are supported? Do you know if it would work with upcoming Lucene 3.0.x? https://jira.jboss.org/jira/browse/ISPN-275
Regards, Lukas http://blog.lukas-vlcek.com/ On Sun, Nov 15, 2009 at 5:33 AM, Sanne Grinovero < s.grinov...@sourcesense.com> wrote: > Hi John, > I didn't run a long running reliable benchmark, so at the moment I > can't really speak of numbers. > Suggestions and help on performance testing are welcome: I guess it > will shine in some situations, not necessarily all, so really choosing > a correct ratio of concurrent writers/searches, number of nodes in the > cluster and resources per node will never be fair enough to compare > this Directory with others. > > On paper the premises are good: it's all in-memory, until it fits: it > will distribute data across nodes and overflow to disk is supported > (called passivation). A permanent store can be configured, so you > could set it to periodically flush incrementally to slower storages > like a database, a filesystem, a cloud storage service. This makes it > possible to avoid losing state even when all nodes are shut down. > A RAMDirectory is AFAIK not recommended as you could hit memory limits > and because it's basically a synchronized HashMap; Infinispan > implements ConcurrentHashMap and doesn't need synchronization. > Even if the data is replicated across nodes each node has it's own > local cache, so when caches are warm and all segments fit in memory it > should be, theoretically, the fastest Directory ever. The more it will > read from disk, the more it will behave similarly to a FSDirectory > with some buffers. > > As per Lucene's design, writes can happen only at one node at a time: > one IndexWriter can own the lock, but IndexReaders and Searchers are > not blocked, so when using this Directory it should behave exactly as > if you had multiple processes sharing a local NIOFSdirectory: > basically the situation is that you can't scale on writers, but you > can scale near-linearly with readers adding in more power from more > machines. > > Besides performance, the reasons to implement this was to be able to > easily add or remove processing power to a service (clouds), make it > easier to share indexes across nodes, and last but not least to remove > single points of failure: all data is distributed and there is no such > notion of Master: services will continue running fine when killing any > node. > > I hope this peeks your interest, sorry if I couldn't provide numbers. > > Regards, > Sanne > > On Sat, Nov 14, 2009 at 11:15 PM, John Wang <john.w...@gmail.com> wrote: > > HI Sanne: > > > > Very interesting! > > > > What kinda performance should we expect with this, comparing to > regular > > FSDIrectory on local HD. > > Thanks > > -John > > > > On Sat, Nov 14, 2009 at 11:44 AM, Sanne Grinovero > > <s.grinov...@sourcesense.com> wrote: > >> > >> Hello all, > >> I'm a Lucene user and fan, I wanted to tell you that we just released > >> a first technology preview of a distributed in memory Directory for > >> Lucene. > >> > >> The release announcement: > >> > >> > http://infinispan.blogspot.com/2009/11/second-release-candidate-for-400.html > >> > >> From there you'll find links to the Wiki, to the sources, to the issue > >> tracker. A minimal demo is included with the sources. > >> > >> This was developed together with Google Summer of Code student Lukasz > >> Moren and much support from the Infinispan and Hibernate Search teams, > >> as we are storing the index segments on Infinispan and using it's > >> atomic distributed locks to implement a Lucene LockFactory. > >> > >> Initial idea was to contribute it directly to Lucene, but as > >> Infinispan is a LGPL dependency we had to distribute it with > >> Infinispan (as the other way around would have introduced some legal > >> issues); still we hope you appreciate the effort and are interested in > >> giving it a try. > >> All kind of feedback is welcome, especially on benchmarking > >> methodologies as I yet have to do some serious performance tests. > >> > >> Main code, build with Maven2: > >> svn co > >> > http://anonsvn.jboss.org/repos/infinispan/tags/4.0.0.CR2/lucene-directory/ > >> infinispan-directory > >> > >> Demo, see the Readme: > >> svn co > >> > http://anonsvn.jboss.org/repos/infinispan/tags/4.0.0.CR2/demos/lucene-directory/ > >> lucene-demo > >> > >> Best Regards, > >> Sanne > >> > >> -- > >> Sanne Grinovero > >> Sourcesense - making sense of Open Source: http://www.sourcesense.com > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-dev-h...@lucene.apache.org > >> > > > > > > > > -- > Sanne Grinovero > Sourcesense - making sense of Open Source: http://www.sourcesense.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org > >
