Re: A new Lucene Directory available

2009-11-16 Thread Sergio Bossa
Sanne, I'd be very interested in knowing what kind of problems you analyzed and resolved regarding the Terracotta clustered solution, as quoted below: I know about the Terracotta efforts, I agree with you and have collected much feedback about which problems were arising directly talking with

Re: A new Lucene Directory available

2009-11-16 Thread Manik Surtani
@Sanne, thanks for announcing this, good stuff! @Earwin, note that this is a tech preview and hardly production-ready code yet. The more eyes that scan the code, try it out, report bugs and bottlenecks, the better. So thanks for spotting ISPN-276, we look forward to more feedback/patches.

Re: A new Lucene Directory available

2009-11-15 Thread Earwin Burrfoot
Terracotta guys easy-clustered Lucene a few years ago. I'm yet to see at least one person saying it worked for him allright. This new directory ain't gonna be faster than RAMDirectory, as syncs on a map doesn't matter, they are taken once per opened file - once per reopen, which is not happening

Re: A new Lucene Directory available

2009-11-15 Thread Sanne Grinovero
Hi Lukas, Our reference during early design was Lucene 2.4.1, but we look forward for compatibility and new tricks. Current trunk is compatible towards Lucene's trunk, but I won't close ISPN-275 until it's confirmed against a released Lucene 3.0.0 : hopefully this will come before Infinispan 4

Re: A new Lucene Directory available

2009-11-15 Thread Sanne Grinovero
Hi Earwin, thanks for the insight, as I mentioned I have no proper benchmarks to back my statements but I can see how it behaves, so absolutely I could be too optimistic. They are currently profiling Infinispan and speeding up some internals, so I'll wait for these tasks to finish to begin testing

Re: A new Lucene Directory available

2009-11-15 Thread Earwin Burrfoot
About the RAMDirectory comparison, as you said yourself the bytes aren't read constantly but just at index reopen so I wouldn't be too worried about the bunch of methods as they're executed once per segment loading; The bytes /are/ read constantly (readByte() method). I believe that is the

Re: A new Lucene Directory available

2009-11-15 Thread Sanne Grinovero
Hi again Earwin, thanks you very much for spotting the byte reading issue, it's definitely not as I wanted it. https://jira.jboss.org/jira/browse/ISPN-276 I never tried to defend an improved updates/s ratio, just maybe compared to scheduled rsyncs :-) Our goal is to scale on queries/sec while

Re: A new Lucene Directory available

2009-11-14 Thread Sanne Grinovero
Hi John, I didn't run a long running reliable benchmark, so at the moment I can't really speak of numbers. Suggestions and help on performance testing are welcome: I guess it will shine in some situations, not necessarily all, so really choosing a correct ratio of concurrent writers/searches,

Re: A new Lucene Directory available

2009-11-14 Thread Lukáš Vlček
Hi, this sounds very interesting. Do you know which versions of Lucene are supported? Do you know if it would work with upcoming Lucene 3.0.x? https://jira.jboss.org/jira/browse/ISPN-275 Regards, Lukas http://blog.lukas-vlcek.com/ On Sun, Nov 15, 2009 at 5:33 AM, Sanne Grinovero