Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

On Tue, Feb 23, 2021 at 2:30 AM wrote: > Hi,- > > I tried MMapDirectory and i allocated as big as index size on my J2EE > Container but > > Don't allocate java heap memory for the index, MMapDirectory does not use java heap memory!

[ANNOUNCE] Apache Lucene 8.8.1 released

The Lucene PMC is pleased to announce the release of Apache Lucene 8.8.1. Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform. Thi

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Ok, but how is this MMapDirectory used then? Best regards On 2/23/21 7:03 AM, Robert Muir wrote: On Tue, Feb 23, 2021 at 2:30 AM > wrote: Hi,-   I tried MMapDirectory and i allocated as big as index size on my J2EE Container but Don't alloc

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

As Uwe suggested some time ago, tmpfs file system usage with MMapDirectory is the only way to get high speedup wrt on disk Lucene index, right? Best regards On 2/23/21 1:44 PM, baris.ka...@oracle.com wrote: Ok, but how is this MMapDirectory used then? Best regards On 2/23/21 7:03 AM, Rob

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Don't give gobs of memory to your java process, you will just make things slower. The kernel will cache your index files. On Tue, Feb 23, 2021 at 1:45 PM wrote: > Ok, but how is this MMapDirectory used then? > > Best regards > > > On 2/23/21 7:03 AM, Robert Muir wrote: > > > > > > On Tue, Feb 23

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Thanks but then how will MMapDirectory help gain speedup? i will try tmpfs and see what happens. i was expecting to get on order of magnitude of speedup from already very fast on disk Lucene indexes. So i was expecting really really really fast response with MMapDirectory. Thanks On 2/23/21

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

speedup over what? You are probably already using MMapDirectory (it is the default). So I don't know what you are trying to achieve, but giving lots of memory to your java process is not going to help. If you just want to prevent the first few queries to a fresh cold machine instance from being sl

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Thanks, but each different query i see some slowdown (not much though) with MMapDirectory and FSDirectory, though. It is a little bit more with FSdirectory. So, MMapDirectory is slightly better in that, too: ie, cold start. What i want to achieve: Problem statement: base case is disk based

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

(edited previous response) Thanks, but each different query at the first run i see some slowdown (not much though) with MMapDirectory and FSDirectory wrt second, third runs (due to cold start), though. Cold start slowdown is a little bit more with FSdirectory. So, MMapDirectory is slightly

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

On Tue, Feb 23, 2021 at 4:07 PM wrote: > What i want to achieve: Problem statement: > > base case is disk based Lucene index with FSDirectory > > speedup case was supposed to be in memory Lucene index with MMapDirectory > On 64-bit systems, FSDirectory just invokes MMapDirectory already. So you d

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Thanks again, Robert. Could you please explain "preload"? Which functionality is that? we discussed in this thread before about a preload. Is there a Lucene url / site that i can look at for preload? Thanks for the explanations. This thread will be useful for many folks i believe. Best regar

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

The preload isn't magical. It only "reads in the whole file" to get it cached, same as if you did that yourself with 'cat' or 'dd'. It "warms" the file. It just does this in an efficient way at the low level to make the warming itself efficient. It madvise()s kernel to announce some read-ahead and

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

So, just cat will do this. Thanks From: Robert Muir Sent: Tuesday, February 23, 2021 4:45 PM To: Baris Kazar Cc: java-user Subject: Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory) The preload isn't magical. It only "reads in the whole

Probably found bug in GraphTokenStreamFiniteStrings

Hi everyone, I faced with some exceptions in my production service based on Lucene, after some investigation I have found the problem and build minimal example as test for GraphTokenStreamFiniteStrings (you can add this into TestGraphTokenStreamFiniteStrings):