Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

So, just cat will do this. Thanks From: Robert Muir Sent: Tuesday, February 23, 2021 4:45 PM To: Baris Kazar Cc: java-user Subject: Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory) The preload isn't magical. It only "reads in the

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

gt;> base case is disk based Lucene index with FSDirectory >> >> speedup case was supposed to be in memory Lucene index with MMapDirectory >> > On 64-bit systems, FSDirectory just invokes MMapDirectory already. So you > don't need to do anything. > > Either

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

memory Lucene index with MMapDirectory On 64-bit systems, FSDirectory just invokes MMapDirectory already. So you don't need to do anything. Either way MMapDirectory or NIOFSDirectory are doing the same thing: reading your index as a normal file and letting the operating system cache it. The MMapDirec

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

On Tue, Feb 23, 2021 at 4:07 PM wrote: > What i want to achieve: Problem statement: > > base case is disk based Lucene index with FSDirectory > > speedup case was supposed to be in memory Lucene index with MMapDirectory > On 64-bit systems, FSDirectory just invokes MMapDirecto

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

(edited previous response) Thanks, but each different query at the first run i see some slowdown (not much though) with MMapDirectory and FSDirectory wrt second, third runs (due to cold start), though. Cold start slowdown is a little bit more with FSdirectory. So, MMapDirectory is slightly

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Thanks, but each different query i see some slowdown (not much though) with MMapDirectory and FSDirectory, though. It is a little bit more with FSdirectory. So, MMapDirectory is slightly better in that, too: ie, cold start. What i want to achieve: Problem statement: base case is disk based

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

speedup over what? You are probably already using MMapDirectory (it is the default). So I don't know what you are trying to achieve, but giving lots of memory to your java process is not going to help. If you just want to prevent the first few queries to a fresh cold machine instance from being

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Thanks but then how will MMapDirectory help gain speedup? i will try tmpfs and see what happens. i was expecting to get on order of magnitude of speedup from already very fast on disk Lucene indexes. So i was expecting really really really fast response with MMapDirectory. Thanks On 2/23

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Don't give gobs of memory to your java process, you will just make things slower. The kernel will cache your index files. On Tue, Feb 23, 2021 at 1:45 PM wrote: > Ok, but how is this MMapDirectory used then? > > Best regards > > > On 2/23/21 7:03 AM, Robert Muir wrote: >

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

As Uwe suggested some time ago, tmpfs file system usage with MMapDirectory is the only way to get high speedup wrt on disk Lucene index, right? Best regards On 2/23/21 1:44 PM, baris.ka...@oracle.com wrote: Ok, but how is this MMapDirectory used then? Best regards On 2/23/21 7:03 AM

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Ok, but how is this MMapDirectory used then? Best regards On 2/23/21 7:03 AM, Robert Muir wrote: On Tue, Feb 23, 2021 at 2:30 AM <mailto:baris.ka...@oracle.com>> wrote: Hi,-   I tried MMapDirectory and i allocated as big as index size on my J2EE Container bu

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

On Tue, Feb 23, 2021 at 2:30 AM wrote: > Hi,- > > I tried MMapDirectory and i allocated as big as index size on my J2EE > Container but > > Don't allocate java heap memory for the index, MMapDirectory does not use java heap memory!

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Hi,-  I tried MMapDirectory and i allocated as big as index size on my J2EE Container but it only gives me at most 25% speedup and even sometimes a small amount of slowdown. How can i effectively use Lucene indexes in memory? Best regards On 12/14/20 6:35 PM, baris.ka...@oracle.com

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Thanks Robert. I think these valuable comments need to be placed on javadocs for future references. i think i am getting enough info for making a decision: i will use MMapDirectory without setPreload and i hope my index will fit into the RAM. i plan to post a blog for findings. Best

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

On Mon, Dec 14, 2020 at 1:59 PM Uwe Schindler wrote: > > Hi, > > as writer of the original bog post, here my comments: > > Yes, MMapDirectory.setPreload() is the feature mentioned in my blog post is > to load everything into memory - but that does not guarantee anything! > Still, I would not

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

his mean i should not use the constructor but instead use the open api? No that means, use MMapDirectory, it should fit your needs. If you have enough memory outside of heap in your operating system that can be used by Lucene to have all pages of the mmaped file in memory then it’s the best yo

RE: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

open index, with MMAP directory" > > > does this mean i should not use the constructor but instead use the open > api? No that means, use MMapDirectory, it should fit your needs. If you have enough memory outside of heap in your operating system that can be used by Lucene to have

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

mkA-w$ eMail: u...@thetaphi.de -Original Message- From: baris.ka...@oracle.com Sent: Sunday, December 13, 2020 10:18 PM To: java-user@lucene.apache.org Cc: BARIS KAZAR Subject: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory) Hi,- it would be nice to

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

n i should not use the constructor but instead use the open api? in other words: which way should be preferred? The example is from both during indexing and searching: /*First way: Using constructor (without setPreload) :*/ MMapDirectory dir = new MMapDirectory(Paths.get(indexDir)

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

hi.de > > > -Original Message- > > From: baris.ka...@oracle.com > > Sent: Sunday, December 13, 2020 10:18 PM > > To: java-user@lucene.apache.org > > Cc: BARIS KAZAR > > Subject: MMapDirectory vs In Memory Lucene Index (i.e., > ByteBuffersDirectory) > &g

RE: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

cle.com > Sent: Sunday, December 13, 2020 10:18 PM > To: java-user@lucene.apache.org > Cc: BARIS KAZAR > Subject: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory) > > Hi,- > > it would be nice to create a Lucene index in files and then effectively load

MMapDirectory usage during indexing and search

Hi,-  are there some examples on how to use MMapDirectory during indexing (i used the constructor to create it) and search? what are the best practices? should i repeat during search what i did during indexing for MMapDirectory i.e, use the constructor to create the MMapDirectory object

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

https://urldefense.com/v3/__https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html?m=1__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJ1O4pdIg$ MMapDirectory will not load the whole index into physical memory. Why should it do this? We just ask the operating

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

store/MMapDirectory.html__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJaN3djDw$ > >> > >> > https://urldefense.com/v3/__https://lucene.apache.org/core/8_5_2/core/org/apache/lucene/store/MMapDirectory.html__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpv

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

/MMapDirectory.html__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJhxlyzBw$ This following blog mentions about such option to run in the memory: (see the underlined sentence below) https://urldefense.com/v3/__https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

gt; > https://lucene.apache.org/core/8_5_2/core/org/apache/lucene/store/MMapDirectory.html > > This following blog mentions about such option > to run in the memory: (see the underlined sentence below) > > > https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on

MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

sentence below) https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html?m=1 MMapDirectory will not load the whole index into physical memory. Why should it do this? We just ask the operating system to map the file into address space for easy access, by no means we

Re: RamDirectory vs MemoryIndex vs MMapDirectory for In-Memory-Index

Thanks Dawid, glad I asked! Am Dienstag, den 25.09.2018, 10:46 +0200 schrieb Dawid Weiss: > Use MMapDirectory on a temporary location, Matthias. If you really > need in-memory indexes, a new Directory implementation is coming > (RAMDirectory will be deprecated, then removed), but the d

Re: RamDirectory vs MemoryIndex vs MMapDirectory for In-Memory-Index

Use MMapDirectory on a temporary location, Matthias. If you really need in-memory indexes, a new Directory implementation is coming (RAMDirectory will be deprecated, then removed), but the difference compared to MMapDirectory is typically not worth the hassle. See this issue for more discussion

RamDirectory vs MemoryIndex vs MMapDirectory for In-Memory-Index

and resource use for small documents * MMapDirectory which should outperform RamDirectory for huge indices (what is "huge?") My plan is to periodically index some properties (string codes, longs, lat/lng points) of a larger database content with Lucene for quicker lookups (compared to slow S

RE: MmapDirectory and IndexReader reuse

; From: Vladimir Kotal [mailto:vladimir.ko...@oracle.com] > Sent: Thursday, July 21, 2016 7:34 PM > To: java-user@lucene.apache.org > Subject: Re: MmapDirectory and IndexReader reuse > > On 07/18/16 05:52 PM, Uwe Schindler wrote: > > Hi, > > > > Have a separate search

Re: MmapDirectory and IndexReader reuse

On 07/21/16 20:18, Michael McCandless wrote: Can't you pass your own SearcherFactory to SearcherManager to do that? Aha ! That should be possible, thanks. v. - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org

Re: MmapDirectory and IndexReader reuse

Can't you pass your own SearcherFactory to SearcherManager to do that? Mike McCandless http://blog.mikemccandless.com On Thu, Jul 21, 2016 at 1:34 PM, Vladimir Kotal wrote: > On 07/18/16 05:52 PM, Uwe Schindler wrote: > >> Hi, >> >> Have a separate searcher manager

Re: MmapDirectory and IndexReader reuse

On 07/18/16 05:52 PM, Uwe Schindler wrote: Hi, Have a separate searcher manager for every directory. On every incoming search request, fetch the actual DirectoryReaders from the searcher managers and build a MultiReader from it. This costs nothing, as MultiReader is just a thin wrapper where

RE: MmapDirectory and IndexReader reuse

://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Vladimir Kotal [mailto:vladimir.ko...@oracle.com] > Sent: Monday, July 18, 2016 5:00 PM > To: java-user@lucene.apache.org > Subject: Re: MmapDirectory and IndexReader reuse > > On 07/15/16 12:00 PM, Uwe Sc

Re: MmapDirectory and IndexReader reuse

On 07/15/16 12:00 PM, Uwe Schindler wrote: Hi, You should keep the IndexReader open for the whole time! Otherwise there are more bottlenecks and slowdowns. If you are updating the Index, you should use SearcherManager that reopens the index reader accordingly. After updating the index you

RE: MmapDirectory and IndexReader reuse

al [mailto:vladimir.ko...@oracle.com] > Sent: Friday, July 15, 2016 11:49 AM > To: java-user@lucene.apache.org > Subject: MmapDirectory and IndexReader reuse > > > Hi all, > > when trying to identify bottlenecks in our application, I found that > each search which involves mul

MmapDirectory and IndexReader reuse

Hi all, when trying to identify bottlenecks in our application, I found that each search which involves multiple indexes is performing lots of mmap()/open() syscalls. This is a natural consequence of using MmapDirectory. So even if file system caches are properly warmed, this might add

Lucene MMapDirectory: Mapping failure

Hi All, We have multiple indexes in our linux system each of which has a decent size (occupying a few gigs). We are facing few issues while opening an IndexReader for some of those indexes. java.io.IOException: Map failed ,% STACK: sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)

RE: Lucene MMapDirectory: Mapping failure

...@thetaphi.de -Original Message- From: Rahul Kotecha [mailto:kotecha.rahul...@gmail.com] Sent: Wednesday, March 11, 2015 8:55 PM To: java-user@lucene.apache.org Subject: Lucene MMapDirectory: Mapping failure Hi All, We have multiple indexes in our linux system each of which has

Re: MMapDirectory or FSDirectory

Hi, Thank you for sharing the blog.I am using FSDirectory.open() in my program.So, I guess I am using MMapDirectory. It takes about 3 minutes when I search for a key(which is actually present in 80% of total data) in all the fields(1000) in this 1 million documents. Best Regards, Sreedevi S

RE: MMapDirectory or FSDirectory

Hi, If you use FSDirectory.open() it will automatically choose MMapDirectory on 64 bit systems. Please note, virtual memory is != physical RAM. A 64 bit machine has *always* 1 Terabyte of virtual address space available, this is unrelated to physical memory (a common misunderstanding about

MMapDirectory or FSDirectory

have virtual memory available of almost 8 GB. Is it advised to use MMapDirectory for increased performance? Many blogs suggest it doesnt bring out much performance difference. Best Regards, Sreedevi S

Re: Lucene search/count performance abrupt degradation (MMapDirectory)

We have detected the problem: the excessive(!) amount of memory allocated to Java heap. This articles helped us find the issue: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Although the article suggests the minimum-possible memory approach as a mere recommendation, too

Re: Lucene search/count performance abrupt degradation (MMapDirectory)

-mmapdirectory-on-64bit.html Although the article suggests the minimum-possible memory approach as a mere recommendation, too much RAM memory assigned to Java heap actually proved to be lethal to Lucene's IO operations, drowned by the swapping overload (indicated by the iotop tool after the search/count

Lucene search/count performance abrupt degradation (MMapDirectory)

of cores and RAM), with indexes of bigger or smaller size do not display any degradation. The whole application is run on Glassfish 3.1.2 server and any attempt to close MMapDirectory and readers (through SearcherManager), or simply undeploy/deploy the application, or stopping index writer process

MMapdirectory Configuration

Finally we are seeing great improvement once we switch to 64-bit java and MMapDirectory. Our Test run (multiple requests) used to take 26 minutes on 32-bit and is now improved to 10 minutes on 64-bit java. We load stored documents from lucene and pass the documents to a third party libray (closed

MMapdirectory

Finally we are seeing great improvement once we switch to 64-bit java and MMapDirectory. Our Test run (multiple requests) used to take 26 minutes on 32-bit and is now improved to 10 minutes on 64-bit java. We load stored documents from lucene and pass the documents to a third party libray

RE: Effectiveness MMapDirectory on NFS Mounted indexes

) and you may also need to use a different IndexFileDeleter. MMapDirectory does not behave different with NFS, but the risk of cruashing your JVM is very high. If the connection to the NFS server drops, the OS kernel will unmap the mapped file and all accesses to it will cause a segmentation

Effectiveness MMapDirectory on NFS Mounted indexes

Hi, I'm using Lucene 2.9.3 on a 64 bit machine. Many a times we are observing that the systems gets into to thrashing mode during merges. We are experimenting with using MMapDirectory. Our index is stored on NFS/CIFS mounted file shares. My question, is this MMapDirectory useful

RE: Optimum Lucene’s MMapDirectory size on 64bit OS

5:20 AM To: java-user@lucene.apache.org Subject: Re: Optimum Lucene’s MMapDirectory size on 64bit OS Thanks Uwe for the insight ! Also, is it advisable to set the lower chunk size for smaller indexes, like below or let Lucene/OS manage by itself. I am just guessing that assigning lower

Optimum Lucene’s MMapDirectory size on 64bit OS

Hi, As per the post The Generics Policeman Blog http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html , I am using the MMapDirectory for faster access(search and update operations ,mainly search) of Lucene 4.8.1 index files. I am contemplating what is the optimal maximum MMap

RE: Optimum Lucene’s MMapDirectory size on 64bit OS

[mailto:gupta.gaurav0...@gmail.com] Sent: Friday, September 26, 2014 9:12 PM To: java-user@lucene.apache.org Subject: Optimum Lucene’s MMapDirectory size on 64bit OS Hi, As per the post The Generics Policeman Blog http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on- 64bit.html , I am using

Re: Optimum Lucene’s MMapDirectory size on 64bit OS

@lucene.apache.org Subject: Optimum Lucene’s MMapDirectory size on 64bit OS Hi, As per the post The Generics Policeman Blog http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on- 64bit.html , I am using the MMapDirectory for faster access(search and update operations ,mainly search

Re: Issue with Lucene 3.6.1 and MMapDirectory

the IndexReader and another thread is currently running a query, the above can happen, because the memory mapped buffer was forcefully unmapped by the MMapDirectory. Since Lucene 3.6.0, Lucene tries its best to prevent this crash from happening, but in high concurrency cases this may fail (because

Re: Issue with Lucene 3.6.1 and MMapDirectory

this mean?] forcefully unmaps the index files), it only happens if you close it! The issue here is: If you close the IndexReader and another thread is currently running a query, the above can happen, because the memory mapped buffer was forcefully unmapped by the MMapDirectory. Since Lucene 3.6.0

RE: Issue with Lucene 3.6.1 and MMapDirectory

here is: If you close the IndexReader and another thread is currently running a query, the above can happen, because the memory mapped buffer was forcefully unmapped by the MMapDirectory. Since Lucene 3.6.0, Lucene tries its best to prevent this crash from happening, but in high concurrency cases

Issue with Lucene 3.6.1 and MMapDirectory

: at memcpy+160()@0x381aa7b060 -- Java stack -- at java/nio/DirectByteBuffer.get(DirectByteBuffer.java:294)[optimized] at org/apache/lucene/store/MMapDirectory$MMapIndexInput.readBytes(MMapDirectory.java:298)[optimized] at org/apache/lucene/store/DataInput.readBytes(DataInput.java:72

AW: Issue with Lucene 3.6.1 and MMapDirectory

/lucenes-searchermanager-simplifies.html may fit your needs? -Ursprüngliche Nachricht- Von: Liviu Matei [mailto:liviu...@gmail.com] Gesendet: Mittwoch, 14. Mai 2014 11:06 An: java-user@lucene.apache.org Betreff: Issue with Lucene 3.6.1 and MMapDirectory Hi, I am encountering

AW: Issue with Lucene 3.6.1 and MMapDirectory

: Liviu Matei [mailto:liviu...@gmail.com] Gesendet: Mittwoch, 14. Mai 2014 11:06 An: java-user@lucene.apache.org Betreff: Issue with Lucene 3.6.1 and MMapDirectory Hi, I am encountering the following issue with Lucene 3.6.1 if you could let me know if I am doing something wrong / there is a mistake I

use MMapDirectory with tmpfs?

Hi there, If I put Lucene segments on tmpfs and use MMapDirectory to access them, would the kernel be so dumb to load the files from tmpfs to another copy of file system cache before map it to the virtual address? Or it just maps tmpfs to the virtual address directly? I tend to believe it's

RE: use MMapDirectory with tmpfs?

.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de -Original Message- From: Reg [mailto:register9...@gmail.com] Sent: Tuesday, October 22, 2013 8:27 AM To: java-user@lucene.apache.org Subject: use MMapDirectory with tmpfs? Hi there, If I put Lucene

Re: use MMapDirectory with tmpfs?

-Original Message- From: Reg [mailto:register9...@gmail.com] Sent: Tuesday, October 22, 2013 8:27 AM To: java-user@lucene.apache.org Subject: use MMapDirectory with tmpfs? Hi there, If I put Lucene segments on tmpfs and use MMapDirectory to access them, would the kernel

DocValues formats hold large byte[][]s even when using MMapDirectory

Hi, I have a search application using Lucene 4.4.0 with various BinaryDocValues and SortedSetDocValues. We use MMapDirectory to help keep the Java heap small / GC pause times short and instead rely on the OS buffer cache to keep things fast, which I gather is generally considered a best

Re: DocValues formats hold large byte[][]s even when using MMapDirectory

McCandless http://blog.mikemccandless.com On Wed, Oct 2, 2013 at 2:11 PM, Steven Schlansker ste...@likeness.com wrote: Hi, I have a search application using Lucene 4.4.0 with various BinaryDocValues and SortedSetDocValues. We use MMapDirectory to help keep the Java heap small / GC pause times

Re: DocValues formats hold large byte[][]s even when using MMapDirectory

BinaryDocValues and SortedSetDocValues. We use MMapDirectory to help keep the Java heap small / GC pause times short and instead rely on the OS buffer cache to keep things fast, which I gather is generally considered a best practice around here. As our index grows, I've noticed that we

Re: DocValues formats hold large byte[][]s even when using MMapDirectory

On Wed, Oct 2, 2013 at 2:37 PM, Steven Schlansker ste...@likeness.com wrote: On Oct 2, 2013, at 11:16 AM, Michael McCandless luc...@mikemccandless.com wrote: In Lucene 4.5 (coming out any day now) we've switched by default to a mostly on disk impl for doc values. Awesome! Looking

Re: MMapDirectory performance - Are searchable field values contiguously stored in FS block?

on two fields in my 1GB index. I get very good performance when loaded the index via MmapDirectory. I attribute this performance to the Operating System File System (FS OS) cache, that keeps the most recently used FS blocks RAM resident. I would like to add 50 more fields to the index, increasing

Re: MMapDirectory performance - Are searchable field values contiguously stored in FS block?

, and that may or may not kill your performance ... depends on how fast your disks are (eg SSD seek cost is minor). Also note that this is not an MMapDirectory problem: any time your index cannot fit in RAM, no matter which directory impl you use, you'll be seeking. Considering what are my alternatives: 1

MMapDirectory performance - Are searchable field values contiguously stored in FS block?

Hi, I have a search workload that focuses on two fields in my 1GB index. I get very good performance when loaded the index via MmapDirectory. I attribute this performance to the Operating System File System (FS OS) cache, that keeps the most recently used FS blocks RAM resident. I would like

Re: MMapDirectory performance - Are searchable field values contiguously stored in FS block?

Nachum gi...@il.ibm.com wrote: Hi, I have a search workload that focuses on two fields in my 1GB index. I get very good performance when loaded the index via MmapDirectory. I attribute this performance to the Operating System File System (FS OS) cache, that keeps the most recently used FS

RE: MMapDirectory need twice more virtual memory than actually need?

MMapDirectory should only be used on 64bit machines with 64bit JVMs. This is described in the Javadocs. The additional sizes are caused by reopens and have their reason in specialties with the Java VM, described in 2.9/3.0 Javadocs of the class, please take a look. You can also enable

答复: MMapDirectory need twice mor e virtual memory than actually n eed?

[mailto:u...@thetaphi.de] 发送时间: 2010年1月30日 16:47 收件人: java-user@lucene.apache.org 主题: RE: MMapDirectory need twice more virtual memory than actually need? MMapDirectory should only be used on 64bit machines with 64bit JVMs. This is described in the Javadocs. The additional sizes are caused by reopens

答复: MMapDirectory need twice mor e virtual memory than actually n eed?

Ps I use Sun JVMs, I will try setUseUnmap(true) to see whether anon memory disappear. -邮件原件- 发件人: Uwe Schindler [mailto:u...@thetaphi.de] 发送时间: 2010年1月30日 16:47 收件人: java-user@lucene.apache.org 主题: RE: MMapDirectory need twice more virtual memory than actually need? MMapDirectory

答复: MMapDirectory need twice mor e virtual memory than actually n eed?

[mailto:u...@thetaphi.de] 发送时间: 2010年1月30日 16:47 收件人: java-user@lucene.apache.org 主题: RE: MMapDirectory need twice more virtual memory than actually need? MMapDirectory should only be used on 64bit machines with 64bit JVMs. This is described in the Javadocs. The additional sizes are caused

RE: MMapDirectory need twice more virtual memory than actually need?

I am sorry, we know nothing about how the JVM internally handles this. But we do not force it to map into memory! MMapDirectory just maps but does not call MappedByteBuffer.load(). So it is completely handled by the kernel when something is swapped in or not. I have a server with 32 GB RAM

MMapDirectory need twice more virtual memory than actually need?

Environment: 64 bit linux,memory 8G When I used pmap instruction to see virtual memory, I found two big anon memory which grows with the index file size. I had the two following pictures to show the problem, can you explain? This is why I got out of memory exception in 32 bit machine.

Re: MMapDirectory vs RAMDirectory

I was able to improve the behavior by setting the mapped ByteBuffer to null in the close method of MMapIndexInput. This seems to be a strong enough 'suggestion' to the gc, as I can see the references go away with process explorer, and the index files can be deleted, usually. Occasionally, a

Re: MMapDirectory vs RAMDirectory

Peter Keegan wrote: There is no 'unmap' method, so my understanding is that the file mapping is valid until the underlying buffer is garbage-collected. However, forcing the gc doesn't help. You're half right. The file mapping is indeed valid until the underlying buffer is garbage

MMapDirectory vs RAMDirectory

Hi, On a 64-bit platform with 30gb RAM and 8 real CPUs, should MMapDirectory or RAMDirectory provide better search performance on a 5gb index? Cheers, Michael - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional

MMapDirectory on Windows Broken?

Hi - I just tried MMapDirectory on windows, running the app I use to populate my index, and it fairly quickly dies. Does it work for anyone? Same code works fine with FSDirectory on Windows, or on Linux with MM. I get: [java] Constructing lucene index in ./test_repos [java] Canon

Re: Good MMapDirectory performance

- I read from Peter Keegan's recent postings: - The Lucene server is using MMapDirectory. I'm running - the jvm with -Xmx16000M. Peak memory usage of the jvm - on Linux is about 6GB and 7.8GB on windows. - We don't have nearly as much memory as Peter but I - wonder whether he is gaining anything

Good MMapDirectory performance

I thought I'd post some good news about MMapDirectory as the comments in the release notes are quite downbeat about its performance. In some environments MMapDirectory provides a big improvement. Our test application is an index of 11.4 million documents which are derived from MARC

Re: What is MMapDirectory?

, October 11, 2005 4:16 PM Subject: Re: What is MMapDirectory? Koji, On Sunday 09 October 2005 14:12, Koji Sekiguchi wrote: Hello, What is MMapDirectory? I've searched mailing list archive, but cannot find it. I could find the following explanation at Lucene 1.9 CHANGES.txt: 8. Add

What is MMapDirectory?

Hello, What is MMapDirectory? I've searched mailing list archive, but cannot find it. I could find the following explanation at Lucene 1.9 CHANGES.txt: 8. Add MMapDirectory, which uses nio to mmap input files. This is still somewhat slower than FSDirectory. However it uses less

What is MMapDirectory?

Hello, What is MMapDirectory? I've searched mailing list archive, but cannot find it. I could find the following explanation at Lucene 1.9 CHANGES.txt: 8. Add MMapDirectory, which uses nio to mmap input files. This is still somewhat slower than FSDirectory. However it uses less