It is my understanding of memory mapped files is that the file is assigned an address range in the virtual address space, and using the MMU/paging facilities the file is mapped into that range. Java cannot work with memory pointers directly, so there is at minimum some JNI calls that are made when filling the ByteBuffer (vs. C which will can read the memory addresses directly). I believe the "direct" buffers in nio improve this somewhat, but still not to the efficiency of C.
I think the only caching performed for memory mapped files is the same that is performed for any file by the OS, i.e. when a page of the mapped file needs to be brought into main memory, it may be available in the disk cache, and will be retrieved from there rather than from disk. I agree with the soft cache... what I planned to do was change the MemoryLRUCache to have a maximum hard size, and a maximum soft size, and when blocks are evicted from the hard cache, they are put into the soft cache using soft references. The soft cache will have a maximum number of elements so that the soft cache will not grow to the maximum heap size. Robert -----Original Message----- From: Yonik Seeley [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 25, 2005 9:13 PM To: java-dev@lucene.apache.org; [EMAIL PROTECTED] Subject: Re: major searching performance improvement Looks like really great stuff Robert! > 2. I agree that creating NioFSDirectory rather than modifying FSDirectory. I > originally felt the memory mapped files would be the fastest, but it also > requires OS calls, the "caching" code is CONSIDERABLY faster, since it does > not need to do any JNI, or make OS calls. What do you mean by OS calls required by memory mapped files? I'm not 100% sure how mmap works in java, but in C the only OS calls are typically to set up and tear down the mapping. Reads from the mmaped region that are already in memory proceed at the same speed as reads from any other piece of memory. > The problem with a SoftCache and indices of that > size, is that the JVM memory consumption would still grow to the limit > before it discarded anything (which may be ideal in some cases). Soft caches aren't great, esp with apps that generate a lot of garbage. What might be better is a multi-level LRU that spills over into soft references at a certain point. -Yonik --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
