Hoss, Thank you for the detailed response. What I found weird was it seemed to take 0.09 seconds to create a RAMDirectory off a 17MB index. Suspiciously fast, but ok.
Yet, when I do a simple fuzzy search on a single field "word: someword~0.76" It was taking .35 seconds. That's a very very long time all things considered. I understand about the OS paging and such but in doing some variations of this to "throw the OS off", I still saw no difference between on-disk and RAM times. But despite that, the times are really slow. Any ideas? thanks again, Darren On Tue, 2008-08-12 at 19:55 -0700, Chris Hostetter wrote: > : On one index, I am seeing no speed change when flipping between > : RAMDirectory IndexSearcher and file system version. > > that is probably because even if you just use an FSDirectory, your OS will > cache the disk "pages" in RAM for you -- all using a RAMDirectory does for > you is garuntee that the entire index is copied into the heap you allocate > for your JVM. If you've got 16GB or RAM, and a 5GB index, and you > allocated 12GB of RAM to the JVM and read your index into a RAMDirectory, > your index will always be in RAM, no matter what other processes do on > your machine. > > If instead you only allocate 6GB of RAM to the JVM, and nothing else is > using up the rest of your RAM, the OS has plenty to load the whole index > into RAM as part of the filesystem cache once you use it -- but if another > process comes along and really needs that RAM (or if something reads a lot > of other pages of disk) your index might get bumped from the filesystem > cache, and the next few reads could be slow. > > : Creating the RAMDirectory from the on-disk index only takes 0.09 > : seconds. It appears it is not loading the data into memory, but maybe > : just the file names of the index? > > passing an FSDIrectory to the constructor of a RAMDIrectory uses the > Directory.copy() method whose source is fairly straight forward and easy > to read -- unless your index is ginormous it's not suprising that it's > "fast" particularly if it's already in the filesystem cache. > > > > > -Hoss > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
