Re: what's better for in memory searching?

2012-06-12 Thread Mikhail Khludnev
If I get it right, it's kind of per process setting swappiness. On Tue, Jun 12, 2012 at 3:57 AM, Li Li wrote: > is this method equivalent to set vm.swappiness which is global? > or it can set the swappiness for jvm process? > > On Tue, Jun 12, 2012 at 5:11 AM, Mikhail Khludnev > wrote: > > Poin

Re: what's better for in memory searching?

2012-06-11 Thread Li Li
is this method equivalent to set vm.swappiness which is global? or it can set the swappiness for jvm process? On Tue, Jun 12, 2012 at 5:11 AM, Mikhail Khludnev wrote: > Point about premature optimization makes sense for me. However some time > ago I've bookmarked potentially useful approach > htt

Re: what's better for in memory searching?

2012-06-11 Thread Mikhail Khludnev
Point about premature optimization makes sense for me. However some time ago I've bookmarked potentially useful approach http://lucene.472066.n3.nabble.com/High-response-time-after-being-idle-tp3616599p3617604.html. On Mon, Jun 11, 2012 at 3:02 PM, Toke Eskildsen wrote: > On Mon, 2012-06-11 at 11

Re: what's better for in memory searching?

2012-06-11 Thread Toke Eskildsen
On Mon, 2012-06-11 at 11:38 +0200, Li Li wrote: > yes, I need average query time less than 10 ms. The faster the better. > I have enough memory for lucene because I know there are not too much > data. there are not many modifications. every day there are about > hundreds of document update. if inde

Re: what's better for in memory searching?

2012-06-11 Thread Paul Libbrecht
Le 11 juin 2012 à 11:16, Li Li a écrit : > do you mean software RAM disk? Right. OS level. > using RAM to simulate disk? Yes. That generally makes a disk which is boost fast in reading and writing. > How to deal with Persistence? Synchronization (slaving?). paul

Re: what's better for in memory searching?

2012-06-11 Thread Li Li
I found this. http://unix.stackexchange.com/questions/10214/per-process-swapiness-for-linux it can provide fine grained control of swapping On Mon, Jun 11, 2012 at 4:45 PM, Michael Kuhlmann wrote: > Set the swapiness to 0 to avoid memory pages being swapped to disk too > early. > > http://en.wi

Re: what's better for in memory searching?

2012-06-11 Thread Li Li
yes, I need average query time less than 10 ms. The faster the better. I have enough memory for lucene because I know there are not too much data. there are not many modifications. every day there are about hundreds of document update. if indexes are not in physical memory, then IO operations will

Re: what's better for in memory searching?

2012-06-11 Thread Michael Kuhlmann
You cannot guarantee this when you're running out of RAM. You'd have a problem then anyway. Why are you caring that much? Did you yet have performance issues? 1GB should load really fast, and both auto warming and OS cache should help a lot as well. With such an index, you usually don't need t

Re: what's better for in memory searching?

2012-06-11 Thread Li Li
I am sorry. I make a mistake. even use RAMDirectory, I can not guarantee they are not swapped out. On Mon, Jun 11, 2012 at 4:45 PM, Michael Kuhlmann wrote: > Set the swapiness to 0 to avoid memory pages being swapped to disk too > early. > > http://en.wikipedia.org/wiki/Swappiness > > -Kuli > > A

Re: what's better for in memory searching?

2012-06-11 Thread Li Li
do you mean software RAM disk? using RAM to simulate disk? How to deal with Persistence? maybe I can hack by increase RAMOutputStream.BUFFER_SIZE from 1024 to 1024*1024. it may have a waste. but I can adjust my merge policy to avoid to much segments. I will have a "big" segment and a "small" segme

Re: what's better for in memory searching?

2012-06-11 Thread Paul Libbrecht
Li Li, have you considered allocating a RAM-Disk? It's not the most flexible thing... but it's certainly close, in performance to a RAMDirectory. MMapping on that is likely to be useless but I doubt you can set it to zero. That'd need experiment. Also, doesn't caching and auto-warming provide th

Re: what's better for in memory searching?

2012-06-11 Thread Li Li
1. this setting is global, I just want my lucene searching program don't swap. for other less important programs, it can still swap. 2. do I need call MappedByteBuffer.load() explicitly? or I have to warm up the indexes to guarantee all my files are in physical memory? On Mon, Jun 11, 2012 at 4:45

Re: what's better for in memory searching?

2012-06-11 Thread Michael Kuhlmann
Set the swapiness to 0 to avoid memory pages being swapped to disk too early. http://en.wikipedia.org/wiki/Swappiness -Kuli Am 11.06.2012 10:38, schrieb Li Li: I have roughly read the codes of RAMDirectory. it use a list of 1024 byte arrays and many overheads. But as far as I know, using MMap

Re: what's better for in memory searching?

2012-06-11 Thread Li Li
I have roughly read the codes of RAMDirectory. it use a list of 1024 byte arrays and many overheads. But as far as I know, using MMapDirectory, I can't prevent the page faults. OS will swap less frequent pages out. Even if I allocate enough memory for JVM, I can guarantee all the files in the direc

Re: what's better for in memory searching?

2012-06-08 Thread Lance Norskog
Yes, use MMapDirectory. It is faster and uses memory more efficiently than RAMDirectory. This sounds wrong, but it is true. With RAMDirectory, Java has to work harder doing garbage collection. On Fri, Jun 8, 2012 at 1:30 AM, Li Li wrote: > hi all >   I want to use lucene 3.6 providing searching s