Re: StandardTokenizer and split tokens

2012-06-22 Thread Mansour Al Akeel
I found the main issue. I was using ByteRef without the length. This fixed the problem. String word = new String(ref.bytes,ref.offset,ref.length); Thank you. On Fri, Jun 22, 2012 at 6:26 PM, Mansour Al Akeel wrote: > Hello all, > > I am tying to write a simple autosugg

RE: any good idea for loading fields into memory?

2012-06-22 Thread Li Li
30 ms is important. for one reason, I am reconstructing a project and integrating everything into lucene. so it should be as fast as before. the second reason is the matching of lucene is just a small part, there are many other steps. also, 40ms is the average time, for some query which matching 10

StandardTokenizer and split tokens

2012-06-22 Thread Mansour Al Akeel
Hello all, I am tying to write a simple autosuggest functionality. I was looking at some auto suggest code, and came over this post http://stackoverflow.com/questions/120180/how-to-do-query-auto-completion-suggestions-in-lucene I have been stuck with the some strange words, trying to see how they

RE: any good idea for loading fields into memory?

2012-06-22 Thread Paul Hill
10 ms vs 40 ms. I'd say so what? Is your overall time noticeably effected by this 30 ms gain? Does the end user notice this 30 ms gain? Where is the time going? Just getting the hits? Getting all documents? Building result set as your app uses it? If it is the hits, have you considered sea

Re: any good idea for loading fields into memory?

2012-06-22 Thread Li Li
using RAMDIRECTORY is not fast enough because field value need to be deserialized. I have compared it with hashmap. it is four times slower. our old map implementation use about 10 ms, while newer one is 40 ms. the reason is we need to return some fields of all hitted documents. the fields are not

Re: any good idea for loading fields into memory?

2012-06-22 Thread Danil Ε’ORIN
If you can afford it, you could add one additional untokenized stored field that will contain the serialized(one way or another) form of the document. Add FieldCache on top of it, and return it right away. But we are getting into the area where you basically have to keep all your documents in mem