Re: Using Lucene for searching tokens, not storing them.

2006-04-19 Thread karl wettin
20 apr 2006 kl. 07.29 skrev karl wettin: 18 apr 2006 kl. 22.08 skrev karl wettin: After adding a couple of binary searches in well needed places (and a couple of new bugs that in a few cases affects the results) I'm now down at 1/8th of the time compared to RAMDirectory. That is really

Re: Using Lucene for searching tokens, not storing them.

2006-04-18 Thread karl wettin
17 apr 2006 kl. 08.16 skrev karl wettin: The code contains lots of things that can be optimized for both memory and CPU. Pretty sure it can be cranked down to use a fraction of the ticks spent by a RAMDirectory. I aim at 1/3. I'm not sure if you people are as amazed as me by this, so I'll

Re: Using Lucene for searching tokens, not storing them.

2006-04-18 Thread Doug Cutting
karl wettin wrote: I'm not sure if you people are as amazed as me by this, so I'll just keep posting reports until someone tells me not to. :-) Keep it up! After adding a couple of binary searches in well needed places (and a couple of new bugs that in a few cases affects the results) I'm

Re: Using Lucene for searching tokens, not storing them.

2006-04-18 Thread karl wettin
18 apr 2006 kl. 22.18 skrev Doug Cutting: Will you be able to contribute this to Apache? Of course. I'll pop it in the Jira as soon it passes all tests. If someone wants to take a look right now, let me know. Right now it's more of a branch than a couple of diffs. I might be able to

Re: Using Lucene for searching tokens, not storing them.

2006-04-17 Thread karl wettin
16 apr 2006 kl. 19.18 skrev karl wettin: For any interested party, I do this because I have a fairly small corpus with very heavy load. I think there is a lot to win by not creating new instances of what not, seeking in the file-centric Directory, parsing pseudo-UTF8, et.c. at query time.