On 7/21/07, Michael McCandless <[EMAIL PROTECTED]> wrote:
>> To further improve "out of the box" performance I would really also
>> like to fix the core analyzers, when possible, to re-use a single
>> Token instance for each Token they produce. This would then mean no
>> objects are created as you step through Tokens in the TokenStream
>> ... so this would give the best performance.
>How much better I wonder? Small object allocation & reclaiming is
>supposed to be very good in current JVMs.
Sorry I cannot give you exact numbers now, but I know for sure that we decided
to take "real analysis" into separate phase that gets executed before entering
Lucene TokenStreram and Indexing due to String in Token and than do just the
simple whitespace tokenisation during indexing. And this was not just out for
fun, there was some real benefit in it.
The issue with performance here is in making transformations on tokens during
analysis (where this applies), you gave very nice example , stemming, that
itself generates new Strings, another nice example is NGram generation in
SpellChecker that generates rater huge numbers of small objects.
The simplest model, tokenize(without modifying)/index ironically also benefits
from char[] as than things go really fast in general so new String() on the
way gets noticed in profiler. While testing new indexing code from Mike, we
also changed our vanilla Tokenizer to use termBuffer and there was again
something like 10-15% boost.
It's been a while since that so I do not know exact numbers, but I learned this
many times the hard way, nothing beats char[] when it comes to text processing.
To stop bothering you people, IMHO, there is a hard work in Analyzer chain to
be done before Token gets ready for prime time in Lucene core, and this is the
place where having String overproduction hurts.
___________________________________________________________
Yahoo! Answers - Got a question? Someone out there knows the answer. Try it
now.
http://uk.answers.yahoo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
