[ 
https://issues.apache.org/jira/browse/LUCENE-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507092
 ] 

Mark Miller commented on LUCENE-937:
------------------------------------

The 15,000 calls are each on a separate document. The documents are reletivley 
small...newspapers articles from Reuters. Anything smaller would have to be 
very small.

I have again carefully tested LinkedList get VS LinkedList iterator and the 
performance is identical as far as I can tell.

I'll do more work to prove my case when I get a free moment, but just to be 
clear:

I am using small documents (15,000 different varying sized docs), measuring the 
total time and dividing by 15,000. The results show a 43% improvement using 
ArrayList(30). I will run a test will even smaller docs when I get a chance. In 
my work with a new Span based Highlighter, I need this speed or my 
implementation is slower than the old Highlighter. With this boost, my Span 
based Highlighter is actually (very)slightly faster. If you decide to keep 
things as they are I will have to roll an alternate CachingTokenFilter for my 
Highlighter (no problem of course <g>).

Perhaps it is best to just leave things as they are and if you need more 
performance on docs with more than a handful of tokens, make your own Caching 
Filter. If the common case is closer to docs the size of newspaper articles or 
larger, a 43% gain is hard to ignore. 

I will get back about the speed when using very short documents.

>>Only the pointers to the objects are contiguous, right? 
One of these days I will actually make that transition from C++ to Java <g> I 
don't know where the speed is coming from then...but its a heck of a difference.

- Mark


> Make CachingTokenFilter faster
> ------------------------------
>
>                 Key: LUCENE-937
>                 URL: https://issues.apache.org/jira/browse/LUCENE-937
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: CachingTokenFilter.patch
>
>
> The wrong data structure was used for the CachingTokenFilter. It should be an 
> ArrayList rather than a LinkedList. There is a noticeable difference in speed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Reply via email to