[
https://issues.apache.org/jira/browse/LUCENE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105181#comment-14105181
]
Steve Rowe commented on LUCENE-5897:
------------------------------------
I removed the buffer expansion logic in {{StandardTokenizerImpl.zzRefill()}},
and the tokenizer still functions - as I had hoped, partial match searches are
limited to the buffer size:
{code:java}
@@ -509,16 +509,6 @@
zzStartRead = 0;
}
- /* is the buffer big enough? */
- if (zzCurrentPos >= zzBuffer.length - zzFinalHighSurrogate) {
- /* if not: blow it up */
- char newBuffer[] = new char[zzBuffer.length*2];
- System.arraycopy(zzBuffer, 0, newBuffer, 0, zzBuffer.length);
- zzBuffer = newBuffer;
- zzEndRead += zzFinalHighSurrogate;
- zzFinalHighSurrogate = 0;
- }
-
/* fill the buffer with new input */
int requested = zzBuffer.length - zzEndRead;
int totalRead = 0;
{code}
and ran Robert's {{testWorthyAdversary()}} with the input length ranging from
100k to 3.2M chars, and varying the buffer size from 4k chars (the default) to
255, compared to the current implementation, where unlimited buffer expansion
is allowed (NBE = no buffer expansion; times are in seconds; Oracle Java
1.7.0_55; OS X 10.9.4):
||Input chars||current impl.||4k buff, NBE||2k buff, NBE||1k buff, NBE||255
buff, NBE||
|100k|29s|3s|1s|<1s|<1s|
|200k|136s|5s|3s|1s|<1s|
|400k|547s|11s|5s|3s|1s|
|800k|2,272s|22s|11s|5s|1s|
|1,600k|9,000s (est.)|43s|23s|11s|3s|
|3,200k|40,000s (est.)|91s|43s|22s|6s|
I didn't actually run the test against the current implementation with 1.6M and
3.2M input chars - the numbers above with (est.) after them are estimates - but
for the ones I did measure, doubling the input length roughly quadruples the
run time.
By contrast, when the buffer length is limited, doubling the input length only
doubles the run time.
When the buffer length is limited, doubling the buffer length doubles the run
time.
Based on this, I'd like to introduce a new max buffer size setter to
StandardTokenizer, which defaults to the initial buffer size. That way, by
default buffer expansion is disabled, but can be re-enabled by setting a max
buffer size larger than the initial buffer size.
I ran luceneutil's {{TestAnalyzerPerf}}, just testing {{StandardAnalyzer}}
using enwiki-20130102-lines.txt, with unpatched trunk against trunk patched to
disable buffer expansion, and with a buffer size of 255 (the default max token
size), 5 runs each:
|| ||Million tokens/sec, trunk||Million tokens/sec, patched||
|run 1|7.162|7.020|
|run 2|7.079|7.245|
|run 3|7.381|7.200|
|run 4|7.352|7.192|
|run 5|7.160|7.169|
|mean|7.227|7.166|
|stddev|0.1323|0.08589|
These are pretty noisy, but comparing the best throughput numbers, the patched
version has 1.8% lower throughput.
Based on the above, I'd also like to:
# set the initial buffer size at the max token length
# when basing the initial buffer size on the max token length, don't go above
1M or 2M chars, to guard against people specifying {{Integer.MAX_VALUE}} for
the max token length
\\
\\
and from above:
\\
\\
# add a max buffer size setter to StandardTokenizer, which defaults to the
initial buffer size.
> performance bug ("adversary") in StandardTokenizer
> --------------------------------------------------
>
> Key: LUCENE-5897
> URL: https://issues.apache.org/jira/browse/LUCENE-5897
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Robert Muir
>
> There seem to be some conditions (I don't know how rare or what conditions)
> that cause StandardTokenizer to essentially hang on input: I havent looked
> hard yet, but as its essentially a DFA I think something wierd might be going
> on.
> An easy way to reproduce is with 1MB of underscores, it will just hang
> forever.
> {code}
> public void testWorthyAdversary() throws Exception {
> char buffer[] = new char[1024 * 1024];
> Arrays.fill(buffer, '_');
> int tokenCount = 0;
> Tokenizer ts = new StandardTokenizer();
> ts.setReader(new StringReader(new String(buffer)));
> ts.reset();
> while (ts.incrementToken()) {
> tokenCount++;
> }
> ts.end();
> ts.close();
> assertEquals(0, tokenCount);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
