[
https://issues.apache.org/jira/browse/OPENNLP-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798226#comment-17798226
]
ASF GitHub Bot commented on OPENNLP-421:
----------------------------------------
rzo1 commented on PR #568:
URL: https://github.com/apache/opennlp/pull/568#issuecomment-1860649042
Here is the other benchmark:
```bash
Benchmark
(internerClazz) (size) Mode Cnt Score Error Units
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.CHMStringDeduplicator 1 thrpt 25
45717169,968 ± 4894532,156 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.CHMStringDeduplicator 100 thrpt 25
544627,570 ± 9900,671 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.CHMStringDeduplicator 10000 thrpt 25
3876,808 ± 211,760 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.CHMStringDeduplicator 1000000 thrpt 25
11,190 ± 1,009 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.CHMStringInterner 1 thrpt 25 43774046,611 ±
11369890,960 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.CHMStringInterner 100 thrpt 25 557563,692 ±
12776,109 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.CHMStringInterner 10000 thrpt 25 2951,642 ±
189,985 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.CHMStringInterner 1000000 thrpt 25 8,232 ±
0,847 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.HMStringInterner 1 thrpt 25 63103971,334 ±
1958932,641 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.HMStringInterner 100 thrpt 25 854268,631 ±
22261,811 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.HMStringInterner 10000 thrpt 25 3191,138 ±
241,943 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.HMStringInterner 1000000 thrpt 25 8,891 ±
1,040 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.JvmStringInterner 1 thrpt 25 8881121,516 ±
108767,585 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.JvmStringInterner 100 thrpt 25 81073,526 ±
342,867 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.JvmStringInterner 10000 thrpt 25 729,416 ±
6,782 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.JvmStringInterner 1000000 thrpt 25 2,346 ±
0,076 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.NoOpStringInterner 1 thrpt 25 132530907,915 ±
368546,226 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.NoOpStringInterner 100 thrpt 25 6446247,778 ±
14485,272 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.NoOpStringInterner 10000 thrpt 25 53441,292 ±
107,629 ops/s
StringListBenchmark.newWithArrayConstructor
opennlp.tools.util.jvm.NoOpStringInterner 1000000 thrpt 25 124,158 ±
1,921 ops/s
```
https://gist.github.com/rzo1/1e85f65ff5d152e168c58e72b2de5375
> Large dictionaries cause JVM OutOfMemoryError: PermGen due to String interning
> ------------------------------------------------------------------------------
>
> Key: OPENNLP-421
> URL: https://issues.apache.org/jira/browse/OPENNLP-421
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Affects Versions: tools-1.5.2-incubating
> Environment: RedHat 5, JDK 1.6.0_29
> Reporter: Jay Hacker
> Assignee: Richard Zowalla
> Priority: Minor
> Labels: performance
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> The current implementation of StringList:
> https://svn.apache.org/viewvc/incubator/opennlp/branches/opennlp-1.5.2-incubating/opennlp-tools/src/main/java/opennlp/tools/util/StringList.java?view=markup
>
> calls intern() on every String. Presumably this is an attempt to reduce
> memory usage for duplicate tokens. Interned Strings are stored in the JVM's
> permanent generation, which has a small fixed size (seems to be about 83 MB
> on modern 64-bit JVMs:
> [http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html]).
> Once this fills up, the JVM crashes with an OutOfMemoryError: PermGen
> space.
> The size of the PermGen can be increased with the -XX:MaxPermSize= option to
> the JVM. However, this option is non-standard and not well known, and it
> would be nice if OpenNLP worked out of the box without deep JVM tuning.
> This immediate problem could be fixed by simply not interning Strings.
> Looking at the Dictionary and DictionaryNameFinder code as a whole, however,
> there is a huge amount of room for performance improvement. Currently,
> DictionaryNameFinder.find works something like this:
> for every token in every tokenlist in the dictionary:
> copy it into a "meta dictionary" of single tokens
> for every possible subsequence of tokens in the sentence: // of which
> there are O(N^2)
> copy the sequence into a new array
> if the last token is in the "meta dictionary":
> make a StringList from the tokens
> look it up in the dictionary
> Dictionary itself is very heavyweight: it's a Set<StringListWrapper>, which
> wraps StringList, which wraps Array<String>. Every entry in the dictionary
> requires at least four allocated objects (in addition to the Strings): Array,
> StringList, StringListWrapper, and HashMap.Entry. Even contains and remove
> allocate new objects!
> From this comment in DictionaryNameFinder:
> // TODO: improve performance here
> It seems like improvements would be welcome. :) Removing some of the object
> overhead would more than make up for interning strings. Should I create a
> new Jira ticket to propose a more efficient design?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
