[
https://issues.apache.org/jira/browse/OPENNLP-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101343#comment-18101343
]
Kristian Rickert commented on OPENNLP-1903:
-------------------------------------------
Found the bug: there was a memory allocation that was exponential for chains.
I fixed it in the branch and re-ran some speed tests against it - and on a 32
core machine, it will scale up to ~18x per core before the IO is saturated and
we get diminishing returns. The core logic was always right, and the fix
wasn't that bad. Doing a final push on the review before I submit a PR for
this but it is on the branch,
https://github.com/apache/opennlp/tree/OPENNLP-1903-NameFinder-Threading
> NameFinderME memory allocation prevents scaling
> -----------------------------------------------
>
> Key: OPENNLP-1903
> URL: https://issues.apache.org/jira/browse/OPENNLP-1903
> Project: OpenNLP
> Issue Type: Bug
> Components: Name Finder
> Affects Versions: 3.0.0-M5
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
>
> While trying to scale the NameFinderME I ran into a bottleneck - I'm opening
> this ticket as a placeholder so I don't lose track of the work
> After around 2-3 threads, the namefinder stopped scaling and churned.
> Results were still thread-safe, but the performance never gained even though
> more compute was being thrown at it.
> I'll throw more data in here as it comes in, but I'm running a yourkit
> profiler against it to see if it can identify the specific lines that are
> causing it.
> The end result, whatever the fix may be, should result in significant
> multi-threaded performance increase. Looking at the code, I suspect there's
> a growing memory allocation issue (not a leak though) that is causing it
> If I'm right, we can consider back-porting it to 2.x as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
