[
https://issues.apache.org/jira/browse/OPENNLP-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1903:
------------------------------------
Description:
While trying to scale the NameFinderME I ran into a bottleneck - I'm opening
this ticket as a placeholder so I don't lose track of the work
After around 2-3 threads, the namefinder stopped scaling and churned. Results
were still thread-safe, but the performance never gained even though more
compute was being thrown at it.
I'll throw more data in here as it comes in, but I'm running a yourkit profiler
against it to see if it can identify the specific lines that are causing it.
The end result, whatever the fix may be, should result in significant
multi-threaded performance increase. Looking at the code, I suspect there's a
growing memory allocation issue (not a leak though) that is causing it
If I'm right, we can consider back-porting it to 2.x as well.
h2. Further analyses:
{{BeamSearch.bestSequences}} performs O(beamSize × numOutcomes × n²) element
copies per document (object count is O(beam × numOutcomes × n)):
# Every candidate expansion ({{{}new Sequence(top, out, p){}}}) copies the
parent's entire {{outcomes}} and {{probs}} lists ({{{}Sequence.java{}}}).
# Every pop copies them again via {{Sequence.getOutcomes()}}
({{{}List.copyOf{}}}, added in OPENNLP-1340) followed by {{{}toArray(){}}}.
For a 215-token chunk with beam 3 and 3 outcomes that is ~550k element copies
per document per model. This churn is invisible single-threaded but saturates
memory write bandwidth under concurrency: throughput plateaus at ~5.8x on a
24-core run regardless of offered concurrency (12 vs 32 threads yield the same
docs/s), and splitting work across independent processes degrades per-core
efficiency by ~55% — the contended resource is below the process (the memory
bus), not any lock (lock profiling shows zero contention; GC pauses are ~2 ms).
The copying dates to the original 2010 beam-search import; OPENNLP-1340 (2023)
added the {{List.copyOf}} on top. It only becomes a _scaling_ ceiling now that
OPENNLP-1816 makes shared ME instances across threads practical.
was:
While trying to scale the NameFinderME I ran into a bottleneck - I'm opening
this ticket as a placeholder so I don't lose track of the work
After around 2-3 threads, the namefinder stopped scaling and churned. Results
were still thread-safe, but the performance never gained even though more
compute was being thrown at it.
I'll throw more data in here as it comes in, but I'm running a yourkit profiler
against it to see if it can identify the specific lines that are causing it.
The end result, whatever the fix may be, should result in significant
multi-threaded performance increase. Looking at the code, I suspect there's a
growing memory allocation issue (not a leak though) that is causing it
If I'm right, we can consider back-porting it to 2.x as well.
> Replace per-candidate Sequence copies with chain nodes in BeamSearch
> --------------------------------------------------------------------
>
> Key: OPENNLP-1903
> URL: https://issues.apache.org/jira/browse/OPENNLP-1903
> Project: OpenNLP
> Issue Type: Improvement
> Components: Name Finder
> Affects Versions: 3.0.0-M5
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M6
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> While trying to scale the NameFinderME I ran into a bottleneck - I'm opening
> this ticket as a placeholder so I don't lose track of the work
> After around 2-3 threads, the namefinder stopped scaling and churned.
> Results were still thread-safe, but the performance never gained even though
> more compute was being thrown at it.
> I'll throw more data in here as it comes in, but I'm running a yourkit
> profiler against it to see if it can identify the specific lines that are
> causing it.
> The end result, whatever the fix may be, should result in significant
> multi-threaded performance increase. Looking at the code, I suspect there's
> a growing memory allocation issue (not a leak though) that is causing it
> If I'm right, we can consider back-porting it to 2.x as well.
> h2. Further analyses:
> {{BeamSearch.bestSequences}} performs O(beamSize × numOutcomes × n²) element
> copies per document (object count is O(beam × numOutcomes × n)):
> # Every candidate expansion ({{{}new Sequence(top, out, p){}}}) copies the
> parent's entire {{outcomes}} and {{probs}} lists ({{{}Sequence.java{}}}).
> # Every pop copies them again via {{Sequence.getOutcomes()}}
> ({{{}List.copyOf{}}}, added in OPENNLP-1340) followed by {{{}toArray(){}}}.
> For a 215-token chunk with beam 3 and 3 outcomes that is ~550k element copies
> per document per model. This churn is invisible single-threaded but saturates
> memory write bandwidth under concurrency: throughput plateaus at ~5.8x on a
> 24-core run regardless of offered concurrency (12 vs 32 threads yield the
> same docs/s), and splitting work across independent processes degrades
> per-core efficiency by ~55% — the contended resource is below the process
> (the memory bus), not any lock (lock profiling shows zero contention; GC
> pauses are ~2 ms).
> The copying dates to the original 2010 beam-search import; OPENNLP-1340
> (2023) added the {{List.copyOf}} on top. It only becomes a _scaling_ ceiling
> now that OPENNLP-1816 makes shared ME instances across threads practical.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
