mawiesne commented on code in PR #1106:
URL: https://github.com/apache/opennlp/pull/1106#discussion_r3541302098
##########
opennlp-docs/src/docbkx/namefinder.xml:
##########
@@ -155,13 +155,81 @@ Span[] nameSpans = nameFinder.find(sentence);]]>
<programlisting language="java">
<![CDATA[File model = new File("/path/to/model.onnx");
File vocab = new File("/path/to/vocab.txt");
-Map<Integer, String> categories = new HashMap<>();
-String[] tokens = new String[]{"George", "Washington", "was", "president",
"of", "the", "United", "States", "."};
-NameFinderDL nameFinderDL = new NameFinderDL(model, vocab, false,
getIds2Labels());
-Span[] spans = nameFinderDL.find(tokens);]]>
+// Maps every model output index to its BIO label. This must be exhaustive
over the model's
+// output indices; an unmapped predicted index raises IllegalStateException
during find().
+Map<Integer, String> ids2Labels = new HashMap<>();
+ids2Labels.put(0, "O");
+ids2Labels.put(1, "B-PER");
+ids2Labels.put(2, "I-PER");
+ids2Labels.put(3, "B-ORG");
+ids2Labels.put(4, "I-ORG");
+ids2Labels.put(5, "B-LOC");
+ids2Labels.put(6, "I-LOC");
+ids2Labels.put(7, "B-MISC");
+ids2Labels.put(8, "I-MISC");
+SentenceDetector sentenceDetector =
+ new SentenceDetectorME(new SentenceModel(new
File("/path/to/en-sent.bin")));
Review Comment:
We should encourage users to use the more modern UD-based model files
instead of the "ancient" ones, that is "en-sent.bin".
Can you rewrite this to make sure `opennlp-en-ud-ewt-sentence-1.3-2.5.4.bin`
is instead mentioned here and elsewhere in the set of changes?
See for reference: https://opennlp.apache.org/models.html
##########
opennlp-docs/src/docbkx/namefinder.xml:
##########
@@ -155,13 +155,81 @@ Span[] nameSpans = nameFinder.find(sentence);]]>
<programlisting language="java">
<![CDATA[File model = new File("/path/to/model.onnx");
File vocab = new File("/path/to/vocab.txt");
-Map<Integer, String> categories = new HashMap<>();
-String[] tokens = new String[]{"George", "Washington", "was", "president",
"of", "the", "United", "States", "."};
-NameFinderDL nameFinderDL = new NameFinderDL(model, vocab, false,
getIds2Labels());
-Span[] spans = nameFinderDL.find(tokens);]]>
+// Maps every model output index to its BIO label. This must be exhaustive
over the model's
+// output indices; an unmapped predicted index raises IllegalStateException
during find().
+Map<Integer, String> ids2Labels = new HashMap<>();
+ids2Labels.put(0, "O");
+ids2Labels.put(1, "B-PER");
+ids2Labels.put(2, "I-PER");
+ids2Labels.put(3, "B-ORG");
+ids2Labels.put(4, "I-ORG");
+ids2Labels.put(5, "B-LOC");
+ids2Labels.put(6, "I-LOC");
+ids2Labels.put(7, "B-MISC");
+ids2Labels.put(8, "I-MISC");
+SentenceDetector sentenceDetector =
+ new SentenceDetectorME(new SentenceModel(new
File("/path/to/en-sent.bin")));
+String[] tokens = {"George", "Washington", "was", "president", "of", "the",
"United", "States", "."};
+NameFinderDL nameFinderDL = new NameFinderDL(model, vocab, ids2Labels,
sentenceDetector);
+// findInOriginal returns spans in the original input's coordinates.
+Span[] spans = nameFinderDL.findInOriginal(tokens);]]>
</programlisting>
For additional examples, refer to the
<code>NameFinderDLEval</code> class.
</para>
+ <para>
+ Long input text is split into
overlapping chunks on the full Unicode
+ <code>White_Space</code> set before
WordPiece tokenization, so spacing such as a
+ no-break space or the CJK ideographic
space is recognized as a delimiter. After
+ inference, reconstructed entity text is
matched back to the caller's original input
+ with a Unicode-aware cursor scan (not a
regular expression), so
+ <code>Span#getCoveredText(...)</code>
returns the source text even when WordPiece
+ rejoins sub-tokens with spaces or when
the source uses non-ASCII whitespace between
+ tokens.
+ </para>
+ <para>
+ <code>findInOriginal</code> is declared
by the <code>OffsetMappingNameFinder</code>
+ capability interface that
<code>NameFinderDL</code> implements, so a caller holding a
+ plain <code>TokenNameFinder</code> can
detect the offset-mapping capability with a
+ <code>finder instanceof
OffsetMappingNameFinder</code> check (no reflection) and fall
+ back to token-index spans otherwise.
+ </para>
+ <para>
+ Optional preprocessing of the joined
input text is available through
+ <code>InferenceOptions</code> and is
off by default:
+
<code>setNormalizeWhitespace(true)</code> folds each Unicode whitespace
character to
+ an ASCII space, and
<code>setNormalizeDashes(true)</code> folds Unicode dashes to the
+ ASCII hyphen-minus. Whitespace folding
is one code point to one character and
+ preserves offsets, and so is dash
folding for Basic Multilingual Plane dashes; a
+ supplementary-plane dash shrinks from
two UTF-16 units to one and shifts later
+ offsets, which
<code>findInOriginal</code> maps back across (see
+ <xref linkend="tools.normalizer.dl"/>).
Full details, the underlying
+ <code>CharClass</code> engine, and the
broader normalization pipeline are documented
+ in <xref linkend="tools.normalizer"/>.
+ </para>
+ <programlisting language="java">
Review Comment:
This listing replicates a lot of the code from the previous example just do
demonstrate the use of InferenceOptions configuration. Please shorten / compact
the code listing, so that we keep it clear and short. Only relevant details
described in the text around the listing should be demonstrated.
##########
opennlp-docs/src/docbkx/namefinder.xml:
##########
@@ -155,13 +155,81 @@ Span[] nameSpans = nameFinder.find(sentence);]]>
<programlisting language="java">
<![CDATA[File model = new File("/path/to/model.onnx");
File vocab = new File("/path/to/vocab.txt");
-Map<Integer, String> categories = new HashMap<>();
-String[] tokens = new String[]{"George", "Washington", "was", "president",
"of", "the", "United", "States", "."};
-NameFinderDL nameFinderDL = new NameFinderDL(model, vocab, false,
getIds2Labels());
-Span[] spans = nameFinderDL.find(tokens);]]>
+// Maps every model output index to its BIO label. This must be exhaustive
over the model's
+// output indices; an unmapped predicted index raises IllegalStateException
during find().
+Map<Integer, String> ids2Labels = new HashMap<>();
+ids2Labels.put(0, "O");
+ids2Labels.put(1, "B-PER");
+ids2Labels.put(2, "I-PER");
+ids2Labels.put(3, "B-ORG");
+ids2Labels.put(4, "I-ORG");
+ids2Labels.put(5, "B-LOC");
+ids2Labels.put(6, "I-LOC");
+ids2Labels.put(7, "B-MISC");
+ids2Labels.put(8, "I-MISC");
+SentenceDetector sentenceDetector =
+ new SentenceDetectorME(new SentenceModel(new
File("/path/to/en-sent.bin")));
+String[] tokens = {"George", "Washington", "was", "president", "of", "the",
"United", "States", "."};
+NameFinderDL nameFinderDL = new NameFinderDL(model, vocab, ids2Labels,
sentenceDetector);
+// findInOriginal returns spans in the original input's coordinates.
+Span[] spans = nameFinderDL.findInOriginal(tokens);]]>
</programlisting>
For additional examples, refer to the
<code>NameFinderDLEval</code> class.
</para>
+ <para>
+ Long input text is split into
overlapping chunks on the full Unicode
+ <code>White_Space</code> set before
WordPiece tokenization, so spacing such as a
+ no-break space or the CJK ideographic
space is recognized as a delimiter. After
+ inference, reconstructed entity text is
matched back to the caller's original input
+ with a Unicode-aware cursor scan (not a
regular expression), so
+ <code>Span#getCoveredText(...)</code>
returns the source text even when WordPiece
+ rejoins sub-tokens with spaces or when
the source uses non-ASCII whitespace between
+ tokens.
+ </para>
+ <para>
+ <code>findInOriginal</code> is declared
by the <code>OffsetMappingNameFinder</code>
+ capability interface that
<code>NameFinderDL</code> implements, so a caller holding a
+ plain <code>TokenNameFinder</code> can
detect the offset-mapping capability with a
+ <code>finder instanceof
OffsetMappingNameFinder</code> check (no reflection) and fall
+ back to token-index spans otherwise.
+ </para>
+ <para>
+ Optional preprocessing of the joined
input text is available through
+ <code>InferenceOptions</code> and is
off by default:
+
<code>setNormalizeWhitespace(true)</code> folds each Unicode whitespace
character to
+ an ASCII space, and
<code>setNormalizeDashes(true)</code> folds Unicode dashes to the
+ ASCII hyphen-minus. Whitespace folding
is one code point to one character and
+ preserves offsets, and so is dash
folding for Basic Multilingual Plane dashes; a
+ supplementary-plane dash shrinks from
two UTF-16 units to one and shifts later
+ offsets, which
<code>findInOriginal</code> maps back across (see
+ <xref linkend="tools.normalizer.dl"/>).
Full details, the underlying
+ <code>CharClass</code> engine, and the
broader normalization pipeline are documented
+ in <xref linkend="tools.normalizer"/>.
+ </para>
+ <programlisting language="java">
+<![CDATA[File model = new File("/path/to/model.onnx");
+File vocab = new File("/path/to/vocab.txt");
+Map<Integer, String> ids2Labels = new HashMap<>(); // the model's BIO labels
+ids2Labels.put(0, "O");
+ids2Labels.put(1, "B-PER");
+ids2Labels.put(2, "I-PER");
+ids2Labels.put(3, "B-ORG");
+ids2Labels.put(4, "I-ORG");
+ids2Labels.put(5, "B-LOC");
+ids2Labels.put(6, "I-LOC");
+ids2Labels.put(7, "B-MISC");
+ids2Labels.put(8, "I-MISC");
+SentenceDetector sentenceDetector =
+ new SentenceDetectorME(new SentenceModel(new
File("/path/to/en-sent.bin")));
Review Comment:
see previous comment in line 171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
