[
https://issues.apache.org/jira/browse/OPENNLP-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126905#comment-18126905
]
ASF GitHub Bot commented on OPENNLP-1991:
-----------------------------------------
krickert commented on code in PR #1323:
URL: https://github.com/apache/opennlp/pull/1323#discussion_r4266430860
##########
opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java:
##########
@@ -544,30 +544,145 @@ public static boolean isUnicodeBlank(CharSequence input)
{
return true;
}
+ /** U+03A3, GREEK CAPITAL LETTER SIGMA, the one code point whose lowering
depends on context. */
+ private static final int GREEK_CAPITAL_SIGMA = 0x03A3;
+
+ /** U+03C2, GREEK SMALL LETTER FINAL SIGMA, the lowering of the capital at
the end of a word. */
+ private static final int GREEK_SMALL_FINAL_SIGMA = 0x03C2;
+
/**
* Converts a {@link CharSequence} to lower case, independent of the current
- * {@link java.util.Locale} via {@link Character#toLowerCase(int)} which uses
- * mapping information from the UnicodeData file.
+ * {@link java.util.Locale}. Each code point maps through
+ * {@link Character#toLowerCase(int)}, the UnicodeData mapping, so no
character
+ * expands. The one contextual rule of the Unicode
+ * <a
href="https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt";>SpecialCasing</a>
+ * data applies as well: a Greek capital sigma at the end of a word, after a
cased
+ * letter and before no further cased letter, becomes the final sigma U+03C2.
*
* @param string The {@link CharSequence} to transform.
* @return The lower-cased String.
*/
public static String toLowerCase(CharSequence string) {
- int[] cp = string.codePoints().map(Character::toLowerCase).toArray();
+ int[] cp = toLowerCaseCodePoints(string);
return new String(cp, 0, cp.length);
}
+ /**
+ * Converts a {@link CharSequence} to lower case into a {@link CharBuffer},
with the
+ * mapping of {@link #toLowerCase(CharSequence)}.
+ *
+ * @param sequence The {@link CharSequence} to transform.
+ * @return The lower-cased characters, positioned at the start.
+ */
public static CharBuffer toLowerCaseCharBuffer(CharSequence sequence) {
CharBuffer result = CharBuffer.allocate(sequence.length());
- for (int cp : sequence.codePoints().toArray()) {
- for (char c : Character.toChars(Character.toLowerCase(cp))) {
- result.append(c);
+ for (int cp : toLowerCaseCodePoints(sequence)) {
+ if (Character.isBmpCodePoint(cp)) {
+ result.put((char) cp);
+ } else {
+ result.put(Character.highSurrogate(cp));
+ result.put(Character.lowSurrogate(cp));
}
}
result.clear();
return result;
}
+ /**
+ * Lowers every code point of a sequence, applying the Final_Sigma context.
+ *
+ * @param sequence The {@link CharSequence} to transform.
+ * @return The lower-cased code points.
+ */
+ private static int[] toLowerCaseCodePoints(CharSequence sequence) {
+ final int[] source = sequence.codePoints().toArray();
+ final int[] lowered = new int[source.length];
+ for (int i = 0; i < source.length; i++) {
Review Comment:
Reverted StringUtil to main. The sigma rule now lives in
FeedforwardNetwork.normalize with a parameterized test there.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardNetwork.java:
##########
@@ -0,0 +1,1011 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.ml.ArrayMath;
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.StringUtil;
+import opennlp.tools.util.model.ArtifactSerializer;
+
+/**
+ * The weights of the feedforward transition parser: embeddings for words,
tags, and arc
+ * labels, one hidden layer with cube activation, and a transition output
layer, after
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The network holds every forward computation the parser and the trainer
share:
+ * {@link #addContribution} is the per-position hidden sum, {@link
#outputScores} the
+ * output layer, {@link #score} the complete inference pass, and
+ * {@link #logSoftmaxInPlace} the normalization of beam decoding. The trainer
writes
+ * its updates into the live arrays the accessors return; the parser never
modifies
+ * them, so a network that is not being trained can be shared between
threads.</p>
+ *
+ * <p>Each vocabulary occupies one consecutive block of embedding rows and
starts with
+ * its reserved symbols, so unknown symbols, absent positions, and the
artificial root
+ * always have a row. Words are matched after {@link #normalize(String)
normalization}.</p>
+ *
+ * @see FeedforwardDependencyModel
+ * @since 3.0.0
+ */
+final class FeedforwardNetwork {
+
+ /** The vocabulary key of every word, tag, or label the network has no
embedding row for. */
+ static final String UNKNOWN = "*UNK*";
+
+ /** The reserved word symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_WORD_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved tag symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_TAG_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved label symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_LABEL_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE};
+
+ /** The embedding rows the reserved symbols of all three vocabularies take.
*/
+ static final int RESERVED_ROWS = RESERVED_WORD_SYMBOLS.length
+ + RESERVED_TAG_SYMBOLS.length + RESERVED_LABEL_SYMBOLS.length;
+
+ /** The embedding row of the unknown symbol in each vocabulary block. */
+ private static final int UNKNOWN_OFFSET = 0;
+
+ /** The embedding row of the absent-position symbol in each vocabulary
block. */
+ private static final int NONE_OFFSET = 1;
+
+ /** The embedding row of the root symbol in the word and tag blocks. */
+ private static final int ROOT_OFFSET = 2;
+
+ private final Map<String, Integer> wordIds;
+ private final Map<String, Integer> tagIds;
+ private final Map<String, Integer> labelIds;
+
+ /** The first embedding row of each vocabulary, indexed like the
vocabularies above. */
+ private final int firstWordRow;
+ private final int firstTagRow;
+ private final int firstLabelRow;
+
+ private final String[] transitions;
+ private final Transition[] decodedTransitions;
+ private final Map<Transition, Integer> transitionIds;
+
+ private final int embeddingSize;
+ private final float[][] embeddings;
+ private final float[][] hiddenWeights;
+ private final float[] hiddenBias;
+ private final float[][] outputWeights;
+ private final float[] outputBias;
+
+ /**
+ * Assembles a network from its vocabularies and weights, copying the maps
so later
+ * changes by the caller do not reach the network.
+ *
+ * @param wordIds The word vocabulary, mapping each normalized word to its
row.
+ * @param tagIds The tag vocabulary.
+ * @param labelIds The dependency label vocabulary.
+ * @param transitions The transition inventory, indexed by output row.
+ * @param embeddingSize The embedding dimensionality.
+ * @param embeddings The embedding rows for words, tags, and labels.
+ * @param hiddenWeights The hidden layer weights.
+ * @param hiddenBias The hidden layer bias.
+ * @param outputWeights The output layer weights.
+ * @param outputBias The output layer bias.
+ * @throws IllegalArgumentException Thrown if a vocabulary lacks a reserved
symbol, the
+ * vocabularies do not partition the embedding rows into consecutive
blocks
+ * that start with the reserved symbols, a transition repeats or
does not
+ * decode, the inventory lacks SHIFT or RIGHT_ARC, or a weight
dimension does
+ * not match.
+ */
+ FeedforwardNetwork(Map<String, Integer> wordIds, Map<String, Integer> tagIds,
+ Map<String, Integer> labelIds, String[] transitions, int embeddingSize,
+ float[][] embeddings, float[][] hiddenWeights, float[] hiddenBias,
+ float[][] outputWeights, float[] outputBias) {
+ this.wordIds = Map.copyOf(wordIds);
+ this.tagIds = Map.copyOf(tagIds);
+ this.labelIds = Map.copyOf(labelIds);
+ this.transitions = transitions.clone();
+ this.embeddingSize = embeddingSize;
+ this.embeddings = embeddings;
+ this.hiddenWeights = hiddenWeights;
+ this.hiddenBias = hiddenBias;
+ this.outputWeights = outputWeights;
+ this.outputBias = outputBias;
+
+ final long rows = (long) wordIds.size() + tagIds.size() + labelIds.size();
+ if (rows != embeddings.length) {
+ throw new IllegalArgumentException("embedding matrix has " +
embeddings.length
+ + " rows for " + rows + " vocabulary entries");
+ }
+ final boolean[] taken = new boolean[embeddings.length];
+ this.firstWordRow = validateVocabulary(this.wordIds, taken, "word
vocabulary",
+ RESERVED_WORD_SYMBOLS);
+ this.firstTagRow = validateVocabulary(this.tagIds, taken, "tag vocabulary",
+ RESERVED_TAG_SYMBOLS);
+ this.firstLabelRow = validateVocabulary(this.labelIds, taken, "label
vocabulary",
+ RESERVED_LABEL_SYMBOLS);
+
+ this.decodedTransitions = new Transition[this.transitions.length];
+ this.transitionIds = new HashMap<>();
+ boolean hasShift = false;
+ boolean hasRightArc = false;
+ for (int i = 0; i < this.transitions.length; i++) {
+ try {
+ decodedTransitions[i] = Transition.decode(this.transitions[i]);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("invalid transition: " +
this.transitions[i], e);
+ }
+ if (transitionIds.put(decodedTransitions[i], i) != null) {
+ throw new IllegalArgumentException("duplicate transition: " +
this.transitions[i]);
+ }
+ hasShift |= decodedTransitions[i].type() == Transition.Type.SHIFT;
+ hasRightArc |= decodedTransitions[i].type() == Transition.Type.RIGHT_ARC;
+ }
+ if (!hasShift) {
+ throw new IllegalArgumentException("transition inventory has no SHIFT
action");
+ }
+ if (!hasRightArc) {
+ throw new IllegalArgumentException("transition inventory has no
RIGHT_ARC action");
+ }
+
+ final int inputSize = FeedforwardContext.FEATURE_COUNT * embeddingSize;
+ requireColumns(embeddings, embeddingSize, "embedding matrix");
+ requireColumns(hiddenWeights, inputSize, "hidden matrix");
+ if (hiddenBias.length != hiddenWeights.length) {
+ throw new IllegalArgumentException("hidden bias length is " +
hiddenBias.length
+ + ", expected " + hiddenWeights.length);
+ }
+ if (outputWeights.length != this.transitions.length) {
+ throw new IllegalArgumentException("output matrix row count is "
+ + outputWeights.length + ", expected " + this.transitions.length);
+ }
+ requireColumns(outputWeights, hiddenWeights.length, "output matrix");
+ if (outputBias.length != this.transitions.length) {
+ throw new IllegalArgumentException("output bias length is " +
outputBias.length
+ + ", expected " + this.transitions.length);
+ }
+ }
+
+ /**
+ * Checks that one vocabulary starts with its reserved symbols and occupies
one
+ * consecutive block of embedding rows that no other vocabulary uses.
+ *
+ * @param ids The vocabulary to check.
+ * @param taken The rows claimed so far across all vocabularies; updated in
place.
+ * @param label The vocabulary name used in error messages.
+ * @param reserved The reserved symbols, in the order of their rows.
+ * @return The first row of the block.
+ * @throws IllegalArgumentException Thrown if the block is not consecutive,
a row is
+ * already taken, or a reserved symbol is missing or misplaced.
+ */
+ private static int validateVocabulary(Map<String, Integer> ids, boolean[]
taken,
+ String label, String[] reserved) {
+ int min = Integer.MAX_VALUE;
+ int max = Integer.MIN_VALUE;
+ for (final Map.Entry<String, Integer> entry : ids.entrySet()) {
+ final int id = entry.getValue();
+ if (id < 0 || id >= taken.length) {
+ throw new IllegalArgumentException(label + " id out of range for "
+ + entry.getKey() + ": " + id);
+ }
+ if (taken[id]) {
+ throw new IllegalArgumentException("duplicate embedding id: " + id);
+ }
+ taken[id] = true;
+ min = Math.min(min, id);
+ max = Math.max(max, id);
+ }
+ if (ids.size() < reserved.length || max - min + 1 != ids.size()) {
+ throw new IllegalArgumentException(label + " must occupy one consecutive
block of "
+ + "embedding rows that starts with its reserved symbols");
+ }
+ for (int i = 0; i < reserved.length; i++) {
+ final Integer id = ids.get(reserved[i]);
+ if (id == null || id != min + i) {
+ throw new IllegalArgumentException(label + " must have " + reserved[i]
+ + " at row " + (min + i));
+ }
+ }
+ return min;
+ }
+
+ /**
+ * @param matrix The matrix to check.
+ * @param columns The required width of every row.
+ * @param label The matrix name used in error messages.
+ * @throws IllegalArgumentException Thrown if a row has another width.
+ */
+ private static void requireColumns(float[][] matrix, int columns, String
label) {
+ for (final float[] row : matrix) {
+ if (row.length != columns) {
+ throw new IllegalArgumentException(label + " column count is " +
row.length
+ + ", expected " + columns);
+ }
+ }
+ }
+
+ /**
+ * Scores every transition for a configuration described by embedding rows.
+ *
+ * @param features The embedding rows of the configuration, as produced by
+ * {@link FeedforwardContext#featureIds}. Must not be {@code
null}.
+ * @param cache The contribution cache to read, or {@code null} to compute
every
+ * contribution.
+ * @return One unnormalized score per transition, indexed like
+ * {@link #transitions()}. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code features} is {@code
null}, does not
+ * have the required length, or contains an invalid embedding row.
+ * @throws IllegalStateException Thrown if the network produces a non-finite
score.
+ */
+ double[] score(int[] features, ContributionCache cache) {
+ ParamChecks.requireNonNullArg(features, "features");
+ if (features.length != FeedforwardContext.FEATURE_COUNT) {
+ throw new IllegalArgumentException("features must contain "
+ + FeedforwardContext.FEATURE_COUNT + " embedding indices");
+ }
+ for (int i = 0; i < features.length; i++) {
+ if (features[i] < 0 || features[i] >= embeddings.length) {
+ throw new IllegalArgumentException(
+ "feature embedding out of range at index " + i + ": " +
features[i]);
+ }
+ }
+ final double[] hidden = new double[hiddenBias.length];
+ preActivations(features, cache, hidden);
+ for (int j = 0; j < hidden.length; j++) {
+ hidden[j] = hidden[j] * hidden[j] * hidden[j];
+ }
+ final double[] scores = new double[transitions.length];
+ outputScores(hidden, scores);
+ for (final double score : scores) {
+ if (!Double.isFinite(score)) {
+ throw new IllegalStateException("the model produced a non-finite
transition score");
+ }
+ }
+ return scores;
+ }
+
+ /**
+ * Computes the hidden pre-activations of a configuration: the bias plus the
+ * contribution of every feature position.
+ *
+ * @param features The embedding rows of the configuration, one per position.
+ * @param cache The contribution cache to read, or {@code null} to compute
every
+ * contribution.
+ * @param pre Receives the pre-activations; overwritten.
+ */
+ void preActivations(int[] features, ContributionCache cache, double[] pre) {
+ for (int j = 0; j < pre.length; j++) {
+ pre[j] = hiddenBias[j];
+ }
+ for (int f = 0; f < features.length; f++) {
+ final double[] cached = cache == null ? null : cache.contribution(f,
features[f]);
+ if (cached != null) {
+ for (int j = 0; j < pre.length; j++) {
+ pre[j] += cached[j];
+ }
+ } else {
+ addContribution(f, features[f], pre);
+ }
+ }
+ }
+
+ /**
+ * Adds the hidden-layer contribution of one embedding row at one feature
position.
+ *
+ * @param position The feature position.
+ * @param row The embedding row at that position.
+ * @param out The hidden vector to add to.
+ */
+ void addContribution(int position, int row, double[] out) {
+ final float[] embedding = embeddings[row];
+ final int offset = position * embeddingSize;
+ for (int j = 0; j < out.length; j++) {
+ final float[] weights = hiddenWeights[j];
+ double sum = 0.0;
+ for (int d = 0; d < embeddingSize; d++) {
+ sum += (double) weights[offset + d] * embedding[d];
+ }
+ out[j] += sum;
+ }
+ }
+
+ /**
+ * Applies the output layer to hidden activations.
+ *
+ * @param hidden The hidden activations after the cube.
+ * @param scores Receives one raw score per transition; overwritten.
+ */
+ void outputScores(double[] hidden, double[] scores) {
+ for (int o = 0; o < scores.length; o++) {
+ final float[] row = outputWeights[o];
+ double sum = outputBias[o];
+ for (int j = 0; j < hidden.length; j++) {
+ sum += row[j] * hidden[j];
+ }
+ scores[o] = sum;
+ }
+ }
+
+ /**
+ * Turns finite raw scores into log-probabilities in place. The maximum is
removed
+ * before the sum of exponentials so a large common offset does not lose the
+ * normalizer to rounding.
+ *
+ * @param scores The raw scores, overwritten with their log-softmax. Must
not be
+ * empty.
+ */
+ static void logSoftmaxInPlace(double[] scores) {
+ final double max = ArrayMath.max(scores);
+ for (int i = 0; i < scores.length; i++) {
+ scores[i] -= max;
+ }
+ final double logSum = ArrayMath.logSumOfExps(scores);
+ for (int i = 0; i < scores.length; i++) {
+ scores[i] -= logSum;
+ }
+ }
+
+ /**
+ * Resolves the tokens of a sentence to their word rows once, so that
extracting the
+ * features of every configuration only indexes.
+ *
+ * @param tokens The sentence tokens. Must not contain {@code null}.
+ * @return The word row per token. Never {@code null}.
+ */
+ int[] wordRows(String[] tokens) {
+ final int[] rows = new int[tokens.length];
+ for (int i = 0; i < tokens.length; i++) {
+ rows[i] = lookup(wordIds, normalize(tokens[i]), firstWordRow);
+ }
+ return rows;
+ }
+
+ /**
+ * Resolves the tags of a sentence to their tag rows once.
+ *
+ * @param tags The part-of-speech tags. Must not contain {@code null}.
+ * @return The tag row per tag. Never {@code null}.
+ */
+ int[] tagRows(String[] tags) {
+ final int[] rows = new int[tags.length];
+ for (int i = 0; i < tags.length; i++) {
+ rows[i] = lookup(tagIds, tags[i], firstTagRow);
+ }
+ return rows;
+ }
+
+ /**
+ * @param relation The arc label of an attached dependent, or {@code null}
for an
+ * absent position.
+ * @return The label row, falling back to the unknown row.
+ */
+ int labelRow(String relation) {
+ return lookup(labelIds, relation, firstLabelRow);
+ }
+
+ /**
+ * @return The word row standing for the artificial root.
+ */
+ int rootWordRow() {
+ return firstWordRow + ROOT_OFFSET;
+ }
+
+ /**
+ * @return The word row standing for an absent position.
+ */
+ int noneWordRow() {
+ return firstWordRow + NONE_OFFSET;
+ }
+
+ /**
+ * @return The tag row standing for the artificial root.
+ */
+ int rootTagRow() {
+ return firstTagRow + ROOT_OFFSET;
+ }
+
+ /**
+ * @return The tag row standing for an absent position.
+ */
+ int noneTagRow() {
+ return firstTagRow + NONE_OFFSET;
+ }
+
+ /**
+ * Resolves a symbol to its embedding row: absences map to the reserved
absent
+ * symbol, symbols without a row of their own to the reserved unknown symbol.
+ *
+ * @param ids The vocabulary to resolve against.
+ * @param symbol The symbol to resolve, or {@code null} for an absent
position.
+ * @param firstRow The first row of the vocabulary block.
+ * @return The embedding row of the symbol or of its fallback.
+ */
+ private static int lookup(Map<String, Integer> ids, String symbol, int
firstRow) {
+ if (symbol == null) {
+ return firstRow + NONE_OFFSET;
+ }
+ final Integer id = ids.get(symbol);
+ return id == null ? firstRow + UNKNOWN_OFFSET : id;
+ }
+
+ /**
+ * Lowercases a word symbol with {@link
StringUtil#toLowerCase(CharSequence)}, which
+ * maps per code point and applies the Greek final sigma rule; reserved
symbols and
+ * absences pass through.
+ *
+ * @param word The word to normalize. May be {@code null}.
+ * @return The vocabulary key of {@code word}, or {@code null} if {@code
word} is
+ * {@code null}.
+ */
+ static String normalize(String word) {
+ if (word == null || isReservedSymbol(word)) {
+ return word;
+ }
+ return StringUtil.toLowerCase(word);
+ }
+
+ /**
+ * @param symbol The symbol to test. May be {@code null}.
+ * @return {@code true} if {@code symbol} is one of the reserved feature
values.
+ */
+ static boolean isReservedSymbol(String symbol) {
+ // the word symbols include the tag and label symbols
+ for (final String reserved : RESERVED_WORD_SYMBOLS) {
+ if (reserved.equals(symbol)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks that a symbol fits the length the {@link Serializer} reads back.
+ *
+ * @param symbol The symbol to check.
+ * @param label The vocabulary name used in error messages.
+ * @throws IllegalArgumentException Thrown if the UTF-8 form of {@code
symbol} is
+ * longer than {@link ResourceLimits#MAX_ENTRIES} bytes.
+ */
+ static void requireStorable(String symbol, String label) {
+ // a UTF-16 unit encodes to at most three UTF-8 bytes, so short symbols
need no encoding
+ if ((long) symbol.length() * 3 <= ResourceLimits.MAX_ENTRIES) {
+ return;
+ }
+ final long bytes = symbol.getBytes(StandardCharsets.UTF_8).length;
+ if (bytes > ResourceLimits.MAX_ENTRIES) {
+ throw new IllegalArgumentException(label + " symbol length " + bytes
+ + " exceeds the limit of " + ResourceLimits.MAX_ENTRIES + " UTF-8
bytes");
+ }
+ }
+
+ /**
+ * @return The transition outcome strings by output index. Never {@code
null}.
+ */
+ String[] transitions() {
+ return transitions.clone();
+ }
+
+ /**
+ * @return The decoded transitions by output index. Never {@code null}.
+ */
+ Transition[] decodedTransitions() {
+ return decodedTransitions.clone();
+ }
+
+ /**
+ * @param transition The transition to look up.
+ * @return Its output index, or {@code -1} if the inventory lacks it.
+ */
+ int transitionIndex(Transition transition) {
+ final Integer id = transitionIds.get(transition);
+ return id == null ? -1 : id;
+ }
+
+ /**
+ * Creates an independent copy of this network: the weights are deep-copied
and the
+ * vocabularies are shared because their maps are immutable.
+ *
+ * @return A copy sharing no mutable state with this network. Never {@code
null}.
+ */
+ FeedforwardNetwork copy() {
+ return new FeedforwardNetwork(wordIds, tagIds, labelIds, transitions,
+ embeddingSize, copyOf(embeddings), copyOf(hiddenWeights),
hiddenBias.clone(),
+ copyOf(outputWeights), outputBias.clone());
+ }
+
+ /**
+ * Deep-copies a matrix, row by row.
+ *
+ * @param matrix The matrix to copy.
+ * @return A copy sharing no rows with {@code matrix}. Never {@code null}.
+ */
+ private static float[][] copyOf(float[][] matrix) {
+ final float[][] copy = new float[matrix.length][];
+ for (int r = 0; r < matrix.length; r++) {
+ copy[r] = matrix[r].clone();
+ }
+ return copy;
+ }
+
+ /**
+ * @param vocabularyRows The combined word, tag, and label row count.
+ * @param transitionCount The output layer width.
+ * @param embeddingSize The embedding dimensionality.
+ * @param hiddenSize The hidden layer width.
+ * @return The number of float values a network of that shape stores.
+ */
+ static long floatValues(long vocabularyRows, long transitionCount, long
embeddingSize,
+ long hiddenSize) {
+ final long inputSize = FeedforwardContext.FEATURE_COUNT * embeddingSize;
+ return vocabularyRows * embeddingSize + hiddenSize * inputSize + hiddenSize
+ + transitionCount * hiddenSize + transitionCount;
+ }
+
+ /**
+ * @return The immutable map from a normalized word to its embedding row.
Never {@code null}.
+ */
+ Map<String, Integer> wordIds() {
+ return wordIds;
+ }
+
+ /**
+ * @return The immutable map from a tag to its embedding row. Never {@code
null}.
+ */
+ Map<String, Integer> tagIds() {
+ return tagIds;
+ }
+
+ /**
+ * @return The immutable map from an arc label to its embedding row. Never
{@code null}.
+ */
+ Map<String, Integer> labelIds() {
+ return labelIds;
+ }
+
+ /**
+ * @return The first embedding row of the word vocabulary block.
+ */
+ int firstWordRow() {
+ return firstWordRow;
+ }
+
+ /**
+ * @return The first embedding row of the tag vocabulary block.
+ */
+ int firstTagRow() {
+ return firstTagRow;
+ }
+
+ /**
+ * @return The first embedding row of the label vocabulary block.
+ */
+ int firstLabelRow() {
+ return firstLabelRow;
+ }
+
+ /**
+ * @return The width of one embedding row.
+ */
+ int embeddingSize() {
+ return embeddingSize;
+ }
+
+ /**
+ * @return The hidden layer width.
+ */
+ int hiddenSize() {
+ return hiddenBias.length;
+ }
+
+ /**
+ * @return The live embedding matrix, one row per vocabulary entry, not a
copy: the
+ * trainer writes its updates into it. Never {@code null}.
+ */
+ float[][] embeddings() {
+ return embeddings;
+ }
+
+ /**
+ * @return The live hidden layer weights, not a copy. Never {@code null}.
+ */
+ float[][] hiddenWeights() {
+ return hiddenWeights;
+ }
+
+ /**
+ * @return The live hidden layer bias, not a copy. Never {@code null}.
+ */
+ float[] hiddenBias() {
+ return hiddenBias;
+ }
+
+ /**
+ * @return The live output layer weights, one row per transition, not a
copy. Never
+ * {@code null}.
+ */
+ float[][] outputWeights() {
+ return outputWeights;
+ }
+
+ /**
+ * @return The live output layer bias, one entry per transition, not a copy.
Never
+ * {@code null}.
+ */
+ float[] outputBias() {
+ return outputBias;
+ }
+
+ /**
+ * Reads and writes a network as a {@link FeedforwardDependencyModel}
artifact: a
+ * four-byte magic, an {@code int} format version, the three vocabularies,
the
+ * transition inventory, the embedding size, and the weight matrices and
vectors.
+ * Symbols are stored as a byte length followed by UTF-8 bytes, so their
length is
+ * not limited by {@link DataOutputStream#writeUTF}.
+ *
+ * <p>Counts read from the stream are bounded by {@link
ResourceLimits#MAX_ENTRIES},
+ * and the float values of all matrices and vectors together by
+ * {@link ResourceLimits#MAX_MATRIX_CELLS}; matrix rows are allocated one at
a time
+ * while reading.</p>
+ */
+ static final class Serializer implements
ArtifactSerializer<FeedforwardNetwork> {
+
+ /** The bytes every serialized network starts with. */
+ static final byte[] MAGIC = {'F', 'F', 'D', 'P'};
+
+ /** The format version this serializer reads and writes. */
+ static final int FORMAT_VERSION = 1;
+
+ /**
+ * {@inheritDoc}
+ *
+ * @throws InvalidFormatException Thrown if the content is not a network
in a
+ * supported version, a count or symbol length is out of range, the
+ * structure is inconsistent, a value is not finite, or the stream
ends
+ * early. Malformed UTF-8 in a symbol is replaced, not rejected.
+ */
+ @Override
+ public FeedforwardNetwork create(InputStream in) throws IOException {
+ ParamChecks.requireNonNullArg(in, "in");
+ final DataInputStream data = new DataInputStream(in);
+ try {
+ return read(data);
+ } catch (EOFException e) {
+ throw new InvalidFormatException("feedforward network ends early", e);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidFormatException(e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Reads the network body after any wrapping has been set up.
+ *
+ * @param data The input to read from.
+ * @return The network. Never {@code null}.
+ * @throws IOException Thrown if reading fails or the content is invalid.
+ */
+ private static FeedforwardNetwork read(DataInputStream data) throws
IOException {
+ final byte[] magic = new byte[MAGIC.length];
+ data.readFully(magic);
+ for (int i = 0; i < magic.length; i++) {
+ if (magic[i] != MAGIC[i]) {
+ throw new InvalidFormatException("not a feedforward dependency
network");
+ }
+ }
+ final int version = data.readInt();
+ if (version != FORMAT_VERSION) {
+ throw new InvalidFormatException("unsupported feedforward network
version: " + version);
+ }
+ final long[] remainingEntries = {ResourceLimits.MAX_ENTRIES};
+ final Map<String, Integer> wordIds = readVocabulary(data, "word
vocabulary",
+ remainingEntries);
+ final Map<String, Integer> tagIds = readVocabulary(data, "tag
vocabulary",
+ remainingEntries);
+ final Map<String, Integer> labelIds = readVocabulary(data, "label
vocabulary",
+ remainingEntries);
+ final long rows = (long) wordIds.size() + tagIds.size() +
labelIds.size();
+ final int transitionCount = readCount(data, "transition count");
+ final String[] transitions = new String[transitionCount];
+ for (int i = 0; i < transitions.length; i++) {
+ transitions[i] = readSymbol(data, "transition");
+ }
+ final int embeddingSize = readCount(data, "embedding size");
Review Comment:
Loader now rejects zero embedding size and an empty hidden matrix with
InvalidFormatException. Added both to invalidNetworks.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardNetwork.java:
##########
@@ -0,0 +1,1011 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.ml.ArrayMath;
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.StringUtil;
+import opennlp.tools.util.model.ArtifactSerializer;
+
+/**
+ * The weights of the feedforward transition parser: embeddings for words,
tags, and arc
+ * labels, one hidden layer with cube activation, and a transition output
layer, after
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The network holds every forward computation the parser and the trainer
share:
+ * {@link #addContribution} is the per-position hidden sum, {@link
#outputScores} the
+ * output layer, {@link #score} the complete inference pass, and
+ * {@link #logSoftmaxInPlace} the normalization of beam decoding. The trainer
writes
+ * its updates into the live arrays the accessors return; the parser never
modifies
+ * them, so a network that is not being trained can be shared between
threads.</p>
+ *
+ * <p>Each vocabulary occupies one consecutive block of embedding rows and
starts with
+ * its reserved symbols, so unknown symbols, absent positions, and the
artificial root
+ * always have a row. Words are matched after {@link #normalize(String)
normalization}.</p>
+ *
+ * @see FeedforwardDependencyModel
+ * @since 3.0.0
+ */
+final class FeedforwardNetwork {
+
+ /** The vocabulary key of every word, tag, or label the network has no
embedding row for. */
+ static final String UNKNOWN = "*UNK*";
+
+ /** The reserved word symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_WORD_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved tag symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_TAG_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved label symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_LABEL_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE};
+
+ /** The embedding rows the reserved symbols of all three vocabularies take.
*/
+ static final int RESERVED_ROWS = RESERVED_WORD_SYMBOLS.length
+ + RESERVED_TAG_SYMBOLS.length + RESERVED_LABEL_SYMBOLS.length;
+
+ /** The embedding row of the unknown symbol in each vocabulary block. */
+ private static final int UNKNOWN_OFFSET = 0;
+
+ /** The embedding row of the absent-position symbol in each vocabulary
block. */
+ private static final int NONE_OFFSET = 1;
+
+ /** The embedding row of the root symbol in the word and tag blocks. */
+ private static final int ROOT_OFFSET = 2;
+
+ private final Map<String, Integer> wordIds;
+ private final Map<String, Integer> tagIds;
+ private final Map<String, Integer> labelIds;
+
+ /** The first embedding row of each vocabulary, indexed like the
vocabularies above. */
+ private final int firstWordRow;
+ private final int firstTagRow;
+ private final int firstLabelRow;
+
+ private final String[] transitions;
+ private final Transition[] decodedTransitions;
+ private final Map<Transition, Integer> transitionIds;
+
+ private final int embeddingSize;
+ private final float[][] embeddings;
+ private final float[][] hiddenWeights;
+ private final float[] hiddenBias;
+ private final float[][] outputWeights;
+ private final float[] outputBias;
+
+ /**
+ * Assembles a network from its vocabularies and weights, copying the maps
so later
+ * changes by the caller do not reach the network.
+ *
+ * @param wordIds The word vocabulary, mapping each normalized word to its
row.
+ * @param tagIds The tag vocabulary.
+ * @param labelIds The dependency label vocabulary.
+ * @param transitions The transition inventory, indexed by output row.
+ * @param embeddingSize The embedding dimensionality.
+ * @param embeddings The embedding rows for words, tags, and labels.
+ * @param hiddenWeights The hidden layer weights.
+ * @param hiddenBias The hidden layer bias.
+ * @param outputWeights The output layer weights.
+ * @param outputBias The output layer bias.
+ * @throws IllegalArgumentException Thrown if a vocabulary lacks a reserved
symbol, the
+ * vocabularies do not partition the embedding rows into consecutive
blocks
+ * that start with the reserved symbols, a transition repeats or
does not
+ * decode, the inventory lacks SHIFT or RIGHT_ARC, or a weight
dimension does
+ * not match.
+ */
+ FeedforwardNetwork(Map<String, Integer> wordIds, Map<String, Integer> tagIds,
+ Map<String, Integer> labelIds, String[] transitions, int embeddingSize,
+ float[][] embeddings, float[][] hiddenWeights, float[] hiddenBias,
+ float[][] outputWeights, float[] outputBias) {
+ this.wordIds = Map.copyOf(wordIds);
+ this.tagIds = Map.copyOf(tagIds);
+ this.labelIds = Map.copyOf(labelIds);
+ this.transitions = transitions.clone();
+ this.embeddingSize = embeddingSize;
+ this.embeddings = embeddings;
+ this.hiddenWeights = hiddenWeights;
+ this.hiddenBias = hiddenBias;
+ this.outputWeights = outputWeights;
+ this.outputBias = outputBias;
+
+ final long rows = (long) wordIds.size() + tagIds.size() + labelIds.size();
+ if (rows != embeddings.length) {
+ throw new IllegalArgumentException("embedding matrix has " +
embeddings.length
+ + " rows for " + rows + " vocabulary entries");
+ }
+ final boolean[] taken = new boolean[embeddings.length];
+ this.firstWordRow = validateVocabulary(this.wordIds, taken, "word
vocabulary",
+ RESERVED_WORD_SYMBOLS);
+ this.firstTagRow = validateVocabulary(this.tagIds, taken, "tag vocabulary",
+ RESERVED_TAG_SYMBOLS);
+ this.firstLabelRow = validateVocabulary(this.labelIds, taken, "label
vocabulary",
+ RESERVED_LABEL_SYMBOLS);
+
+ this.decodedTransitions = new Transition[this.transitions.length];
+ this.transitionIds = new HashMap<>();
+ boolean hasShift = false;
+ boolean hasRightArc = false;
+ for (int i = 0; i < this.transitions.length; i++) {
+ try {
+ decodedTransitions[i] = Transition.decode(this.transitions[i]);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("invalid transition: " +
this.transitions[i], e);
+ }
+ if (transitionIds.put(decodedTransitions[i], i) != null) {
+ throw new IllegalArgumentException("duplicate transition: " +
this.transitions[i]);
+ }
+ hasShift |= decodedTransitions[i].type() == Transition.Type.SHIFT;
+ hasRightArc |= decodedTransitions[i].type() == Transition.Type.RIGHT_ARC;
+ }
+ if (!hasShift) {
+ throw new IllegalArgumentException("transition inventory has no SHIFT
action");
+ }
+ if (!hasRightArc) {
+ throw new IllegalArgumentException("transition inventory has no
RIGHT_ARC action");
+ }
+
+ final int inputSize = FeedforwardContext.FEATURE_COUNT * embeddingSize;
+ requireColumns(embeddings, embeddingSize, "embedding matrix");
+ requireColumns(hiddenWeights, inputSize, "hidden matrix");
+ if (hiddenBias.length != hiddenWeights.length) {
+ throw new IllegalArgumentException("hidden bias length is " +
hiddenBias.length
+ + ", expected " + hiddenWeights.length);
+ }
+ if (outputWeights.length != this.transitions.length) {
+ throw new IllegalArgumentException("output matrix row count is "
+ + outputWeights.length + ", expected " + this.transitions.length);
+ }
+ requireColumns(outputWeights, hiddenWeights.length, "output matrix");
+ if (outputBias.length != this.transitions.length) {
+ throw new IllegalArgumentException("output bias length is " +
outputBias.length
+ + ", expected " + this.transitions.length);
+ }
+ }
+
+ /**
+ * Checks that one vocabulary starts with its reserved symbols and occupies
one
+ * consecutive block of embedding rows that no other vocabulary uses.
+ *
+ * @param ids The vocabulary to check.
+ * @param taken The rows claimed so far across all vocabularies; updated in
place.
+ * @param label The vocabulary name used in error messages.
+ * @param reserved The reserved symbols, in the order of their rows.
+ * @return The first row of the block.
+ * @throws IllegalArgumentException Thrown if the block is not consecutive,
a row is
+ * already taken, or a reserved symbol is missing or misplaced.
+ */
+ private static int validateVocabulary(Map<String, Integer> ids, boolean[]
taken,
+ String label, String[] reserved) {
+ int min = Integer.MAX_VALUE;
+ int max = Integer.MIN_VALUE;
+ for (final Map.Entry<String, Integer> entry : ids.entrySet()) {
+ final int id = entry.getValue();
+ if (id < 0 || id >= taken.length) {
+ throw new IllegalArgumentException(label + " id out of range for "
+ + entry.getKey() + ": " + id);
+ }
+ if (taken[id]) {
+ throw new IllegalArgumentException("duplicate embedding id: " + id);
+ }
+ taken[id] = true;
+ min = Math.min(min, id);
+ max = Math.max(max, id);
+ }
+ if (ids.size() < reserved.length || max - min + 1 != ids.size()) {
+ throw new IllegalArgumentException(label + " must occupy one consecutive
block of "
+ + "embedding rows that starts with its reserved symbols");
+ }
+ for (int i = 0; i < reserved.length; i++) {
+ final Integer id = ids.get(reserved[i]);
+ if (id == null || id != min + i) {
+ throw new IllegalArgumentException(label + " must have " + reserved[i]
+ + " at row " + (min + i));
+ }
+ }
+ return min;
+ }
+
+ /**
+ * @param matrix The matrix to check.
+ * @param columns The required width of every row.
+ * @param label The matrix name used in error messages.
+ * @throws IllegalArgumentException Thrown if a row has another width.
+ */
+ private static void requireColumns(float[][] matrix, int columns, String
label) {
+ for (final float[] row : matrix) {
+ if (row.length != columns) {
+ throw new IllegalArgumentException(label + " column count is " +
row.length
+ + ", expected " + columns);
+ }
+ }
+ }
+
+ /**
+ * Scores every transition for a configuration described by embedding rows.
+ *
+ * @param features The embedding rows of the configuration, as produced by
+ * {@link FeedforwardContext#featureIds}. Must not be {@code
null}.
+ * @param cache The contribution cache to read, or {@code null} to compute
every
+ * contribution.
+ * @return One unnormalized score per transition, indexed like
+ * {@link #transitions()}. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code features} is {@code
null}, does not
+ * have the required length, or contains an invalid embedding row.
+ * @throws IllegalStateException Thrown if the network produces a non-finite
score.
+ */
+ double[] score(int[] features, ContributionCache cache) {
+ ParamChecks.requireNonNullArg(features, "features");
+ if (features.length != FeedforwardContext.FEATURE_COUNT) {
+ throw new IllegalArgumentException("features must contain "
+ + FeedforwardContext.FEATURE_COUNT + " embedding indices");
+ }
+ for (int i = 0; i < features.length; i++) {
+ if (features[i] < 0 || features[i] >= embeddings.length) {
+ throw new IllegalArgumentException(
+ "feature embedding out of range at index " + i + ": " +
features[i]);
+ }
+ }
+ final double[] hidden = new double[hiddenBias.length];
+ preActivations(features, cache, hidden);
+ for (int j = 0; j < hidden.length; j++) {
+ hidden[j] = hidden[j] * hidden[j] * hidden[j];
+ }
+ final double[] scores = new double[transitions.length];
+ outputScores(hidden, scores);
+ for (final double score : scores) {
+ if (!Double.isFinite(score)) {
+ throw new IllegalStateException("the model produced a non-finite
transition score");
+ }
+ }
+ return scores;
+ }
+
+ /**
+ * Computes the hidden pre-activations of a configuration: the bias plus the
+ * contribution of every feature position.
+ *
+ * @param features The embedding rows of the configuration, one per position.
+ * @param cache The contribution cache to read, or {@code null} to compute
every
+ * contribution.
+ * @param pre Receives the pre-activations; overwritten.
+ */
+ void preActivations(int[] features, ContributionCache cache, double[] pre) {
+ for (int j = 0; j < pre.length; j++) {
+ pre[j] = hiddenBias[j];
+ }
+ for (int f = 0; f < features.length; f++) {
+ final double[] cached = cache == null ? null : cache.contribution(f,
features[f]);
+ if (cached != null) {
+ for (int j = 0; j < pre.length; j++) {
+ pre[j] += cached[j];
+ }
+ } else {
+ addContribution(f, features[f], pre);
+ }
+ }
+ }
+
+ /**
+ * Adds the hidden-layer contribution of one embedding row at one feature
position.
+ *
+ * @param position The feature position.
+ * @param row The embedding row at that position.
+ * @param out The hidden vector to add to.
+ */
+ void addContribution(int position, int row, double[] out) {
+ final float[] embedding = embeddings[row];
+ final int offset = position * embeddingSize;
+ for (int j = 0; j < out.length; j++) {
+ final float[] weights = hiddenWeights[j];
+ double sum = 0.0;
+ for (int d = 0; d < embeddingSize; d++) {
+ sum += (double) weights[offset + d] * embedding[d];
+ }
+ out[j] += sum;
+ }
+ }
+
+ /**
+ * Applies the output layer to hidden activations.
+ *
+ * @param hidden The hidden activations after the cube.
+ * @param scores Receives one raw score per transition; overwritten.
+ */
+ void outputScores(double[] hidden, double[] scores) {
+ for (int o = 0; o < scores.length; o++) {
+ final float[] row = outputWeights[o];
+ double sum = outputBias[o];
+ for (int j = 0; j < hidden.length; j++) {
+ sum += row[j] * hidden[j];
+ }
+ scores[o] = sum;
+ }
+ }
+
+ /**
+ * Turns finite raw scores into log-probabilities in place. The maximum is
removed
+ * before the sum of exponentials so a large common offset does not lose the
+ * normalizer to rounding.
+ *
+ * @param scores The raw scores, overwritten with their log-softmax. Must
not be
+ * empty.
+ */
+ static void logSoftmaxInPlace(double[] scores) {
+ final double max = ArrayMath.max(scores);
+ for (int i = 0; i < scores.length; i++) {
+ scores[i] -= max;
+ }
+ final double logSum = ArrayMath.logSumOfExps(scores);
+ for (int i = 0; i < scores.length; i++) {
+ scores[i] -= logSum;
+ }
+ }
+
+ /**
+ * Resolves the tokens of a sentence to their word rows once, so that
extracting the
+ * features of every configuration only indexes.
+ *
+ * @param tokens The sentence tokens. Must not contain {@code null}.
+ * @return The word row per token. Never {@code null}.
+ */
+ int[] wordRows(String[] tokens) {
+ final int[] rows = new int[tokens.length];
+ for (int i = 0; i < tokens.length; i++) {
+ rows[i] = lookup(wordIds, normalize(tokens[i]), firstWordRow);
+ }
+ return rows;
+ }
+
+ /**
+ * Resolves the tags of a sentence to their tag rows once.
+ *
+ * @param tags The part-of-speech tags. Must not contain {@code null}.
+ * @return The tag row per tag. Never {@code null}.
+ */
+ int[] tagRows(String[] tags) {
+ final int[] rows = new int[tags.length];
+ for (int i = 0; i < tags.length; i++) {
+ rows[i] = lookup(tagIds, tags[i], firstTagRow);
+ }
+ return rows;
+ }
+
+ /**
+ * @param relation The arc label of an attached dependent, or {@code null}
for an
+ * absent position.
+ * @return The label row, falling back to the unknown row.
+ */
+ int labelRow(String relation) {
+ return lookup(labelIds, relation, firstLabelRow);
+ }
+
+ /**
+ * @return The word row standing for the artificial root.
+ */
+ int rootWordRow() {
+ return firstWordRow + ROOT_OFFSET;
+ }
+
+ /**
+ * @return The word row standing for an absent position.
+ */
+ int noneWordRow() {
+ return firstWordRow + NONE_OFFSET;
+ }
+
+ /**
+ * @return The tag row standing for the artificial root.
+ */
+ int rootTagRow() {
+ return firstTagRow + ROOT_OFFSET;
+ }
+
+ /**
+ * @return The tag row standing for an absent position.
+ */
+ int noneTagRow() {
+ return firstTagRow + NONE_OFFSET;
+ }
+
+ /**
+ * Resolves a symbol to its embedding row: absences map to the reserved
absent
+ * symbol, symbols without a row of their own to the reserved unknown symbol.
+ *
+ * @param ids The vocabulary to resolve against.
+ * @param symbol The symbol to resolve, or {@code null} for an absent
position.
+ * @param firstRow The first row of the vocabulary block.
+ * @return The embedding row of the symbol or of its fallback.
+ */
+ private static int lookup(Map<String, Integer> ids, String symbol, int
firstRow) {
+ if (symbol == null) {
+ return firstRow + NONE_OFFSET;
+ }
+ final Integer id = ids.get(symbol);
+ return id == null ? firstRow + UNKNOWN_OFFSET : id;
+ }
+
+ /**
+ * Lowercases a word symbol with {@link
StringUtil#toLowerCase(CharSequence)}, which
+ * maps per code point and applies the Greek final sigma rule; reserved
symbols and
+ * absences pass through.
+ *
+ * @param word The word to normalize. May be {@code null}.
+ * @return The vocabulary key of {@code word}, or {@code null} if {@code
word} is
+ * {@code null}.
+ */
+ static String normalize(String word) {
+ if (word == null || isReservedSymbol(word)) {
+ return word;
+ }
+ return StringUtil.toLowerCase(word);
+ }
+
+ /**
+ * @param symbol The symbol to test. May be {@code null}.
+ * @return {@code true} if {@code symbol} is one of the reserved feature
values.
+ */
+ static boolean isReservedSymbol(String symbol) {
+ // the word symbols include the tag and label symbols
+ for (final String reserved : RESERVED_WORD_SYMBOLS) {
+ if (reserved.equals(symbol)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks that a symbol fits the length the {@link Serializer} reads back.
+ *
+ * @param symbol The symbol to check.
+ * @param label The vocabulary name used in error messages.
+ * @throws IllegalArgumentException Thrown if the UTF-8 form of {@code
symbol} is
+ * longer than {@link ResourceLimits#MAX_ENTRIES} bytes.
+ */
+ static void requireStorable(String symbol, String label) {
+ // a UTF-16 unit encodes to at most three UTF-8 bytes, so short symbols
need no encoding
+ if ((long) symbol.length() * 3 <= ResourceLimits.MAX_ENTRIES) {
+ return;
+ }
+ final long bytes = symbol.getBytes(StandardCharsets.UTF_8).length;
+ if (bytes > ResourceLimits.MAX_ENTRIES) {
+ throw new IllegalArgumentException(label + " symbol length " + bytes
+ + " exceeds the limit of " + ResourceLimits.MAX_ENTRIES + " UTF-8
bytes");
+ }
+ }
+
+ /**
+ * @return The transition outcome strings by output index. Never {@code
null}.
+ */
+ String[] transitions() {
+ return transitions.clone();
+ }
+
+ /**
+ * @return The decoded transitions by output index. Never {@code null}.
+ */
+ Transition[] decodedTransitions() {
+ return decodedTransitions.clone();
+ }
+
+ /**
+ * @param transition The transition to look up.
+ * @return Its output index, or {@code -1} if the inventory lacks it.
+ */
+ int transitionIndex(Transition transition) {
+ final Integer id = transitionIds.get(transition);
+ return id == null ? -1 : id;
+ }
+
+ /**
+ * Creates an independent copy of this network: the weights are deep-copied
and the
+ * vocabularies are shared because their maps are immutable.
+ *
+ * @return A copy sharing no mutable state with this network. Never {@code
null}.
+ */
+ FeedforwardNetwork copy() {
+ return new FeedforwardNetwork(wordIds, tagIds, labelIds, transitions,
+ embeddingSize, copyOf(embeddings), copyOf(hiddenWeights),
hiddenBias.clone(),
+ copyOf(outputWeights), outputBias.clone());
+ }
+
+ /**
+ * Deep-copies a matrix, row by row.
+ *
+ * @param matrix The matrix to copy.
+ * @return A copy sharing no rows with {@code matrix}. Never {@code null}.
+ */
+ private static float[][] copyOf(float[][] matrix) {
+ final float[][] copy = new float[matrix.length][];
+ for (int r = 0; r < matrix.length; r++) {
+ copy[r] = matrix[r].clone();
+ }
+ return copy;
+ }
+
+ /**
+ * @param vocabularyRows The combined word, tag, and label row count.
+ * @param transitionCount The output layer width.
+ * @param embeddingSize The embedding dimensionality.
+ * @param hiddenSize The hidden layer width.
+ * @return The number of float values a network of that shape stores.
+ */
+ static long floatValues(long vocabularyRows, long transitionCount, long
embeddingSize,
+ long hiddenSize) {
+ final long inputSize = FeedforwardContext.FEATURE_COUNT * embeddingSize;
+ return vocabularyRows * embeddingSize + hiddenSize * inputSize + hiddenSize
+ + transitionCount * hiddenSize + transitionCount;
+ }
+
+ /**
+ * @return The immutable map from a normalized word to its embedding row.
Never {@code null}.
+ */
+ Map<String, Integer> wordIds() {
+ return wordIds;
+ }
+
+ /**
+ * @return The immutable map from a tag to its embedding row. Never {@code
null}.
+ */
+ Map<String, Integer> tagIds() {
+ return tagIds;
+ }
+
+ /**
+ * @return The immutable map from an arc label to its embedding row. Never
{@code null}.
+ */
+ Map<String, Integer> labelIds() {
+ return labelIds;
+ }
+
+ /**
+ * @return The first embedding row of the word vocabulary block.
+ */
+ int firstWordRow() {
+ return firstWordRow;
+ }
+
+ /**
+ * @return The first embedding row of the tag vocabulary block.
+ */
+ int firstTagRow() {
+ return firstTagRow;
+ }
+
+ /**
+ * @return The first embedding row of the label vocabulary block.
+ */
+ int firstLabelRow() {
+ return firstLabelRow;
+ }
+
+ /**
+ * @return The width of one embedding row.
+ */
+ int embeddingSize() {
+ return embeddingSize;
+ }
+
+ /**
+ * @return The hidden layer width.
+ */
+ int hiddenSize() {
+ return hiddenBias.length;
+ }
+
+ /**
+ * @return The live embedding matrix, one row per vocabulary entry, not a
copy: the
+ * trainer writes its updates into it. Never {@code null}.
+ */
+ float[][] embeddings() {
+ return embeddings;
+ }
+
+ /**
+ * @return The live hidden layer weights, not a copy. Never {@code null}.
+ */
+ float[][] hiddenWeights() {
+ return hiddenWeights;
+ }
+
+ /**
+ * @return The live hidden layer bias, not a copy. Never {@code null}.
+ */
+ float[] hiddenBias() {
+ return hiddenBias;
+ }
+
+ /**
+ * @return The live output layer weights, one row per transition, not a
copy. Never
+ * {@code null}.
+ */
+ float[][] outputWeights() {
+ return outputWeights;
+ }
+
+ /**
+ * @return The live output layer bias, one entry per transition, not a copy.
Never
+ * {@code null}.
+ */
+ float[] outputBias() {
+ return outputBias;
+ }
+
+ /**
+ * Reads and writes a network as a {@link FeedforwardDependencyModel}
artifact: a
+ * four-byte magic, an {@code int} format version, the three vocabularies,
the
+ * transition inventory, the embedding size, and the weight matrices and
vectors.
+ * Symbols are stored as a byte length followed by UTF-8 bytes, so their
length is
+ * not limited by {@link DataOutputStream#writeUTF}.
+ *
+ * <p>Counts read from the stream are bounded by {@link
ResourceLimits#MAX_ENTRIES},
+ * and the float values of all matrices and vectors together by
+ * {@link ResourceLimits#MAX_MATRIX_CELLS}; matrix rows are allocated one at
a time
+ * while reading.</p>
+ */
+ static final class Serializer implements
ArtifactSerializer<FeedforwardNetwork> {
+
+ /** The bytes every serialized network starts with. */
+ static final byte[] MAGIC = {'F', 'F', 'D', 'P'};
+
+ /** The format version this serializer reads and writes. */
+ static final int FORMAT_VERSION = 1;
+
+ /**
+ * {@inheritDoc}
+ *
+ * @throws InvalidFormatException Thrown if the content is not a network
in a
+ * supported version, a count or symbol length is out of range, the
+ * structure is inconsistent, a value is not finite, or the stream
ends
+ * early. Malformed UTF-8 in a symbol is replaced, not rejected.
+ */
+ @Override
+ public FeedforwardNetwork create(InputStream in) throws IOException {
+ ParamChecks.requireNonNullArg(in, "in");
+ final DataInputStream data = new DataInputStream(in);
+ try {
+ return read(data);
+ } catch (EOFException e) {
+ throw new InvalidFormatException("feedforward network ends early", e);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidFormatException(e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Reads the network body after any wrapping has been set up.
+ *
+ * @param data The input to read from.
+ * @return The network. Never {@code null}.
+ * @throws IOException Thrown if reading fails or the content is invalid.
+ */
+ private static FeedforwardNetwork read(DataInputStream data) throws
IOException {
+ final byte[] magic = new byte[MAGIC.length];
+ data.readFully(magic);
+ for (int i = 0; i < magic.length; i++) {
+ if (magic[i] != MAGIC[i]) {
+ throw new InvalidFormatException("not a feedforward dependency
network");
+ }
+ }
+ final int version = data.readInt();
+ if (version != FORMAT_VERSION) {
+ throw new InvalidFormatException("unsupported feedforward network
version: " + version);
+ }
+ final long[] remainingEntries = {ResourceLimits.MAX_ENTRIES};
+ final Map<String, Integer> wordIds = readVocabulary(data, "word
vocabulary",
+ remainingEntries);
+ final Map<String, Integer> tagIds = readVocabulary(data, "tag
vocabulary",
+ remainingEntries);
+ final Map<String, Integer> labelIds = readVocabulary(data, "label
vocabulary",
+ remainingEntries);
+ final long rows = (long) wordIds.size() + tagIds.size() +
labelIds.size();
+ final int transitionCount = readCount(data, "transition count");
+ final String[] transitions = new String[transitionCount];
Review Comment:
The transitions now go into a list as they are read.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyTrainer.java:
##########
@@ -0,0 +1,1123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.function.Function;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import opennlp.tools.ml.BeamSearch;
+import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.Parameters;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.TrainingParameters;
+import opennlp.tools.util.model.BaseModel;
+
+/**
+ * Trains the {@link FeedforwardDependencyModel}: oracle-derived transition
examples,
+ * minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and
inverted
+ * dropout on the hidden layer, the training recipe of
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The hyperparameters are read from {@link TrainingParameters}: the keys
of this
+ * class plus {@link Parameters#ITERATIONS_PARAM} for the number of epochs,
+ * {@link Parameters#CUTOFF_PARAM} for the minimum word frequency, and
+ * {@link BeamSearch#BEAM_SIZE_PARAMETER} for the refinement beam. Other keys
are
+ * ignored.</p>
+ *
+ * <p>Words below the frequency cutoff share the reserved unknown row, which is
+ * therefore trained only when such words exist; the unknown rows of the tag
and label
+ * vocabularies keep their random initialization. Absent template positions
share a
+ * trained padding row. Non-projective samples have no arc-standard derivation
and are
+ * skipped. Training is deterministic for a fixed {@link #SEED_PARAM}.</p>
+ *
+ * @since 3.0.0
+ */
+public final class FeedforwardDependencyTrainer {
+
+ /** The training parameter key of the embedding dimensionality; default
{@value #DEFAULT_EMBEDDING_SIZE}. */
+ public static final String EMBEDDING_SIZE_PARAM = "EmbeddingSize";
+
+ /** The training parameter key of the hidden layer width; default {@value
#DEFAULT_HIDDEN_SIZE}. */
+ public static final String HIDDEN_SIZE_PARAM = "HiddenSize";
+
+ /** The training parameter key of the minibatch size; default {@value
#DEFAULT_BATCH_SIZE}. */
+ public static final String BATCH_SIZE_PARAM = "BatchSize";
+
+ /** The training parameter key of the AdaGrad step size; default {@value
#DEFAULT_LEARNING_RATE}. */
+ public static final String LEARNING_RATE_PARAM = "LearningRate";
+
+ /** The training parameter key of the L2 penalty on dense weights; default
{@value #DEFAULT_L2}. */
+ public static final String L2_PARAM = "L2";
+
+ /** The training parameter key of the hidden dropout probability; default
{@value #DEFAULT_DROPOUT}. */
+ public static final String DROPOUT_PARAM = "Dropout";
+
+ /**
+ * The training parameter key of the random seed, read as an {@code int};
default
+ * {@value #DEFAULT_SEED}.
+ */
+ public static final String SEED_PARAM = "Seed";
+
+ /** The default embedding dimensionality. */
+ public static final int DEFAULT_EMBEDDING_SIZE = 50;
+
+ /** The default hidden layer width. */
+ public static final int DEFAULT_HIDDEN_SIZE = 200;
+
+ /** The default number of epochs. */
+ public static final int DEFAULT_EPOCHS = 10;
+
+ /** The default minibatch size. */
+ public static final int DEFAULT_BATCH_SIZE = 256;
+
+ /** The default AdaGrad step size. */
+ public static final double DEFAULT_LEARNING_RATE = 0.02;
+
+ /** The default L2 penalty. */
+ public static final double DEFAULT_L2 = 1e-8;
+
+ /** The default hidden dropout probability. */
+ public static final double DEFAULT_DROPOUT = 0.5;
+
+ /** The default minimum word frequency for an embedding row. */
+ public static final int DEFAULT_WORD_CUTOFF = 2;
+
+ /** The default random seed. */
+ public static final int DEFAULT_SEED = 17;
+
+ /** The default refinement beam size. */
+ public static final int DEFAULT_BEAM_SIZE = 4;
+
+ /** The maximum embedding width. */
+ static final int MAX_EMBEDDING_SIZE = 4_096;
+
+ /** The maximum hidden-layer width. */
+ static final int MAX_HIDDEN_SIZE = 65_536;
+
+ private static final Logger logger =
+ LoggerFactory.getLogger(FeedforwardDependencyTrainer.class);
+
+ /** The AdaGrad denominator offset that keeps the first steps finite. */
+ private static final double ADAGRAD_EPSILON = 1e-6;
+
+ /** The derivative factor of the cube activation, {@code d/dx x^3 = 3x^2}. */
+ private static final double CUBE_DERIVATIVE_FACTOR = 3.0;
+
+ /** The numerator of the Glorot uniform bound, {@code sqrt(6 / (fanIn +
fanOut))}. */
+ private static final double GLOROT_NUMERATOR = 6.0;
+
+ /** The half-width of the uniform range embeddings are drawn from. */
+ private static final double EMBEDDING_INIT_SCALE = 0.01;
+
+ /** The smallest probability the reported training loss takes the log of. */
+ private static final double LOSS_PROBABILITY_FLOOR = 1e-12;
+
+ /** SHIFT and at least one RIGHT_ARC are required to parse a sentence. */
+ private static final int MIN_TRANSITIONS = 2;
+
+ /** The smallest beam that can lose the gold derivation to another path. */
+ private static final int MIN_REFINE_BEAM_SIZE = 2;
+
+ /** Prevents construction of this utility class. */
+ private FeedforwardDependencyTrainer() {
+ }
+
+ /**
+ * The parsed hyperparameters of one training or refinement run.
+ *
+ * @param embeddingSize The embedding dimensionality. Must be positive.
+ * @param hiddenSize The hidden layer width. Must be positive.
+ * @param epochs The number of passes over the examples. Must be positive.
+ * @param batchSize The minibatch size. Must be positive.
+ * @param learningRate The AdaGrad step size. Must be positive.
+ * @param l2 The L2 penalty applied to the dense weights. Must not be
negative.
+ * @param dropout The hidden dropout probability. Must be in {@code [0, 1)}.
+ * @param wordCutoff The minimum frequency for a word to get its own
embedding. Must
+ * not be negative.
+ * @param seed The random seed making a run reproducible.
+ * @param beamSize The refinement beam width; {@link #refine} requires at
least 2.
+ */
+ record Settings(int embeddingSize, int hiddenSize, int epochs, int batchSize,
+ double learningRate, double l2, double dropout, int wordCutoff, long
seed,
+ int beamSize) {
+
+ /**
+ * Validates the hyperparameters.
+ *
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range or the smallest network of this shape exceeds
+ * {@link ResourceLimits#MAX_MATRIX_CELLS} float values.
+ */
+ Settings {
+ ParamChecks.requirePositive(embeddingSize, EMBEDDING_SIZE_PARAM);
+ ParamChecks.requirePositive(hiddenSize, HIDDEN_SIZE_PARAM);
+ ParamChecks.requirePositive(epochs, Parameters.ITERATIONS_PARAM);
+ ParamChecks.requirePositive(batchSize, BATCH_SIZE_PARAM);
+ if (embeddingSize > MAX_EMBEDDING_SIZE) {
+ throw new IllegalArgumentException(EMBEDDING_SIZE_PARAM + " exceeds "
+ MAX_EMBEDDING_SIZE
+ + ": " + embeddingSize);
+ }
+ if (hiddenSize > MAX_HIDDEN_SIZE) {
+ throw new IllegalArgumentException(HIDDEN_SIZE_PARAM + " exceeds " +
MAX_HIDDEN_SIZE
+ + ": " + hiddenSize);
+ }
+ requireWithinCellLimit(FeedforwardNetwork.RESERVED_ROWS, MIN_TRANSITIONS,
+ embeddingSize, hiddenSize);
+ ParamChecks.requirePositive(learningRate, LEARNING_RATE_PARAM);
+ ParamChecks.requireNonNegative(ParamChecks.requireFinite(l2, L2_PARAM),
L2_PARAM);
+ if (!(dropout >= 0.0 && dropout < 1.0)) {
+ throw new IllegalArgumentException(DROPOUT_PARAM + " must be in [0,
1): " + dropout);
+ }
+ ParamChecks.requireNonNegative(wordCutoff, Parameters.CUTOFF_PARAM);
+ }
+
+ /**
+ * Reads the hyperparameters, substituting the defaults for absent keys.
+ *
+ * @param parameters The training parameters. Must not be {@code null}.
+ * @return The parsed settings. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range.
+ */
+ static Settings from(TrainingParameters parameters) {
+ return new Settings(
+ parameters.getIntParameter(EMBEDDING_SIZE_PARAM,
DEFAULT_EMBEDDING_SIZE),
+ parameters.getIntParameter(HIDDEN_SIZE_PARAM, DEFAULT_HIDDEN_SIZE),
+ parameters.getIntParameter(Parameters.ITERATIONS_PARAM,
DEFAULT_EPOCHS),
+ parameters.getIntParameter(BATCH_SIZE_PARAM, DEFAULT_BATCH_SIZE),
+ parameters.getDoubleParameter(LEARNING_RATE_PARAM,
DEFAULT_LEARNING_RATE),
+ parameters.getDoubleParameter(L2_PARAM, DEFAULT_L2),
+ parameters.getDoubleParameter(DROPOUT_PARAM, DEFAULT_DROPOUT),
+ parameters.getIntParameter(Parameters.CUTOFF_PARAM,
DEFAULT_WORD_CUTOFF),
+ parameters.getIntParameter(SEED_PARAM, DEFAULT_SEED),
+ parameters.getIntParameter(BeamSearch.BEAM_SIZE_PARAMETER,
DEFAULT_BEAM_SIZE));
+ }
+ }
+
+ /**
+ * Trains a model from dependency samples.
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}. Must not be {@code
null}.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}, a
+ * hyperparameter is out of range, no trainable example can be
derived from
+ * the samples, a vocabulary exceeds {@link
ResourceLimits#MAX_ENTRIES}
+ * entries or a symbol as many UTF-8 bytes, or the network would
store more
+ * than {@link ResourceLimits#MAX_MATRIX_CELLS} values.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel train(String languageCode,
+ ObjectStream<DependencySample> samples, TrainingParameters parameters)
+ throws IOException {
+ return train(languageCode, samples, parameters, null);
+ }
+
+ /**
+ * Trains a model from dependency samples, seeding word embeddings from a
pretrained
+ * source.
+ *
+ * <p>The provider is consulted once per vocabulary word during
initialization. Words
+ * it returns {@code null} for keep their random initialization. All
embeddings remain
+ * trainable and are stored in the model.</p>
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}. Must not be {@code
null}.
+ * @param pretrained Maps a normalized word to its pretrained vector of
exactly
+ * {@link #EMBEDDING_SIZE_PARAM} dimensions, or {@code
null} for
+ * unknown words. May be {@code null} to disable seeding.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if {@code languageCode}, {@code
samples},
+ * or {@code parameters} is {@code null}, a hyperparameter is out of
range, no
+ * trainable example can be derived, a vocabulary exceeds
+ * {@link ResourceLimits#MAX_ENTRIES} entries or a symbol as many
UTF-8 bytes,
+ * the network would store more than {@link
ResourceLimits#MAX_MATRIX_CELLS}
+ * values, or a pretrained vector has the wrong dimensionality or a
non-finite
+ * value.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel train(String languageCode,
+ ObjectStream<DependencySample> samples, TrainingParameters parameters,
+ Function<String, float[]> pretrained) throws IOException {
+ ParamChecks.requireNonNullArg(languageCode, "languageCode");
+ ParamChecks.requireNonNullArg(samples, "samples");
+ ParamChecks.requireNonNullArg(parameters, "parameters");
+ final Settings settings = Settings.from(parameters);
+ final List<Derivation> derivations = derive(samples);
+ if (derivations.isEmpty()) {
+ throw new IllegalArgumentException("no trainable examples in the
samples");
+ }
+ final FeedforwardNetwork network = initialize(derivations, settings);
+ if (pretrained != null) {
+ seed(network, pretrained, settings);
+ }
+ int exampleCount = 0;
+ for (final Derivation derivation : derivations) {
+ exampleCount += derivation.oracle().size();
+ }
+ final int[][] features = new int[exampleCount][];
+ final int[] gold = new int[exampleCount];
+ collectExamples(derivations, network, features, gold);
+ optimize(network, features, gold, settings);
+ return new FeedforwardDependencyModel(languageCode, network,
manifestEntries(settings));
+ }
+
+ /**
+ * Refines a locally trained model with beam search and the early-update
method of
+ * <a href="https://aclanthology.org/P04-1015/";>Collins and Roark
(2004)</a>. The loss
+ * is conditional likelihood over candidate paths scored by summed
log-probabilities.
+ *
+ * <p>Refinement trains a copy with per-sentence AdaGrad steps and no
dropout.
+ * {@link Parameters#ITERATIONS_PARAM} sets the number of passes and
+ * {@link BeamSearch#BEAM_SIZE_PARAMETER} the beam width, which is also the
beam size
+ * to parse the result with. The transition inventory of {@code model} is not
+ * extended.</p>
+ *
+ * @param model The locally trained model. Not modified. Must not be {@code
null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}; the epochs, learning
rate, L2
+ * penalty, seed, and beam size apply. Must not be {@code
null}.
+ * @return A new refined model, distinct from {@code model}. Never {@code
null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}, a
+ * hyperparameter is out of range, the beam size is below 2, no
trainable
+ * sample can be derived, or a sample requires a transition {@code
model}
+ * does not know.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel refine(FeedforwardDependencyModel
model,
+ ObjectStream<DependencySample> samples, TrainingParameters parameters)
+ throws IOException {
+ ParamChecks.requireNonNullArg(model, "model");
+ ParamChecks.requireNonNullArg(samples, "samples");
+ ParamChecks.requireNonNullArg(parameters, "parameters");
+ final Settings settings = Settings.from(parameters);
+ if (settings.beamSize() < MIN_REFINE_BEAM_SIZE) {
+ throw new IllegalArgumentException(BeamSearch.BEAM_SIZE_PARAMETER
+ + " must be at least " + MIN_REFINE_BEAM_SIZE + ": " +
settings.beamSize());
+ }
+ final FeedforwardNetwork refined = model.network().copy();
+ final List<Derivation> derivations = derive(samples);
+ if (derivations.isEmpty()) {
+ throw new IllegalArgumentException("no trainable samples for
refinement");
+ }
+ final int[][] oracles = new int[derivations.size()][];
+ for (int d = 0; d < oracles.length; d++) {
+ final List<Transition> oracle = derivations.get(d).oracle();
+ oracles[d] = new int[oracle.size()];
+ for (int i = 0; i < oracles[d].length; i++) {
+ final int id = refined.transitionIndex(oracle.get(i));
+ if (id < 0) {
+ throw new IllegalArgumentException(
+ "unknown transition in the refinement samples: " +
oracle.get(i).encode());
+ }
+ oracles[d][i] = id;
+ }
+ }
+
+ final Optimizer optimizer = new Optimizer(refined, settings);
+ final Transition[] transitions = refined.decodedTransitions();
+ final Random random = new Random(settings.seed());
+ final int[] order = identity(derivations.size());
+ for (int epoch = 1; epoch <= settings.epochs(); epoch++) {
+ final long epochStart = System.currentTimeMillis();
+ shuffle(order, random);
+ double loss = 0.0;
+ for (final int index : order) {
+ loss += optimizer.refineSentence(derivations.get(index).sample(),
oracles[index],
+ transitions, settings.beamSize());
+ }
+ checkFinite(refined);
+ logger.info("refine epoch {}: loss {} over {} sentences in {} ms", epoch,
+ loss / order.length, order.length, System.currentTimeMillis() -
epochStart);
+ }
+ final Map<String, String> manifestEntries = manifestEntries(settings);
+ manifestEntries.put(BeamSearch.BEAM_SIZE_PARAMETER,
Integer.toString(settings.beamSize()));
+ return new FeedforwardDependencyModel(model.getLanguage(), refined,
manifestEntries);
+ }
+
+ /**
+ * @param settings The hyperparameters of the run.
+ * @return The manifest entries recording the run. Never {@code null}.
+ */
+ private static Map<String, String> manifestEntries(Settings settings) {
+ final Map<String, String> entries = new HashMap<>();
+ entries.put(BaseModel.TRAINING_ITERATIONS_PROPERTY,
Integer.toString(settings.epochs()));
+ entries.put(BaseModel.TRAINING_CUTOFF_PROPERTY,
Integer.toString(settings.wordCutoff()));
+ return entries;
+ }
+
+ /**
+ * A projective training sample with its arc-standard oracle derivation.
+ *
+ * @param sample The sample.
+ * @param oracle The gold transitions, in order.
+ */
+ private record Derivation(DependencySample sample, List<Transition> oracle) {
+ }
+
+ /**
+ * Reads a sample stream into memory and derives the oracle of every
projective
+ * sample once; the trainer passes over the derivations repeatedly.
+ *
+ * @param samples The stream to drain.
+ * @return The derivations in stream order, without the non-projective
samples.
+ * Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ */
+ private static List<Derivation> derive(ObjectStream<DependencySample>
samples)
+ throws IOException {
+ final List<Derivation> derivations = new ArrayList<>();
+ int skipped = 0;
+ DependencySample sample;
+ while ((sample = samples.read()) != null) {
+ if (ArcStandardOracle.isProjective(sample.getTree())) {
+ derivations.add(new Derivation(sample,
ArcStandardOracle.transitions(sample.getTree())));
+ } else {
+ skipped++;
+ }
+ }
+ if (skipped > 0) {
+ logger.warn("Skipped {} non-projective sample(s) without an arc-standard
derivation.",
+ skipped);
+ }
+ return derivations;
+ }
+
+ /**
+ * Overwrites the random word rows with pretrained vectors where available.
+ *
+ * @param network The freshly initialized network whose word rows are seeded.
+ * @param pretrained The pretrained vector source, returning {@code null}
for a word it
+ * does not cover.
+ * @param settings The hyperparameters, which fix the expected vector width.
+ * @throws IllegalArgumentException Thrown if a pretrained vector has a
different width
+ * than the embedding size or contains a non-finite value.
+ */
+ private static void seed(FeedforwardNetwork network,
+ Function<String, float[]> pretrained, Settings settings) {
+ int seeded = 0;
+ for (final Map.Entry<String, Integer> entry :
network.wordIds().entrySet()) {
+ if (FeedforwardNetwork.isReservedSymbol(entry.getKey())) {
+ continue;
+ }
+ final float[] vector = pretrained.apply(entry.getKey());
+ if (vector == null) {
+ continue;
+ }
+ if (vector.length != settings.embeddingSize()) {
+ throw new IllegalArgumentException("pretrained vector for '" +
entry.getKey()
+ + "' has " + vector.length + " dimensions, expected "
+ + settings.embeddingSize());
+ }
+ for (final float value : vector) {
+ if (!Float.isFinite(value)) {
+ throw new IllegalArgumentException(
+ "pretrained vector for '" + entry.getKey() + "' contains a
non-finite value");
+ }
+ }
+ System.arraycopy(vector, 0, network.embeddings()[entry.getValue()], 0,
vector.length);
+ seeded++;
+ }
+ logger.info("seeded {} of {} word embeddings from the pretrained source",
seeded,
+ network.wordIds().size());
+ }
+
+ /**
+ * Builds the vocabularies and randomly initialized weights. Symbols are
assigned
+ * rows in sorted order after the reserved symbols, so the network depends
only on
+ * the samples and the settings. A word, tag, or label in the training data
that
+ * spells a reserved symbol shares the reserved row, as an unknown symbol
does.
+ *
+ * @param derivations The projective training samples with their oracles.
+ * @param settings The hyperparameters.
+ * @return The untrained network. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if a vocabulary or the transition
inventory
+ * exceeds {@link ResourceLimits#MAX_ENTRIES} entries, a symbol
exceeds as many
+ * UTF-8 bytes, or the network would store more than
+ * {@link ResourceLimits#MAX_MATRIX_CELLS} values.
+ */
+ private static FeedforwardNetwork initialize(List<Derivation> derivations,
+ Settings settings) {
+ final Map<String, Integer> wordCounts = new HashMap<>();
+ final Set<String> tagSet = new HashSet<>();
+ final Set<String> labelSet = new HashSet<>();
+ final Set<String> transitionSet = new HashSet<>();
+ for (final Derivation derivation : derivations) {
+ final DependencySample s = derivation.sample();
+ for (final String token : s.getTokens()) {
+ wordCounts.merge(FeedforwardNetwork.normalize(token), 1, Integer::sum);
+ }
+ for (final String tag : s.getTags()) {
+ tagSet.add(tag);
+ }
+ final DependencyTree tree = s.getTree();
+ for (int i = 0; i < tree.size(); i++) {
+ labelSet.add(tree.relationOf(i));
+ }
+ for (final Transition transition : derivation.oracle()) {
+ transitionSet.add(transition.encode());
+ }
+ }
+ final List<String> words = new ArrayList<>();
+ for (final Map.Entry<String, Integer> entry : wordCounts.entrySet()) {
+ if (entry.getValue() >= settings.wordCutoff()) {
+ words.add(entry.getKey());
+ }
+ }
+
+ int row = 0;
+ final Map<String, Integer> wordIds = new HashMap<>();
+ row = assignRows(wordIds, row, FeedforwardNetwork.RESERVED_WORD_SYMBOLS,
words,
+ "word vocabulary");
+ final Map<String, Integer> tagIds = new HashMap<>();
+ row = assignRows(tagIds, row, FeedforwardNetwork.RESERVED_TAG_SYMBOLS,
+ new ArrayList<>(tagSet), "tag vocabulary");
+ final Map<String, Integer> labelIds = new HashMap<>();
+ row = assignRows(labelIds, row, FeedforwardNetwork.RESERVED_LABEL_SYMBOLS,
+ new ArrayList<>(labelSet), "label vocabulary");
+
+ final String[] transitions = transitionSet.toArray(String[]::new);
+ Arrays.sort(transitions);
+ ResourceLimits.requireWithinMaxEntries(transitions.length, "transition
count");
+ for (final String transition : transitions) {
+ FeedforwardNetwork.requireStorable(transition, "transition");
+ }
+ requireWithinCellLimit(row, transitions.length, settings.embeddingSize(),
+ settings.hiddenSize());
+
+ final Random random = new Random(settings.seed());
+ final int inputSize = FeedforwardContext.FEATURE_COUNT *
settings.embeddingSize();
+ final float[][] embeddings = uniform(random, row, settings.embeddingSize(),
+ EMBEDDING_INIT_SCALE);
+ final float[][] hiddenWeights = uniform(random, settings.hiddenSize(),
inputSize,
+ Math.sqrt(GLOROT_NUMERATOR / (inputSize + settings.hiddenSize())));
+ final float[][] outputWeights = uniform(random, transitions.length,
+ settings.hiddenSize(),
+ Math.sqrt(GLOROT_NUMERATOR / (settings.hiddenSize() +
transitions.length)));
+ return new FeedforwardNetwork(wordIds, tagIds, labelIds, transitions,
+ settings.embeddingSize(), embeddings, hiddenWeights,
+ new float[settings.hiddenSize()], outputWeights, new
float[transitions.length]);
+ }
+
+ /**
+ * Assigns the next embedding rows of one vocabulary: the reserved symbols
first, then
+ * the observed symbols in sorted order, skipping observed symbols that
spell a
+ * reserved one.
+ *
+ * @param ids The vocabulary to fill.
+ * @param row The next free embedding row.
+ * @param reserved The reserved symbols, in the order they take their rows.
+ * @param observed The symbols seen in the training data; sorted in place.
+ * @param label The vocabulary name used in error messages.
+ * @return The next free embedding row after the vocabulary.
+ * @throws IllegalArgumentException Thrown if the vocabulary exceeds
+ * {@link ResourceLimits#MAX_ENTRIES} entries or a symbol as many
UTF-8 bytes.
+ */
+ private static int assignRows(Map<String, Integer> ids, int row, String[]
reserved,
+ List<String> observed, String label) {
+ int next = row;
+ for (final String symbol : reserved) {
+ ids.put(symbol, next++);
+ }
+ observed.sort(null);
+ for (final String symbol : observed) {
+ if (!ids.containsKey(symbol)) {
+ FeedforwardNetwork.requireStorable(symbol, label);
+ ids.put(symbol, next++);
+ }
+ }
+ ResourceLimits.requireWithinMaxEntries(ids.size(), label + " size");
Review Comment:
Trainer now checks word, tag and label counts together against
ResourceLimits.MAX_ENTRIES, same limit the loader uses. Added tests at the
limit and one past it.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyTrainer.java:
##########
@@ -0,0 +1,1123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.function.Function;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import opennlp.tools.ml.BeamSearch;
+import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.Parameters;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.TrainingParameters;
+import opennlp.tools.util.model.BaseModel;
+
+/**
+ * Trains the {@link FeedforwardDependencyModel}: oracle-derived transition
examples,
+ * minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and
inverted
+ * dropout on the hidden layer, the training recipe of
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The hyperparameters are read from {@link TrainingParameters}: the keys
of this
+ * class plus {@link Parameters#ITERATIONS_PARAM} for the number of epochs,
+ * {@link Parameters#CUTOFF_PARAM} for the minimum word frequency, and
+ * {@link BeamSearch#BEAM_SIZE_PARAMETER} for the refinement beam. Other keys
are
+ * ignored.</p>
+ *
+ * <p>Words below the frequency cutoff share the reserved unknown row, which is
+ * therefore trained only when such words exist; the unknown rows of the tag
and label
+ * vocabularies keep their random initialization. Absent template positions
share a
+ * trained padding row. Non-projective samples have no arc-standard derivation
and are
+ * skipped. Training is deterministic for a fixed {@link #SEED_PARAM}.</p>
+ *
+ * @since 3.0.0
+ */
+public final class FeedforwardDependencyTrainer {
+
+ /** The training parameter key of the embedding dimensionality; default
{@value #DEFAULT_EMBEDDING_SIZE}. */
+ public static final String EMBEDDING_SIZE_PARAM = "EmbeddingSize";
+
+ /** The training parameter key of the hidden layer width; default {@value
#DEFAULT_HIDDEN_SIZE}. */
+ public static final String HIDDEN_SIZE_PARAM = "HiddenSize";
+
+ /** The training parameter key of the minibatch size; default {@value
#DEFAULT_BATCH_SIZE}. */
+ public static final String BATCH_SIZE_PARAM = "BatchSize";
+
+ /** The training parameter key of the AdaGrad step size; default {@value
#DEFAULT_LEARNING_RATE}. */
+ public static final String LEARNING_RATE_PARAM = "LearningRate";
+
+ /** The training parameter key of the L2 penalty on dense weights; default
{@value #DEFAULT_L2}. */
+ public static final String L2_PARAM = "L2";
+
+ /** The training parameter key of the hidden dropout probability; default
{@value #DEFAULT_DROPOUT}. */
+ public static final String DROPOUT_PARAM = "Dropout";
+
+ /**
+ * The training parameter key of the random seed, read as an {@code int};
default
+ * {@value #DEFAULT_SEED}.
+ */
+ public static final String SEED_PARAM = "Seed";
+
+ /** The default embedding dimensionality. */
+ public static final int DEFAULT_EMBEDDING_SIZE = 50;
+
+ /** The default hidden layer width. */
+ public static final int DEFAULT_HIDDEN_SIZE = 200;
+
+ /** The default number of epochs. */
+ public static final int DEFAULT_EPOCHS = 10;
+
+ /** The default minibatch size. */
+ public static final int DEFAULT_BATCH_SIZE = 256;
+
+ /** The default AdaGrad step size. */
+ public static final double DEFAULT_LEARNING_RATE = 0.02;
+
+ /** The default L2 penalty. */
+ public static final double DEFAULT_L2 = 1e-8;
+
+ /** The default hidden dropout probability. */
+ public static final double DEFAULT_DROPOUT = 0.5;
+
+ /** The default minimum word frequency for an embedding row. */
+ public static final int DEFAULT_WORD_CUTOFF = 2;
+
+ /** The default random seed. */
+ public static final int DEFAULT_SEED = 17;
+
+ /** The default refinement beam size. */
+ public static final int DEFAULT_BEAM_SIZE = 4;
+
+ /** The maximum embedding width. */
+ static final int MAX_EMBEDDING_SIZE = 4_096;
+
+ /** The maximum hidden-layer width. */
+ static final int MAX_HIDDEN_SIZE = 65_536;
+
+ private static final Logger logger =
+ LoggerFactory.getLogger(FeedforwardDependencyTrainer.class);
+
+ /** The AdaGrad denominator offset that keeps the first steps finite. */
+ private static final double ADAGRAD_EPSILON = 1e-6;
+
+ /** The derivative factor of the cube activation, {@code d/dx x^3 = 3x^2}. */
+ private static final double CUBE_DERIVATIVE_FACTOR = 3.0;
+
+ /** The numerator of the Glorot uniform bound, {@code sqrt(6 / (fanIn +
fanOut))}. */
+ private static final double GLOROT_NUMERATOR = 6.0;
+
+ /** The half-width of the uniform range embeddings are drawn from. */
+ private static final double EMBEDDING_INIT_SCALE = 0.01;
+
+ /** The smallest probability the reported training loss takes the log of. */
+ private static final double LOSS_PROBABILITY_FLOOR = 1e-12;
+
+ /** SHIFT and at least one RIGHT_ARC are required to parse a sentence. */
+ private static final int MIN_TRANSITIONS = 2;
+
+ /** The smallest beam that can lose the gold derivation to another path. */
+ private static final int MIN_REFINE_BEAM_SIZE = 2;
+
+ /** Prevents construction of this utility class. */
+ private FeedforwardDependencyTrainer() {
+ }
+
+ /**
+ * The parsed hyperparameters of one training or refinement run.
+ *
+ * @param embeddingSize The embedding dimensionality. Must be positive.
+ * @param hiddenSize The hidden layer width. Must be positive.
+ * @param epochs The number of passes over the examples. Must be positive.
+ * @param batchSize The minibatch size. Must be positive.
+ * @param learningRate The AdaGrad step size. Must be positive.
+ * @param l2 The L2 penalty applied to the dense weights. Must not be
negative.
+ * @param dropout The hidden dropout probability. Must be in {@code [0, 1)}.
+ * @param wordCutoff The minimum frequency for a word to get its own
embedding. Must
+ * not be negative.
+ * @param seed The random seed making a run reproducible.
+ * @param beamSize The refinement beam width; {@link #refine} requires at
least 2.
+ */
+ record Settings(int embeddingSize, int hiddenSize, int epochs, int batchSize,
+ double learningRate, double l2, double dropout, int wordCutoff, long
seed,
+ int beamSize) {
+
+ /**
+ * Validates the hyperparameters.
+ *
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range or the smallest network of this shape exceeds
+ * {@link ResourceLimits#MAX_MATRIX_CELLS} float values.
+ */
+ Settings {
+ ParamChecks.requirePositive(embeddingSize, EMBEDDING_SIZE_PARAM);
+ ParamChecks.requirePositive(hiddenSize, HIDDEN_SIZE_PARAM);
+ ParamChecks.requirePositive(epochs, Parameters.ITERATIONS_PARAM);
+ ParamChecks.requirePositive(batchSize, BATCH_SIZE_PARAM);
+ if (embeddingSize > MAX_EMBEDDING_SIZE) {
+ throw new IllegalArgumentException(EMBEDDING_SIZE_PARAM + " exceeds "
+ MAX_EMBEDDING_SIZE
+ + ": " + embeddingSize);
+ }
+ if (hiddenSize > MAX_HIDDEN_SIZE) {
+ throw new IllegalArgumentException(HIDDEN_SIZE_PARAM + " exceeds " +
MAX_HIDDEN_SIZE
+ + ": " + hiddenSize);
+ }
+ requireWithinCellLimit(FeedforwardNetwork.RESERVED_ROWS, MIN_TRANSITIONS,
+ embeddingSize, hiddenSize);
+ ParamChecks.requirePositive(learningRate, LEARNING_RATE_PARAM);
+ ParamChecks.requireNonNegative(ParamChecks.requireFinite(l2, L2_PARAM),
L2_PARAM);
+ if (!(dropout >= 0.0 && dropout < 1.0)) {
+ throw new IllegalArgumentException(DROPOUT_PARAM + " must be in [0,
1): " + dropout);
+ }
+ ParamChecks.requireNonNegative(wordCutoff, Parameters.CUTOFF_PARAM);
+ }
+
+ /**
+ * Reads the hyperparameters, substituting the defaults for absent keys.
+ *
+ * @param parameters The training parameters. Must not be {@code null}.
+ * @return The parsed settings. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range.
+ */
+ static Settings from(TrainingParameters parameters) {
+ return new Settings(
+ parameters.getIntParameter(EMBEDDING_SIZE_PARAM,
DEFAULT_EMBEDDING_SIZE),
+ parameters.getIntParameter(HIDDEN_SIZE_PARAM, DEFAULT_HIDDEN_SIZE),
+ parameters.getIntParameter(Parameters.ITERATIONS_PARAM,
DEFAULT_EPOCHS),
+ parameters.getIntParameter(BATCH_SIZE_PARAM, DEFAULT_BATCH_SIZE),
+ parameters.getDoubleParameter(LEARNING_RATE_PARAM,
DEFAULT_LEARNING_RATE),
+ parameters.getDoubleParameter(L2_PARAM, DEFAULT_L2),
+ parameters.getDoubleParameter(DROPOUT_PARAM, DEFAULT_DROPOUT),
+ parameters.getIntParameter(Parameters.CUTOFF_PARAM,
DEFAULT_WORD_CUTOFF),
+ parameters.getIntParameter(SEED_PARAM, DEFAULT_SEED),
+ parameters.getIntParameter(BeamSearch.BEAM_SIZE_PARAMETER,
DEFAULT_BEAM_SIZE));
+ }
+ }
+
+ /**
+ * Trains a model from dependency samples.
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}. Must not be {@code
null}.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}, a
+ * hyperparameter is out of range, no trainable example can be
derived from
+ * the samples, a vocabulary exceeds {@link
ResourceLimits#MAX_ENTRIES}
+ * entries or a symbol as many UTF-8 bytes, or the network would
store more
+ * than {@link ResourceLimits#MAX_MATRIX_CELLS} values.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel train(String languageCode,
+ ObjectStream<DependencySample> samples, TrainingParameters parameters)
+ throws IOException {
+ return train(languageCode, samples, parameters, null);
+ }
+
+ /**
+ * Trains a model from dependency samples, seeding word embeddings from a
pretrained
+ * source.
+ *
+ * <p>The provider is consulted once per vocabulary word during
initialization. Words
+ * it returns {@code null} for keep their random initialization. All
embeddings remain
+ * trainable and are stored in the model.</p>
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}. Must not be {@code
null}.
+ * @param pretrained Maps a normalized word to its pretrained vector of
exactly
+ * {@link #EMBEDDING_SIZE_PARAM} dimensions, or {@code
null} for
+ * unknown words. May be {@code null} to disable seeding.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if {@code languageCode}, {@code
samples},
+ * or {@code parameters} is {@code null}, a hyperparameter is out of
range, no
+ * trainable example can be derived, a vocabulary exceeds
+ * {@link ResourceLimits#MAX_ENTRIES} entries or a symbol as many
UTF-8 bytes,
+ * the network would store more than {@link
ResourceLimits#MAX_MATRIX_CELLS}
+ * values, or a pretrained vector has the wrong dimensionality or a
non-finite
+ * value.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel train(String languageCode,
+ ObjectStream<DependencySample> samples, TrainingParameters parameters,
+ Function<String, float[]> pretrained) throws IOException {
+ ParamChecks.requireNonNullArg(languageCode, "languageCode");
+ ParamChecks.requireNonNullArg(samples, "samples");
+ ParamChecks.requireNonNullArg(parameters, "parameters");
+ final Settings settings = Settings.from(parameters);
+ final List<Derivation> derivations = derive(samples);
+ if (derivations.isEmpty()) {
+ throw new IllegalArgumentException("no trainable examples in the
samples");
+ }
+ final FeedforwardNetwork network = initialize(derivations, settings);
+ if (pretrained != null) {
+ seed(network, pretrained, settings);
+ }
+ int exampleCount = 0;
+ for (final Derivation derivation : derivations) {
+ exampleCount += derivation.oracle().size();
+ }
+ final int[][] features = new int[exampleCount][];
+ final int[] gold = new int[exampleCount];
+ collectExamples(derivations, network, features, gold);
+ optimize(network, features, gold, settings);
+ return new FeedforwardDependencyModel(languageCode, network,
manifestEntries(settings));
+ }
+
+ /**
+ * Refines a locally trained model with beam search and the early-update
method of
+ * <a href="https://aclanthology.org/P04-1015/";>Collins and Roark
(2004)</a>. The loss
+ * is conditional likelihood over candidate paths scored by summed
log-probabilities.
+ *
+ * <p>Refinement trains a copy with per-sentence AdaGrad steps and no
dropout.
+ * {@link Parameters#ITERATIONS_PARAM} sets the number of passes and
+ * {@link BeamSearch#BEAM_SIZE_PARAMETER} the beam width, which is also the
beam size
+ * to parse the result with. The transition inventory of {@code model} is not
+ * extended.</p>
+ *
+ * @param model The locally trained model. Not modified. Must not be {@code
null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}; the epochs, learning
rate, L2
+ * penalty, seed, and beam size apply. Must not be {@code
null}.
+ * @return A new refined model, distinct from {@code model}. Never {@code
null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}, a
+ * hyperparameter is out of range, the beam size is below 2, no
trainable
+ * sample can be derived, or a sample requires a transition {@code
model}
+ * does not know.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
Review Comment:
Added to the Javadoc. refine throws IllegalStateException if a score is
non-finite while decoding.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyTrainer.java:
##########
@@ -0,0 +1,1123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.function.Function;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import opennlp.tools.ml.BeamSearch;
+import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.Parameters;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.TrainingParameters;
+import opennlp.tools.util.model.BaseModel;
+
+/**
+ * Trains the {@link FeedforwardDependencyModel}: oracle-derived transition
examples,
+ * minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and
inverted
+ * dropout on the hidden layer, the training recipe of
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The hyperparameters are read from {@link TrainingParameters}: the keys
of this
+ * class plus {@link Parameters#ITERATIONS_PARAM} for the number of epochs,
+ * {@link Parameters#CUTOFF_PARAM} for the minimum word frequency, and
+ * {@link BeamSearch#BEAM_SIZE_PARAMETER} for the refinement beam. Other keys
are
+ * ignored.</p>
+ *
+ * <p>Words below the frequency cutoff share the reserved unknown row, which is
+ * therefore trained only when such words exist; the unknown rows of the tag
and label
+ * vocabularies keep their random initialization. Absent template positions
share a
+ * trained padding row. Non-projective samples have no arc-standard derivation
and are
+ * skipped. Training is deterministic for a fixed {@link #SEED_PARAM}.</p>
+ *
+ * @since 3.0.0
+ */
+public final class FeedforwardDependencyTrainer {
+
+ /** The training parameter key of the embedding dimensionality; default
{@value #DEFAULT_EMBEDDING_SIZE}. */
+ public static final String EMBEDDING_SIZE_PARAM = "EmbeddingSize";
+
+ /** The training parameter key of the hidden layer width; default {@value
#DEFAULT_HIDDEN_SIZE}. */
+ public static final String HIDDEN_SIZE_PARAM = "HiddenSize";
+
+ /** The training parameter key of the minibatch size; default {@value
#DEFAULT_BATCH_SIZE}. */
+ public static final String BATCH_SIZE_PARAM = "BatchSize";
+
+ /** The training parameter key of the AdaGrad step size; default {@value
#DEFAULT_LEARNING_RATE}. */
+ public static final String LEARNING_RATE_PARAM = "LearningRate";
+
+ /** The training parameter key of the L2 penalty on dense weights; default
{@value #DEFAULT_L2}. */
+ public static final String L2_PARAM = "L2";
+
+ /** The training parameter key of the hidden dropout probability; default
{@value #DEFAULT_DROPOUT}. */
+ public static final String DROPOUT_PARAM = "Dropout";
+
+ /**
+ * The training parameter key of the random seed, read as an {@code int};
default
+ * {@value #DEFAULT_SEED}.
+ */
+ public static final String SEED_PARAM = "Seed";
+
+ /** The default embedding dimensionality. */
+ public static final int DEFAULT_EMBEDDING_SIZE = 50;
+
+ /** The default hidden layer width. */
+ public static final int DEFAULT_HIDDEN_SIZE = 200;
+
+ /** The default number of epochs. */
+ public static final int DEFAULT_EPOCHS = 10;
+
+ /** The default minibatch size. */
+ public static final int DEFAULT_BATCH_SIZE = 256;
+
+ /** The default AdaGrad step size. */
+ public static final double DEFAULT_LEARNING_RATE = 0.02;
+
+ /** The default L2 penalty. */
+ public static final double DEFAULT_L2 = 1e-8;
+
+ /** The default hidden dropout probability. */
+ public static final double DEFAULT_DROPOUT = 0.5;
+
+ /** The default minimum word frequency for an embedding row. */
+ public static final int DEFAULT_WORD_CUTOFF = 2;
+
+ /** The default random seed. */
+ public static final int DEFAULT_SEED = 17;
+
+ /** The default refinement beam size. */
+ public static final int DEFAULT_BEAM_SIZE = 4;
+
+ /** The maximum embedding width. */
+ static final int MAX_EMBEDDING_SIZE = 4_096;
+
+ /** The maximum hidden-layer width. */
+ static final int MAX_HIDDEN_SIZE = 65_536;
+
+ private static final Logger logger =
+ LoggerFactory.getLogger(FeedforwardDependencyTrainer.class);
+
+ /** The AdaGrad denominator offset that keeps the first steps finite. */
+ private static final double ADAGRAD_EPSILON = 1e-6;
+
+ /** The derivative factor of the cube activation, {@code d/dx x^3 = 3x^2}. */
+ private static final double CUBE_DERIVATIVE_FACTOR = 3.0;
+
+ /** The numerator of the Glorot uniform bound, {@code sqrt(6 / (fanIn +
fanOut))}. */
+ private static final double GLOROT_NUMERATOR = 6.0;
+
+ /** The half-width of the uniform range embeddings are drawn from. */
+ private static final double EMBEDDING_INIT_SCALE = 0.01;
+
+ /** The smallest probability the reported training loss takes the log of. */
+ private static final double LOSS_PROBABILITY_FLOOR = 1e-12;
+
+ /** SHIFT and at least one RIGHT_ARC are required to parse a sentence. */
+ private static final int MIN_TRANSITIONS = 2;
+
+ /** The smallest beam that can lose the gold derivation to another path. */
+ private static final int MIN_REFINE_BEAM_SIZE = 2;
+
+ /** Prevents construction of this utility class. */
+ private FeedforwardDependencyTrainer() {
+ }
+
+ /**
+ * The parsed hyperparameters of one training or refinement run.
+ *
+ * @param embeddingSize The embedding dimensionality. Must be positive.
+ * @param hiddenSize The hidden layer width. Must be positive.
+ * @param epochs The number of passes over the examples. Must be positive.
+ * @param batchSize The minibatch size. Must be positive.
+ * @param learningRate The AdaGrad step size. Must be positive.
+ * @param l2 The L2 penalty applied to the dense weights. Must not be
negative.
+ * @param dropout The hidden dropout probability. Must be in {@code [0, 1)}.
+ * @param wordCutoff The minimum frequency for a word to get its own
embedding. Must
+ * not be negative.
+ * @param seed The random seed making a run reproducible.
+ * @param beamSize The refinement beam width; {@link #refine} requires at
least 2.
+ */
+ record Settings(int embeddingSize, int hiddenSize, int epochs, int batchSize,
+ double learningRate, double l2, double dropout, int wordCutoff, long
seed,
+ int beamSize) {
+
+ /**
+ * Validates the hyperparameters.
+ *
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range or the smallest network of this shape exceeds
+ * {@link ResourceLimits#MAX_MATRIX_CELLS} float values.
+ */
+ Settings {
+ ParamChecks.requirePositive(embeddingSize, EMBEDDING_SIZE_PARAM);
+ ParamChecks.requirePositive(hiddenSize, HIDDEN_SIZE_PARAM);
+ ParamChecks.requirePositive(epochs, Parameters.ITERATIONS_PARAM);
+ ParamChecks.requirePositive(batchSize, BATCH_SIZE_PARAM);
+ if (embeddingSize > MAX_EMBEDDING_SIZE) {
+ throw new IllegalArgumentException(EMBEDDING_SIZE_PARAM + " exceeds "
+ MAX_EMBEDDING_SIZE
+ + ": " + embeddingSize);
+ }
+ if (hiddenSize > MAX_HIDDEN_SIZE) {
+ throw new IllegalArgumentException(HIDDEN_SIZE_PARAM + " exceeds " +
MAX_HIDDEN_SIZE
+ + ": " + hiddenSize);
+ }
+ requireWithinCellLimit(FeedforwardNetwork.RESERVED_ROWS, MIN_TRANSITIONS,
+ embeddingSize, hiddenSize);
+ ParamChecks.requirePositive(learningRate, LEARNING_RATE_PARAM);
+ ParamChecks.requireNonNegative(ParamChecks.requireFinite(l2, L2_PARAM),
L2_PARAM);
+ if (!(dropout >= 0.0 && dropout < 1.0)) {
+ throw new IllegalArgumentException(DROPOUT_PARAM + " must be in [0,
1): " + dropout);
+ }
+ ParamChecks.requireNonNegative(wordCutoff, Parameters.CUTOFF_PARAM);
+ }
+
+ /**
+ * Reads the hyperparameters, substituting the defaults for absent keys.
+ *
+ * @param parameters The training parameters. Must not be {@code null}.
+ * @return The parsed settings. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range.
+ */
+ static Settings from(TrainingParameters parameters) {
+ return new Settings(
+ parameters.getIntParameter(EMBEDDING_SIZE_PARAM,
DEFAULT_EMBEDDING_SIZE),
+ parameters.getIntParameter(HIDDEN_SIZE_PARAM, DEFAULT_HIDDEN_SIZE),
+ parameters.getIntParameter(Parameters.ITERATIONS_PARAM,
DEFAULT_EPOCHS),
+ parameters.getIntParameter(BATCH_SIZE_PARAM, DEFAULT_BATCH_SIZE),
+ parameters.getDoubleParameter(LEARNING_RATE_PARAM,
DEFAULT_LEARNING_RATE),
+ parameters.getDoubleParameter(L2_PARAM, DEFAULT_L2),
+ parameters.getDoubleParameter(DROPOUT_PARAM, DEFAULT_DROPOUT),
+ parameters.getIntParameter(Parameters.CUTOFF_PARAM,
DEFAULT_WORD_CUTOFF),
+ parameters.getIntParameter(SEED_PARAM, DEFAULT_SEED),
+ parameters.getIntParameter(BeamSearch.BEAM_SIZE_PARAMETER,
DEFAULT_BEAM_SIZE));
+ }
+ }
+
+ /**
+ * Trains a model from dependency samples.
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}. Must not be {@code
null}.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}, a
+ * hyperparameter is out of range, no trainable example can be
derived from
+ * the samples, a vocabulary exceeds {@link
ResourceLimits#MAX_ENTRIES}
+ * entries or a symbol as many UTF-8 bytes, or the network would
store more
+ * than {@link ResourceLimits#MAX_MATRIX_CELLS} values.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel train(String languageCode,
+ ObjectStream<DependencySample> samples, TrainingParameters parameters)
+ throws IOException {
+ return train(languageCode, samples, parameters, null);
+ }
+
+ /**
+ * Trains a model from dependency samples, seeding word embeddings from a
pretrained
+ * source.
+ *
+ * <p>The provider is consulted once per vocabulary word during
initialization. Words
+ * it returns {@code null} for keep their random initialization. All
embeddings remain
+ * trainable and are stored in the model.</p>
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}. Must not be {@code
null}.
+ * @param pretrained Maps a normalized word to its pretrained vector of
exactly
+ * {@link #EMBEDDING_SIZE_PARAM} dimensions, or {@code
null} for
+ * unknown words. May be {@code null} to disable seeding.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if {@code languageCode}, {@code
samples},
+ * or {@code parameters} is {@code null}, a hyperparameter is out of
range, no
+ * trainable example can be derived, a vocabulary exceeds
+ * {@link ResourceLimits#MAX_ENTRIES} entries or a symbol as many
UTF-8 bytes,
+ * the network would store more than {@link
ResourceLimits#MAX_MATRIX_CELLS}
+ * values, or a pretrained vector has the wrong dimensionality or a
non-finite
+ * value.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel train(String languageCode,
+ ObjectStream<DependencySample> samples, TrainingParameters parameters,
+ Function<String, float[]> pretrained) throws IOException {
+ ParamChecks.requireNonNullArg(languageCode, "languageCode");
+ ParamChecks.requireNonNullArg(samples, "samples");
+ ParamChecks.requireNonNullArg(parameters, "parameters");
+ final Settings settings = Settings.from(parameters);
+ final List<Derivation> derivations = derive(samples);
+ if (derivations.isEmpty()) {
+ throw new IllegalArgumentException("no trainable examples in the
samples");
+ }
+ final FeedforwardNetwork network = initialize(derivations, settings);
+ if (pretrained != null) {
+ seed(network, pretrained, settings);
+ }
+ int exampleCount = 0;
+ for (final Derivation derivation : derivations) {
+ exampleCount += derivation.oracle().size();
+ }
+ final int[][] features = new int[exampleCount][];
+ final int[] gold = new int[exampleCount];
+ collectExamples(derivations, network, features, gold);
+ optimize(network, features, gold, settings);
+ return new FeedforwardDependencyModel(languageCode, network,
manifestEntries(settings));
+ }
+
+ /**
+ * Refines a locally trained model with beam search and the early-update
method of
+ * <a href="https://aclanthology.org/P04-1015/";>Collins and Roark
(2004)</a>. The loss
+ * is conditional likelihood over candidate paths scored by summed
log-probabilities.
+ *
+ * <p>Refinement trains a copy with per-sentence AdaGrad steps and no
dropout.
+ * {@link Parameters#ITERATIONS_PARAM} sets the number of passes and
+ * {@link BeamSearch#BEAM_SIZE_PARAMETER} the beam width, which is also the
beam size
Review Comment:
Kept BeamSize since #1473 uses it as the default beam for a refined model.
Cutoff now comes from the input model since refinement does not change the word
list. Test checks both.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyTrainer.java:
##########
@@ -0,0 +1,1123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.function.Function;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import opennlp.tools.ml.BeamSearch;
+import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.Parameters;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.TrainingParameters;
+import opennlp.tools.util.model.BaseModel;
+
+/**
+ * Trains the {@link FeedforwardDependencyModel}: oracle-derived transition
examples,
+ * minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and
inverted
+ * dropout on the hidden layer, the training recipe of
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The hyperparameters are read from {@link TrainingParameters}: the keys
of this
+ * class plus {@link Parameters#ITERATIONS_PARAM} for the number of epochs,
+ * {@link Parameters#CUTOFF_PARAM} for the minimum word frequency, and
+ * {@link BeamSearch#BEAM_SIZE_PARAMETER} for the refinement beam. Other keys
are
+ * ignored.</p>
+ *
+ * <p>Words below the frequency cutoff share the reserved unknown row, which is
+ * therefore trained only when such words exist; the unknown rows of the tag
and label
+ * vocabularies keep their random initialization. Absent template positions
share a
+ * trained padding row. Non-projective samples have no arc-standard derivation
and are
+ * skipped. Training is deterministic for a fixed {@link #SEED_PARAM}.</p>
+ *
+ * @since 3.0.0
+ */
+public final class FeedforwardDependencyTrainer {
+
+ /** The training parameter key of the embedding dimensionality; default
{@value #DEFAULT_EMBEDDING_SIZE}. */
+ public static final String EMBEDDING_SIZE_PARAM = "EmbeddingSize";
+
+ /** The training parameter key of the hidden layer width; default {@value
#DEFAULT_HIDDEN_SIZE}. */
+ public static final String HIDDEN_SIZE_PARAM = "HiddenSize";
+
+ /** The training parameter key of the minibatch size; default {@value
#DEFAULT_BATCH_SIZE}. */
+ public static final String BATCH_SIZE_PARAM = "BatchSize";
+
+ /** The training parameter key of the AdaGrad step size; default {@value
#DEFAULT_LEARNING_RATE}. */
+ public static final String LEARNING_RATE_PARAM = "LearningRate";
+
+ /** The training parameter key of the L2 penalty on dense weights; default
{@value #DEFAULT_L2}. */
+ public static final String L2_PARAM = "L2";
Review Comment:
Removed L2_PARAM; the trainer reads QNTrainer.L2COST_PARAM ("L2Cost").
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyTrainer.java:
##########
@@ -0,0 +1,1123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.function.Function;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import opennlp.tools.ml.BeamSearch;
+import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.Parameters;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.TrainingParameters;
+import opennlp.tools.util.model.BaseModel;
+
+/**
+ * Trains the {@link FeedforwardDependencyModel}: oracle-derived transition
examples,
+ * minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and
inverted
+ * dropout on the hidden layer, the training recipe of
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The hyperparameters are read from {@link TrainingParameters}: the keys
of this
+ * class plus {@link Parameters#ITERATIONS_PARAM} for the number of epochs,
+ * {@link Parameters#CUTOFF_PARAM} for the minimum word frequency, and
+ * {@link BeamSearch#BEAM_SIZE_PARAMETER} for the refinement beam. Other keys
are
+ * ignored.</p>
+ *
+ * <p>Words below the frequency cutoff share the reserved unknown row, which is
+ * therefore trained only when such words exist; the unknown rows of the tag
and label
+ * vocabularies keep their random initialization. Absent template positions
share a
+ * trained padding row. Non-projective samples have no arc-standard derivation
and are
+ * skipped. Training is deterministic for a fixed {@link #SEED_PARAM}.</p>
+ *
+ * @since 3.0.0
+ */
+public final class FeedforwardDependencyTrainer {
+
+ /** The training parameter key of the embedding dimensionality; default
{@value #DEFAULT_EMBEDDING_SIZE}. */
+ public static final String EMBEDDING_SIZE_PARAM = "EmbeddingSize";
+
+ /** The training parameter key of the hidden layer width; default {@value
#DEFAULT_HIDDEN_SIZE}. */
+ public static final String HIDDEN_SIZE_PARAM = "HiddenSize";
+
+ /** The training parameter key of the minibatch size; default {@value
#DEFAULT_BATCH_SIZE}. */
+ public static final String BATCH_SIZE_PARAM = "BatchSize";
+
+ /** The training parameter key of the AdaGrad step size; default {@value
#DEFAULT_LEARNING_RATE}. */
+ public static final String LEARNING_RATE_PARAM = "LearningRate";
+
+ /** The training parameter key of the L2 penalty on dense weights; default
{@value #DEFAULT_L2}. */
+ public static final String L2_PARAM = "L2";
+
+ /** The training parameter key of the hidden dropout probability; default
{@value #DEFAULT_DROPOUT}. */
+ public static final String DROPOUT_PARAM = "Dropout";
+
+ /**
+ * The training parameter key of the random seed, read as an {@code int};
default
+ * {@value #DEFAULT_SEED}.
+ */
+ public static final String SEED_PARAM = "Seed";
+
+ /** The default embedding dimensionality. */
+ public static final int DEFAULT_EMBEDDING_SIZE = 50;
+
+ /** The default hidden layer width. */
+ public static final int DEFAULT_HIDDEN_SIZE = 200;
+
+ /** The default number of epochs. */
+ public static final int DEFAULT_EPOCHS = 10;
+
+ /** The default minibatch size. */
+ public static final int DEFAULT_BATCH_SIZE = 256;
+
+ /** The default AdaGrad step size. */
+ public static final double DEFAULT_LEARNING_RATE = 0.02;
+
+ /** The default L2 penalty. */
+ public static final double DEFAULT_L2 = 1e-8;
+
+ /** The default hidden dropout probability. */
+ public static final double DEFAULT_DROPOUT = 0.5;
+
+ /** The default minimum word frequency for an embedding row. */
+ public static final int DEFAULT_WORD_CUTOFF = 2;
+
+ /** The default random seed. */
+ public static final int DEFAULT_SEED = 17;
+
+ /** The default refinement beam size. */
+ public static final int DEFAULT_BEAM_SIZE = 4;
+
+ /** The maximum embedding width. */
+ static final int MAX_EMBEDDING_SIZE = 4_096;
+
+ /** The maximum hidden-layer width. */
+ static final int MAX_HIDDEN_SIZE = 65_536;
+
+ private static final Logger logger =
+ LoggerFactory.getLogger(FeedforwardDependencyTrainer.class);
+
+ /** The AdaGrad denominator offset that keeps the first steps finite. */
+ private static final double ADAGRAD_EPSILON = 1e-6;
+
+ /** The derivative factor of the cube activation, {@code d/dx x^3 = 3x^2}. */
+ private static final double CUBE_DERIVATIVE_FACTOR = 3.0;
+
+ /** The numerator of the Glorot uniform bound, {@code sqrt(6 / (fanIn +
fanOut))}. */
+ private static final double GLOROT_NUMERATOR = 6.0;
+
+ /** The half-width of the uniform range embeddings are drawn from. */
+ private static final double EMBEDDING_INIT_SCALE = 0.01;
+
+ /** The smallest probability the reported training loss takes the log of. */
+ private static final double LOSS_PROBABILITY_FLOOR = 1e-12;
+
+ /** SHIFT and at least one RIGHT_ARC are required to parse a sentence. */
+ private static final int MIN_TRANSITIONS = 2;
+
+ /** The smallest beam that can lose the gold derivation to another path. */
+ private static final int MIN_REFINE_BEAM_SIZE = 2;
+
+ /** Prevents construction of this utility class. */
+ private FeedforwardDependencyTrainer() {
+ }
+
+ /**
+ * The parsed hyperparameters of one training or refinement run.
+ *
+ * @param embeddingSize The embedding dimensionality. Must be positive.
+ * @param hiddenSize The hidden layer width. Must be positive.
+ * @param epochs The number of passes over the examples. Must be positive.
+ * @param batchSize The minibatch size. Must be positive.
+ * @param learningRate The AdaGrad step size. Must be positive.
+ * @param l2 The L2 penalty applied to the dense weights. Must not be
negative.
+ * @param dropout The hidden dropout probability. Must be in {@code [0, 1)}.
+ * @param wordCutoff The minimum frequency for a word to get its own
embedding. Must
+ * not be negative.
+ * @param seed The random seed making a run reproducible.
+ * @param beamSize The refinement beam width; {@link #refine} requires at
least 2.
+ */
+ record Settings(int embeddingSize, int hiddenSize, int epochs, int batchSize,
+ double learningRate, double l2, double dropout, int wordCutoff, long
seed,
+ int beamSize) {
+
+ /**
+ * Validates the hyperparameters.
+ *
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range or the smallest network of this shape exceeds
+ * {@link ResourceLimits#MAX_MATRIX_CELLS} float values.
+ */
+ Settings {
+ ParamChecks.requirePositive(embeddingSize, EMBEDDING_SIZE_PARAM);
+ ParamChecks.requirePositive(hiddenSize, HIDDEN_SIZE_PARAM);
+ ParamChecks.requirePositive(epochs, Parameters.ITERATIONS_PARAM);
+ ParamChecks.requirePositive(batchSize, BATCH_SIZE_PARAM);
+ if (embeddingSize > MAX_EMBEDDING_SIZE) {
+ throw new IllegalArgumentException(EMBEDDING_SIZE_PARAM + " exceeds "
+ MAX_EMBEDDING_SIZE
+ + ": " + embeddingSize);
+ }
+ if (hiddenSize > MAX_HIDDEN_SIZE) {
+ throw new IllegalArgumentException(HIDDEN_SIZE_PARAM + " exceeds " +
MAX_HIDDEN_SIZE
+ + ": " + hiddenSize);
+ }
+ requireWithinCellLimit(FeedforwardNetwork.RESERVED_ROWS, MIN_TRANSITIONS,
+ embeddingSize, hiddenSize);
+ ParamChecks.requirePositive(learningRate, LEARNING_RATE_PARAM);
+ ParamChecks.requireNonNegative(ParamChecks.requireFinite(l2, L2_PARAM),
L2_PARAM);
+ if (!(dropout >= 0.0 && dropout < 1.0)) {
+ throw new IllegalArgumentException(DROPOUT_PARAM + " must be in [0,
1): " + dropout);
+ }
+ ParamChecks.requireNonNegative(wordCutoff, Parameters.CUTOFF_PARAM);
+ }
+
+ /**
+ * Reads the hyperparameters, substituting the defaults for absent keys.
+ *
+ * @param parameters The training parameters. Must not be {@code null}.
+ * @return The parsed settings. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if a value is outside its
documented
+ * range.
+ */
+ static Settings from(TrainingParameters parameters) {
+ return new Settings(
+ parameters.getIntParameter(EMBEDDING_SIZE_PARAM,
DEFAULT_EMBEDDING_SIZE),
+ parameters.getIntParameter(HIDDEN_SIZE_PARAM, DEFAULT_HIDDEN_SIZE),
+ parameters.getIntParameter(Parameters.ITERATIONS_PARAM,
DEFAULT_EPOCHS),
+ parameters.getIntParameter(BATCH_SIZE_PARAM, DEFAULT_BATCH_SIZE),
+ parameters.getDoubleParameter(LEARNING_RATE_PARAM,
DEFAULT_LEARNING_RATE),
+ parameters.getDoubleParameter(L2_PARAM, DEFAULT_L2),
+ parameters.getDoubleParameter(DROPOUT_PARAM, DEFAULT_DROPOUT),
+ parameters.getIntParameter(Parameters.CUTOFF_PARAM,
DEFAULT_WORD_CUTOFF),
+ parameters.getIntParameter(SEED_PARAM, DEFAULT_SEED),
+ parameters.getIntParameter(BeamSearch.BEAM_SIZE_PARAMETER,
DEFAULT_BEAM_SIZE));
+ }
+ }
+
+ /**
+ * Trains a model from dependency samples.
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}. Must not be {@code
null}.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}, a
+ * hyperparameter is out of range, no trainable example can be
derived from
+ * the samples, a vocabulary exceeds {@link
ResourceLimits#MAX_ENTRIES}
+ * entries or a symbol as many UTF-8 bytes, or the network would
store more
+ * than {@link ResourceLimits#MAX_MATRIX_CELLS} values.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel train(String languageCode,
+ ObjectStream<DependencySample> samples, TrainingParameters parameters)
+ throws IOException {
+ return train(languageCode, samples, parameters, null);
+ }
+
+ /**
+ * Trains a model from dependency samples, seeding word embeddings from a
pretrained
+ * source.
+ *
+ * <p>The provider is consulted once per vocabulary word during
initialization. Words
+ * it returns {@code null} for keep their random initialization. All
embeddings remain
+ * trainable and are stored in the model.</p>
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param samples The training samples. Must not be {@code null}.
+ * @param parameters The {@link TrainingParameters}. Must not be {@code
null}.
+ * @param pretrained Maps a normalized word to its pretrained vector of
exactly
+ * {@link #EMBEDDING_SIZE_PARAM} dimensions, or {@code
null} for
+ * unknown words. May be {@code null} to disable seeding.
+ * @return A trained {@link FeedforwardDependencyModel}. Never {@code null}.
+ * @throws IOException Thrown if reading the samples fails.
+ * @throws IllegalArgumentException Thrown if {@code languageCode}, {@code
samples},
+ * or {@code parameters} is {@code null}, a hyperparameter is out of
range, no
+ * trainable example can be derived, a vocabulary exceeds
+ * {@link ResourceLimits#MAX_ENTRIES} entries or a symbol as many
UTF-8 bytes,
+ * the network would store more than {@link
ResourceLimits#MAX_MATRIX_CELLS}
+ * values, or a pretrained vector has the wrong dimensionality or a
non-finite
+ * value.
+ * @throws IllegalStateException Thrown if an epoch produces a non-finite
parameter.
+ */
+ public static FeedforwardDependencyModel train(String languageCode,
Review Comment:
Dropped it along with the seeding code and its test.
> Add a feedforward neural dependency parser
> ------------------------------------------
>
> Key: OPENNLP-1991
> URL: https://issues.apache.org/jira/browse/OPENNLP-1991
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Add a feedforward neural implementation of the dependency parser API
> introduced by OPENNLP-547. Use OpenNLP’s standard model container, artifact
> serializers, TrainingParameters, factory, and CLI conventions. Include
> deterministic training, greedy and beam decoding, model round-trip tests, and
> held-out Universal Dependencies evaluation. Address the deferred review
> feedback on shared scoring/training logic and parser-owned caches. Implement
> this without new dependencies.
> The list above may still get this ticket to further split. For now this is a
> placeholder to capture the code that was moved out of OPENNLP-547. I'll edit
> after 547 is merged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
