[
https://issues.apache.org/jira/browse/OPENNLP-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126906#comment-18126906
]
ASF GitHub Bot commented on OPENNLP-1991:
-----------------------------------------
krickert commented on code in PR #1323:
URL: https://github.com/apache/opennlp/pull/1323#discussion_r4266431916
##########
opennlp-api/src/main/java/opennlp/tools/util/ParamChecks.java:
##########
@@ -172,6 +173,41 @@ public static double requireNonNegative(double value,
String name) {
return value;
}
+ /**
+ * Returns {@code value} if it is greater than zero.
+ * <p>
+ * The message of a thrown exception is {@code "<name> must be positive"}.
+ *
+ * @param value The argument to check.
+ * @param name The parameter name used in the exception message.
+ * @return {@code value}, always greater than zero.
+ * @throws IllegalArgumentException Thrown if {@code value} is zero or
negative.
+ */
+ public static int requirePositive(int value, String name) {
+ if (value <= 0) {
+ throw new IllegalArgumentException(name + POSITIVE_SUFFIX);
+ }
+ return value;
+ }
+
+ /**
+ * Returns {@code value} if it is finite and greater than zero.
+ * <p>
+ * The message of a thrown exception is {@code "<name> must be positive"}.
+ *
+ * @param value The argument to check.
+ * @param name The parameter name used in the exception message.
+ * @return {@code value}, always finite and greater than zero.
+ * @throws IllegalArgumentException Thrown if {@code value} is zero,
negative, infinite,
+ * or {@code NaN}.
+ */
+ public static double requirePositive(double value, String name) {
Review Comment:
Not intended. requirePositive now allows infinity like requireNonNegative
does. The trainer also calls requireFinite for the learning rate. Updated the
test.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardNetwork.java:
##########
@@ -0,0 +1,1011 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.ml.ArrayMath;
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.StringUtil;
+import opennlp.tools.util.model.ArtifactSerializer;
+
+/**
+ * The weights of the feedforward transition parser: embeddings for words,
tags, and arc
+ * labels, one hidden layer with cube activation, and a transition output
layer, after
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The network holds every forward computation the parser and the trainer
share:
+ * {@link #addContribution} is the per-position hidden sum, {@link
#outputScores} the
+ * output layer, {@link #score} the complete inference pass, and
+ * {@link #logSoftmaxInPlace} the normalization of beam decoding. The trainer
writes
+ * its updates into the live arrays the accessors return; the parser never
modifies
+ * them, so a network that is not being trained can be shared between
threads.</p>
+ *
+ * <p>Each vocabulary occupies one consecutive block of embedding rows and
starts with
+ * its reserved symbols, so unknown symbols, absent positions, and the
artificial root
+ * always have a row. Words are matched after {@link #normalize(String)
normalization}.</p>
+ *
+ * @see FeedforwardDependencyModel
+ * @since 3.0.0
+ */
+final class FeedforwardNetwork {
+
+ /** The vocabulary key of every word, tag, or label the network has no
embedding row for. */
+ static final String UNKNOWN = "*UNK*";
+
+ /** The reserved word symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_WORD_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved tag symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_TAG_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved label symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_LABEL_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE};
+
+ /** The embedding rows the reserved symbols of all three vocabularies take.
*/
+ static final int RESERVED_ROWS = RESERVED_WORD_SYMBOLS.length
+ + RESERVED_TAG_SYMBOLS.length + RESERVED_LABEL_SYMBOLS.length;
+
+ /** The embedding row of the unknown symbol in each vocabulary block. */
+ private static final int UNKNOWN_OFFSET = 0;
+
+ /** The embedding row of the absent-position symbol in each vocabulary
block. */
+ private static final int NONE_OFFSET = 1;
+
+ /** The embedding row of the root symbol in the word and tag blocks. */
+ private static final int ROOT_OFFSET = 2;
+
+ private final Map<String, Integer> wordIds;
+ private final Map<String, Integer> tagIds;
+ private final Map<String, Integer> labelIds;
+
+ /** The first embedding row of each vocabulary, indexed like the
vocabularies above. */
+ private final int firstWordRow;
+ private final int firstTagRow;
+ private final int firstLabelRow;
+
+ private final String[] transitions;
+ private final Transition[] decodedTransitions;
+ private final Map<Transition, Integer> transitionIds;
+
+ private final int embeddingSize;
+ private final float[][] embeddings;
+ private final float[][] hiddenWeights;
+ private final float[] hiddenBias;
+ private final float[][] outputWeights;
+ private final float[] outputBias;
+
+ /**
+ * Assembles a network from its vocabularies and weights, copying the maps
so later
+ * changes by the caller do not reach the network.
+ *
+ * @param wordIds The word vocabulary, mapping each normalized word to its
row.
+ * @param tagIds The tag vocabulary.
+ * @param labelIds The dependency label vocabulary.
+ * @param transitions The transition inventory, indexed by output row.
+ * @param embeddingSize The embedding dimensionality.
+ * @param embeddings The embedding rows for words, tags, and labels.
+ * @param hiddenWeights The hidden layer weights.
+ * @param hiddenBias The hidden layer bias.
+ * @param outputWeights The output layer weights.
+ * @param outputBias The output layer bias.
+ * @throws IllegalArgumentException Thrown if a vocabulary lacks a reserved
symbol, the
+ * vocabularies do not partition the embedding rows into consecutive
blocks
+ * that start with the reserved symbols, a transition repeats or
does not
+ * decode, the inventory lacks SHIFT or RIGHT_ARC, or a weight
dimension does
+ * not match.
+ */
+ FeedforwardNetwork(Map<String, Integer> wordIds, Map<String, Integer> tagIds,
+ Map<String, Integer> labelIds, String[] transitions, int embeddingSize,
+ float[][] embeddings, float[][] hiddenWeights, float[] hiddenBias,
+ float[][] outputWeights, float[] outputBias) {
+ this.wordIds = Map.copyOf(wordIds);
+ this.tagIds = Map.copyOf(tagIds);
+ this.labelIds = Map.copyOf(labelIds);
+ this.transitions = transitions.clone();
+ this.embeddingSize = embeddingSize;
+ this.embeddings = embeddings;
+ this.hiddenWeights = hiddenWeights;
+ this.hiddenBias = hiddenBias;
+ this.outputWeights = outputWeights;
+ this.outputBias = outputBias;
+
+ final long rows = (long) wordIds.size() + tagIds.size() + labelIds.size();
+ if (rows != embeddings.length) {
+ throw new IllegalArgumentException("embedding matrix has " +
embeddings.length
+ + " rows for " + rows + " vocabulary entries");
+ }
+ final boolean[] taken = new boolean[embeddings.length];
+ this.firstWordRow = validateVocabulary(this.wordIds, taken, "word
vocabulary",
+ RESERVED_WORD_SYMBOLS);
+ this.firstTagRow = validateVocabulary(this.tagIds, taken, "tag vocabulary",
+ RESERVED_TAG_SYMBOLS);
+ this.firstLabelRow = validateVocabulary(this.labelIds, taken, "label
vocabulary",
+ RESERVED_LABEL_SYMBOLS);
+
+ this.decodedTransitions = new Transition[this.transitions.length];
Review Comment:
Moved to Transition.decodeInventory so both parsers share it and give the
same messages (e.g. "not a transition: JUMP").
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardNetwork.java:
##########
@@ -0,0 +1,1011 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.ml.ArrayMath;
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.StringUtil;
+import opennlp.tools.util.model.ArtifactSerializer;
+
+/**
+ * The weights of the feedforward transition parser: embeddings for words,
tags, and arc
+ * labels, one hidden layer with cube activation, and a transition output
layer, after
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The network holds every forward computation the parser and the trainer
share:
+ * {@link #addContribution} is the per-position hidden sum, {@link
#outputScores} the
+ * output layer, {@link #score} the complete inference pass, and
+ * {@link #logSoftmaxInPlace} the normalization of beam decoding. The trainer
writes
+ * its updates into the live arrays the accessors return; the parser never
modifies
+ * them, so a network that is not being trained can be shared between
threads.</p>
+ *
+ * <p>Each vocabulary occupies one consecutive block of embedding rows and
starts with
+ * its reserved symbols, so unknown symbols, absent positions, and the
artificial root
+ * always have a row. Words are matched after {@link #normalize(String)
normalization}.</p>
+ *
+ * @see FeedforwardDependencyModel
+ * @since 3.0.0
+ */
+final class FeedforwardNetwork {
+
+ /** The vocabulary key of every word, tag, or label the network has no
embedding row for. */
+ static final String UNKNOWN = "*UNK*";
+
+ /** The reserved word symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_WORD_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved tag symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_TAG_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved label symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_LABEL_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE};
+
+ /** The embedding rows the reserved symbols of all three vocabularies take.
*/
+ static final int RESERVED_ROWS = RESERVED_WORD_SYMBOLS.length
+ + RESERVED_TAG_SYMBOLS.length + RESERVED_LABEL_SYMBOLS.length;
+
+ /** The embedding row of the unknown symbol in each vocabulary block. */
+ private static final int UNKNOWN_OFFSET = 0;
+
+ /** The embedding row of the absent-position symbol in each vocabulary
block. */
+ private static final int NONE_OFFSET = 1;
+
+ /** The embedding row of the root symbol in the word and tag blocks. */
+ private static final int ROOT_OFFSET = 2;
+
+ private final Map<String, Integer> wordIds;
+ private final Map<String, Integer> tagIds;
+ private final Map<String, Integer> labelIds;
+
+ /** The first embedding row of each vocabulary, indexed like the
vocabularies above. */
+ private final int firstWordRow;
+ private final int firstTagRow;
+ private final int firstLabelRow;
+
+ private final String[] transitions;
+ private final Transition[] decodedTransitions;
+ private final Map<Transition, Integer> transitionIds;
+
+ private final int embeddingSize;
+ private final float[][] embeddings;
+ private final float[][] hiddenWeights;
+ private final float[] hiddenBias;
+ private final float[][] outputWeights;
+ private final float[] outputBias;
+
+ /**
+ * Assembles a network from its vocabularies and weights, copying the maps
so later
+ * changes by the caller do not reach the network.
+ *
+ * @param wordIds The word vocabulary, mapping each normalized word to its
row.
+ * @param tagIds The tag vocabulary.
+ * @param labelIds The dependency label vocabulary.
+ * @param transitions The transition inventory, indexed by output row.
+ * @param embeddingSize The embedding dimensionality.
+ * @param embeddings The embedding rows for words, tags, and labels.
+ * @param hiddenWeights The hidden layer weights.
+ * @param hiddenBias The hidden layer bias.
+ * @param outputWeights The output layer weights.
+ * @param outputBias The output layer bias.
+ * @throws IllegalArgumentException Thrown if a vocabulary lacks a reserved
symbol, the
+ * vocabularies do not partition the embedding rows into consecutive
blocks
+ * that start with the reserved symbols, a transition repeats or
does not
+ * decode, the inventory lacks SHIFT or RIGHT_ARC, or a weight
dimension does
+ * not match.
+ */
+ FeedforwardNetwork(Map<String, Integer> wordIds, Map<String, Integer> tagIds,
+ Map<String, Integer> labelIds, String[] transitions, int embeddingSize,
+ float[][] embeddings, float[][] hiddenWeights, float[] hiddenBias,
+ float[][] outputWeights, float[] outputBias) {
+ this.wordIds = Map.copyOf(wordIds);
+ this.tagIds = Map.copyOf(tagIds);
+ this.labelIds = Map.copyOf(labelIds);
+ this.transitions = transitions.clone();
+ this.embeddingSize = embeddingSize;
+ this.embeddings = embeddings;
+ this.hiddenWeights = hiddenWeights;
+ this.hiddenBias = hiddenBias;
+ this.outputWeights = outputWeights;
+ this.outputBias = outputBias;
+
+ final long rows = (long) wordIds.size() + tagIds.size() + labelIds.size();
+ if (rows != embeddings.length) {
+ throw new IllegalArgumentException("embedding matrix has " +
embeddings.length
+ + " rows for " + rows + " vocabulary entries");
+ }
+ final boolean[] taken = new boolean[embeddings.length];
+ this.firstWordRow = validateVocabulary(this.wordIds, taken, "word
vocabulary",
+ RESERVED_WORD_SYMBOLS);
+ this.firstTagRow = validateVocabulary(this.tagIds, taken, "tag vocabulary",
+ RESERVED_TAG_SYMBOLS);
+ this.firstLabelRow = validateVocabulary(this.labelIds, taken, "label
vocabulary",
+ RESERVED_LABEL_SYMBOLS);
+
+ this.decodedTransitions = new Transition[this.transitions.length];
+ this.transitionIds = new HashMap<>();
+ boolean hasShift = false;
+ boolean hasRightArc = false;
+ for (int i = 0; i < this.transitions.length; i++) {
+ try {
+ decodedTransitions[i] = Transition.decode(this.transitions[i]);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("invalid transition: " +
this.transitions[i], e);
+ }
+ if (transitionIds.put(decodedTransitions[i], i) != null) {
+ throw new IllegalArgumentException("duplicate transition: " +
this.transitions[i]);
+ }
+ hasShift |= decodedTransitions[i].type() == Transition.Type.SHIFT;
+ hasRightArc |= decodedTransitions[i].type() == Transition.Type.RIGHT_ARC;
+ }
+ if (!hasShift) {
+ throw new IllegalArgumentException("transition inventory has no SHIFT
action");
+ }
+ if (!hasRightArc) {
+ throw new IllegalArgumentException("transition inventory has no
RIGHT_ARC action");
+ }
+
+ final int inputSize = FeedforwardContext.FEATURE_COUNT * embeddingSize;
+ requireColumns(embeddings, embeddingSize, "embedding matrix");
+ requireColumns(hiddenWeights, inputSize, "hidden matrix");
+ if (hiddenBias.length != hiddenWeights.length) {
+ throw new IllegalArgumentException("hidden bias length is " +
hiddenBias.length
+ + ", expected " + hiddenWeights.length);
+ }
+ if (outputWeights.length != this.transitions.length) {
+ throw new IllegalArgumentException("output matrix row count is "
+ + outputWeights.length + ", expected " + this.transitions.length);
+ }
+ requireColumns(outputWeights, hiddenWeights.length, "output matrix");
+ if (outputBias.length != this.transitions.length) {
+ throw new IllegalArgumentException("output bias length is " +
outputBias.length
+ + ", expected " + this.transitions.length);
+ }
+ }
+
+ /**
+ * Checks that one vocabulary starts with its reserved symbols and occupies
one
+ * consecutive block of embedding rows that no other vocabulary uses.
+ *
+ * @param ids The vocabulary to check.
+ * @param taken The rows claimed so far across all vocabularies; updated in
place.
+ * @param label The vocabulary name used in error messages.
+ * @param reserved The reserved symbols, in the order of their rows.
+ * @return The first row of the block.
+ * @throws IllegalArgumentException Thrown if the block is not consecutive,
a row is
+ * already taken, or a reserved symbol is missing or misplaced.
+ */
+ private static int validateVocabulary(Map<String, Integer> ids, boolean[]
taken,
+ String label, String[] reserved) {
+ int min = Integer.MAX_VALUE;
+ int max = Integer.MIN_VALUE;
+ for (final Map.Entry<String, Integer> entry : ids.entrySet()) {
+ final int id = entry.getValue();
+ if (id < 0 || id >= taken.length) {
+ throw new IllegalArgumentException(label + " id out of range for "
+ + entry.getKey() + ": " + id);
+ }
+ if (taken[id]) {
+ throw new IllegalArgumentException("duplicate embedding id: " + id);
+ }
+ taken[id] = true;
+ min = Math.min(min, id);
+ max = Math.max(max, id);
+ }
+ if (ids.size() < reserved.length || max - min + 1 != ids.size()) {
+ throw new IllegalArgumentException(label + " must occupy one consecutive
block of "
+ + "embedding rows that starts with its reserved symbols");
+ }
+ for (int i = 0; i < reserved.length; i++) {
+ final Integer id = ids.get(reserved[i]);
+ if (id == null || id != min + i) {
+ throw new IllegalArgumentException(label + " must have " + reserved[i]
+ + " at row " + (min + i));
+ }
+ }
+ return min;
+ }
+
+ /**
+ * @param matrix The matrix to check.
+ * @param columns The required width of every row.
+ * @param label The matrix name used in error messages.
+ * @throws IllegalArgumentException Thrown if a row has another width.
+ */
+ private static void requireColumns(float[][] matrix, int columns, String
label) {
+ for (final float[] row : matrix) {
+ if (row.length != columns) {
+ throw new IllegalArgumentException(label + " column count is " +
row.length
+ + ", expected " + columns);
+ }
+ }
+ }
+
+ /**
+ * Scores every transition for a configuration described by embedding rows.
+ *
+ * @param features The embedding rows of the configuration, as produced by
+ * {@link FeedforwardContext#featureIds}. Must not be {@code
null}.
+ * @param cache The contribution cache to read, or {@code null} to compute
every
+ * contribution.
+ * @return One unnormalized score per transition, indexed like
+ * {@link #transitions()}. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code features} is {@code
null}, does not
+ * have the required length, or contains an invalid embedding row.
+ * @throws IllegalStateException Thrown if the network produces a non-finite
score.
+ */
+ double[] score(int[] features, ContributionCache cache) {
+ ParamChecks.requireNonNullArg(features, "features");
+ if (features.length != FeedforwardContext.FEATURE_COUNT) {
+ throw new IllegalArgumentException("features must contain "
+ + FeedforwardContext.FEATURE_COUNT + " embedding indices");
+ }
+ for (int i = 0; i < features.length; i++) {
+ if (features[i] < 0 || features[i] >= embeddings.length) {
+ throw new IllegalArgumentException(
+ "feature embedding out of range at index " + i + ": " +
features[i]);
+ }
+ }
+ final double[] hidden = new double[hiddenBias.length];
Review Comment:
score now writes into buffers passed by the caller. Decoders allocate them
once per sentence, refinement reuses the optimizer buffers.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardNetwork.java:
##########
@@ -0,0 +1,1011 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.ml.ArrayMath;
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.StringUtil;
+import opennlp.tools.util.model.ArtifactSerializer;
+
+/**
+ * The weights of the feedforward transition parser: embeddings for words,
tags, and arc
+ * labels, one hidden layer with cube activation, and a transition output
layer, after
+ * <a href="https://aclanthology.org/D14-1082/";>Chen and Manning (2014)</a>.
+ *
+ * <p>The network holds every forward computation the parser and the trainer
share:
+ * {@link #addContribution} is the per-position hidden sum, {@link
#outputScores} the
+ * output layer, {@link #score} the complete inference pass, and
+ * {@link #logSoftmaxInPlace} the normalization of beam decoding. The trainer
writes
+ * its updates into the live arrays the accessors return; the parser never
modifies
+ * them, so a network that is not being trained can be shared between
threads.</p>
+ *
+ * <p>Each vocabulary occupies one consecutive block of embedding rows and
starts with
+ * its reserved symbols, so unknown symbols, absent positions, and the
artificial root
+ * always have a row. Words are matched after {@link #normalize(String)
normalization}.</p>
+ *
+ * @see FeedforwardDependencyModel
+ * @since 3.0.0
+ */
+final class FeedforwardNetwork {
+
+ /** The vocabulary key of every word, tag, or label the network has no
embedding row for. */
+ static final String UNKNOWN = "*UNK*";
+
+ /** The reserved word symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_WORD_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved tag symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_TAG_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE};
+
+ /** The reserved label symbols, in the order of their embedding rows. */
+ static final String[] RESERVED_LABEL_SYMBOLS = {UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE};
+
+ /** The embedding rows the reserved symbols of all three vocabularies take.
*/
+ static final int RESERVED_ROWS = RESERVED_WORD_SYMBOLS.length
+ + RESERVED_TAG_SYMBOLS.length + RESERVED_LABEL_SYMBOLS.length;
+
+ /** The embedding row of the unknown symbol in each vocabulary block. */
+ private static final int UNKNOWN_OFFSET = 0;
+
+ /** The embedding row of the absent-position symbol in each vocabulary
block. */
+ private static final int NONE_OFFSET = 1;
+
+ /** The embedding row of the root symbol in the word and tag blocks. */
+ private static final int ROOT_OFFSET = 2;
+
+ private final Map<String, Integer> wordIds;
+ private final Map<String, Integer> tagIds;
+ private final Map<String, Integer> labelIds;
+
+ /** The first embedding row of each vocabulary, indexed like the
vocabularies above. */
+ private final int firstWordRow;
+ private final int firstTagRow;
+ private final int firstLabelRow;
+
+ private final String[] transitions;
+ private final Transition[] decodedTransitions;
+ private final Map<Transition, Integer> transitionIds;
+
+ private final int embeddingSize;
+ private final float[][] embeddings;
+ private final float[][] hiddenWeights;
+ private final float[] hiddenBias;
+ private final float[][] outputWeights;
+ private final float[] outputBias;
+
+ /**
+ * Assembles a network from its vocabularies and weights, copying the maps
so later
+ * changes by the caller do not reach the network.
+ *
+ * @param wordIds The word vocabulary, mapping each normalized word to its
row.
+ * @param tagIds The tag vocabulary.
+ * @param labelIds The dependency label vocabulary.
+ * @param transitions The transition inventory, indexed by output row.
+ * @param embeddingSize The embedding dimensionality.
+ * @param embeddings The embedding rows for words, tags, and labels.
+ * @param hiddenWeights The hidden layer weights.
+ * @param hiddenBias The hidden layer bias.
+ * @param outputWeights The output layer weights.
+ * @param outputBias The output layer bias.
+ * @throws IllegalArgumentException Thrown if a vocabulary lacks a reserved
symbol, the
+ * vocabularies do not partition the embedding rows into consecutive
blocks
+ * that start with the reserved symbols, a transition repeats or
does not
+ * decode, the inventory lacks SHIFT or RIGHT_ARC, or a weight
dimension does
+ * not match.
+ */
+ FeedforwardNetwork(Map<String, Integer> wordIds, Map<String, Integer> tagIds,
+ Map<String, Integer> labelIds, String[] transitions, int embeddingSize,
+ float[][] embeddings, float[][] hiddenWeights, float[] hiddenBias,
+ float[][] outputWeights, float[] outputBias) {
+ this.wordIds = Map.copyOf(wordIds);
+ this.tagIds = Map.copyOf(tagIds);
+ this.labelIds = Map.copyOf(labelIds);
+ this.transitions = transitions.clone();
+ this.embeddingSize = embeddingSize;
+ this.embeddings = embeddings;
+ this.hiddenWeights = hiddenWeights;
+ this.hiddenBias = hiddenBias;
+ this.outputWeights = outputWeights;
+ this.outputBias = outputBias;
+
+ final long rows = (long) wordIds.size() + tagIds.size() + labelIds.size();
+ if (rows != embeddings.length) {
+ throw new IllegalArgumentException("embedding matrix has " +
embeddings.length
+ + " rows for " + rows + " vocabulary entries");
+ }
+ final boolean[] taken = new boolean[embeddings.length];
+ this.firstWordRow = validateVocabulary(this.wordIds, taken, "word
vocabulary",
+ RESERVED_WORD_SYMBOLS);
+ this.firstTagRow = validateVocabulary(this.tagIds, taken, "tag vocabulary",
+ RESERVED_TAG_SYMBOLS);
+ this.firstLabelRow = validateVocabulary(this.labelIds, taken, "label
vocabulary",
+ RESERVED_LABEL_SYMBOLS);
+
+ this.decodedTransitions = new Transition[this.transitions.length];
+ this.transitionIds = new HashMap<>();
+ boolean hasShift = false;
+ boolean hasRightArc = false;
+ for (int i = 0; i < this.transitions.length; i++) {
+ try {
+ decodedTransitions[i] = Transition.decode(this.transitions[i]);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("invalid transition: " +
this.transitions[i], e);
+ }
+ if (transitionIds.put(decodedTransitions[i], i) != null) {
+ throw new IllegalArgumentException("duplicate transition: " +
this.transitions[i]);
+ }
+ hasShift |= decodedTransitions[i].type() == Transition.Type.SHIFT;
+ hasRightArc |= decodedTransitions[i].type() == Transition.Type.RIGHT_ARC;
+ }
+ if (!hasShift) {
+ throw new IllegalArgumentException("transition inventory has no SHIFT
action");
+ }
+ if (!hasRightArc) {
+ throw new IllegalArgumentException("transition inventory has no
RIGHT_ARC action");
+ }
+
+ final int inputSize = FeedforwardContext.FEATURE_COUNT * embeddingSize;
+ requireColumns(embeddings, embeddingSize, "embedding matrix");
+ requireColumns(hiddenWeights, inputSize, "hidden matrix");
+ if (hiddenBias.length != hiddenWeights.length) {
+ throw new IllegalArgumentException("hidden bias length is " +
hiddenBias.length
+ + ", expected " + hiddenWeights.length);
+ }
+ if (outputWeights.length != this.transitions.length) {
+ throw new IllegalArgumentException("output matrix row count is "
+ + outputWeights.length + ", expected " + this.transitions.length);
+ }
+ requireColumns(outputWeights, hiddenWeights.length, "output matrix");
+ if (outputBias.length != this.transitions.length) {
+ throw new IllegalArgumentException("output bias length is " +
outputBias.length
+ + ", expected " + this.transitions.length);
+ }
+ }
+
+ /**
+ * Checks that one vocabulary starts with its reserved symbols and occupies
one
+ * consecutive block of embedding rows that no other vocabulary uses.
+ *
+ * @param ids The vocabulary to check.
+ * @param taken The rows claimed so far across all vocabularies; updated in
place.
+ * @param label The vocabulary name used in error messages.
+ * @param reserved The reserved symbols, in the order of their rows.
+ * @return The first row of the block.
+ * @throws IllegalArgumentException Thrown if the block is not consecutive,
a row is
+ * already taken, or a reserved symbol is missing or misplaced.
+ */
+ private static int validateVocabulary(Map<String, Integer> ids, boolean[]
taken,
+ String label, String[] reserved) {
+ int min = Integer.MAX_VALUE;
+ int max = Integer.MIN_VALUE;
+ for (final Map.Entry<String, Integer> entry : ids.entrySet()) {
+ final int id = entry.getValue();
+ if (id < 0 || id >= taken.length) {
+ throw new IllegalArgumentException(label + " id out of range for "
+ + entry.getKey() + ": " + id);
+ }
+ if (taken[id]) {
+ throw new IllegalArgumentException("duplicate embedding id: " + id);
+ }
+ taken[id] = true;
+ min = Math.min(min, id);
+ max = Math.max(max, id);
+ }
+ if (ids.size() < reserved.length || max - min + 1 != ids.size()) {
+ throw new IllegalArgumentException(label + " must occupy one consecutive
block of "
+ + "embedding rows that starts with its reserved symbols");
+ }
+ for (int i = 0; i < reserved.length; i++) {
+ final Integer id = ids.get(reserved[i]);
+ if (id == null || id != min + i) {
+ throw new IllegalArgumentException(label + " must have " + reserved[i]
+ + " at row " + (min + i));
+ }
+ }
+ return min;
+ }
+
+ /**
+ * @param matrix The matrix to check.
+ * @param columns The required width of every row.
+ * @param label The matrix name used in error messages.
+ * @throws IllegalArgumentException Thrown if a row has another width.
+ */
+ private static void requireColumns(float[][] matrix, int columns, String
label) {
+ for (final float[] row : matrix) {
+ if (row.length != columns) {
+ throw new IllegalArgumentException(label + " column count is " +
row.length
+ + ", expected " + columns);
+ }
+ }
+ }
+
+ /**
+ * Scores every transition for a configuration described by embedding rows.
+ *
+ * @param features The embedding rows of the configuration, as produced by
+ * {@link FeedforwardContext#featureIds}. Must not be {@code
null}.
+ * @param cache The contribution cache to read, or {@code null} to compute
every
+ * contribution.
+ * @return One unnormalized score per transition, indexed like
+ * {@link #transitions()}. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code features} is {@code
null}, does not
+ * have the required length, or contains an invalid embedding row.
+ * @throws IllegalStateException Thrown if the network produces a non-finite
score.
+ */
+ double[] score(int[] features, ContributionCache cache) {
+ ParamChecks.requireNonNullArg(features, "features");
+ if (features.length != FeedforwardContext.FEATURE_COUNT) {
+ throw new IllegalArgumentException("features must contain "
+ + FeedforwardContext.FEATURE_COUNT + " embedding indices");
+ }
+ for (int i = 0; i < features.length; i++) {
+ if (features[i] < 0 || features[i] >= embeddings.length) {
+ throw new IllegalArgumentException(
+ "feature embedding out of range at index " + i + ": " +
features[i]);
+ }
+ }
+ final double[] hidden = new double[hiddenBias.length];
+ preActivations(features, cache, hidden);
+ for (int j = 0; j < hidden.length; j++) {
+ hidden[j] = hidden[j] * hidden[j] * hidden[j];
+ }
+ final double[] scores = new double[transitions.length];
+ outputScores(hidden, scores);
+ for (final double score : scores) {
+ if (!Double.isFinite(score)) {
+ throw new IllegalStateException("the model produced a non-finite
transition score");
+ }
+ }
+ return scores;
+ }
+
+ /**
+ * Computes the hidden pre-activations of a configuration: the bias plus the
+ * contribution of every feature position.
+ *
+ * @param features The embedding rows of the configuration, one per position.
+ * @param cache The contribution cache to read, or {@code null} to compute
every
+ * contribution.
+ * @param pre Receives the pre-activations; overwritten.
+ */
+ void preActivations(int[] features, ContributionCache cache, double[] pre) {
+ for (int j = 0; j < pre.length; j++) {
+ pre[j] = hiddenBias[j];
+ }
+ for (int f = 0; f < features.length; f++) {
+ final double[] cached = cache == null ? null : cache.contribution(f,
features[f]);
+ if (cached != null) {
+ for (int j = 0; j < pre.length; j++) {
+ pre[j] += cached[j];
+ }
+ } else {
+ addContribution(f, features[f], pre);
+ }
+ }
+ }
+
+ /**
+ * Adds the hidden-layer contribution of one embedding row at one feature
position.
+ *
+ * @param position The feature position.
+ * @param row The embedding row at that position.
+ * @param out The hidden vector to add to.
+ */
+ void addContribution(int position, int row, double[] out) {
+ final float[] embedding = embeddings[row];
+ final int offset = position * embeddingSize;
+ for (int j = 0; j < out.length; j++) {
+ final float[] weights = hiddenWeights[j];
+ double sum = 0.0;
+ for (int d = 0; d < embeddingSize; d++) {
+ sum += (double) weights[offset + d] * embedding[d];
+ }
+ out[j] += sum;
+ }
+ }
+
+ /**
+ * Applies the output layer to hidden activations.
+ *
+ * @param hidden The hidden activations after the cube.
+ * @param scores Receives one raw score per transition; overwritten.
+ */
+ void outputScores(double[] hidden, double[] scores) {
+ for (int o = 0; o < scores.length; o++) {
+ final float[] row = outputWeights[o];
+ double sum = outputBias[o];
+ for (int j = 0; j < hidden.length; j++) {
+ sum += row[j] * hidden[j];
+ }
+ scores[o] = sum;
+ }
+ }
+
+ /**
+ * Turns finite raw scores into log-probabilities in place. The maximum is
removed
+ * before the sum of exponentials so a large common offset does not lose the
+ * normalizer to rounding.
+ *
+ * @param scores The raw scores, overwritten with their log-softmax. Must
not be
+ * empty.
+ */
+ static void logSoftmaxInPlace(double[] scores) {
+ final double max = ArrayMath.max(scores);
+ for (int i = 0; i < scores.length; i++) {
+ scores[i] -= max;
+ }
+ final double logSum = ArrayMath.logSumOfExps(scores);
+ for (int i = 0; i < scores.length; i++) {
+ scores[i] -= logSum;
+ }
+ }
+
+ /**
+ * Resolves the tokens of a sentence to their word rows once, so that
extracting the
+ * features of every configuration only indexes.
+ *
+ * @param tokens The sentence tokens. Must not contain {@code null}.
+ * @return The word row per token. Never {@code null}.
+ */
+ int[] wordRows(String[] tokens) {
+ final int[] rows = new int[tokens.length];
+ for (int i = 0; i < tokens.length; i++) {
+ rows[i] = lookup(wordIds, normalize(tokens[i]), firstWordRow);
+ }
+ return rows;
+ }
+
+ /**
+ * Resolves the tags of a sentence to their tag rows once.
+ *
+ * @param tags The part-of-speech tags. Must not contain {@code null}.
+ * @return The tag row per tag. Never {@code null}.
+ */
+ int[] tagRows(String[] tags) {
+ final int[] rows = new int[tags.length];
+ for (int i = 0; i < tags.length; i++) {
+ rows[i] = lookup(tagIds, tags[i], firstTagRow);
+ }
+ return rows;
+ }
+
+ /**
+ * @param relation The arc label of an attached dependent, or {@code null}
for an
+ * absent position.
+ * @return The label row, falling back to the unknown row.
+ */
+ int labelRow(String relation) {
+ return lookup(labelIds, relation, firstLabelRow);
+ }
+
+ /**
+ * @return The word row standing for the artificial root.
+ */
+ int rootWordRow() {
+ return firstWordRow + ROOT_OFFSET;
+ }
+
+ /**
+ * @return The word row standing for an absent position.
+ */
+ int noneWordRow() {
+ return firstWordRow + NONE_OFFSET;
+ }
+
+ /**
+ * @return The tag row standing for the artificial root.
+ */
+ int rootTagRow() {
+ return firstTagRow + ROOT_OFFSET;
+ }
+
+ /**
+ * @return The tag row standing for an absent position.
+ */
+ int noneTagRow() {
+ return firstTagRow + NONE_OFFSET;
+ }
+
+ /**
+ * Resolves a symbol to its embedding row: absences map to the reserved
absent
+ * symbol, symbols without a row of their own to the reserved unknown symbol.
+ *
+ * @param ids The vocabulary to resolve against.
+ * @param symbol The symbol to resolve, or {@code null} for an absent
position.
+ * @param firstRow The first row of the vocabulary block.
+ * @return The embedding row of the symbol or of its fallback.
+ */
+ private static int lookup(Map<String, Integer> ids, String symbol, int
firstRow) {
+ if (symbol == null) {
+ return firstRow + NONE_OFFSET;
+ }
+ final Integer id = ids.get(symbol);
+ return id == null ? firstRow + UNKNOWN_OFFSET : id;
+ }
+
+ /**
+ * Lowercases a word symbol with {@link
StringUtil#toLowerCase(CharSequence)}, which
+ * maps per code point and applies the Greek final sigma rule; reserved
symbols and
+ * absences pass through.
+ *
+ * @param word The word to normalize. May be {@code null}.
+ * @return The vocabulary key of {@code word}, or {@code null} if {@code
word} is
+ * {@code null}.
+ */
+ static String normalize(String word) {
+ if (word == null || isReservedSymbol(word)) {
+ return word;
+ }
+ return StringUtil.toLowerCase(word);
+ }
+
+ /**
+ * @param symbol The symbol to test. May be {@code null}.
+ * @return {@code true} if {@code symbol} is one of the reserved feature
values.
+ */
+ static boolean isReservedSymbol(String symbol) {
+ // the word symbols include the tag and label symbols
+ for (final String reserved : RESERVED_WORD_SYMBOLS) {
+ if (reserved.equals(symbol)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks that a symbol fits the length the {@link Serializer} reads back.
+ *
+ * @param symbol The symbol to check.
+ * @param label The vocabulary name used in error messages.
+ * @throws IllegalArgumentException Thrown if the UTF-8 form of {@code
symbol} is
+ * longer than {@link ResourceLimits#MAX_ENTRIES} bytes.
+ */
+ static void requireStorable(String symbol, String label) {
+ // a UTF-16 unit encodes to at most three UTF-8 bytes, so short symbols
need no encoding
+ if ((long) symbol.length() * 3 <= ResourceLimits.MAX_ENTRIES) {
+ return;
+ }
+ final long bytes = symbol.getBytes(StandardCharsets.UTF_8).length;
+ if (bytes > ResourceLimits.MAX_ENTRIES) {
+ throw new IllegalArgumentException(label + " symbol length " + bytes
+ + " exceeds the limit of " + ResourceLimits.MAX_ENTRIES + " UTF-8
bytes");
+ }
+ }
+
+ /**
+ * @return The transition outcome strings by output index. Never {@code
null}.
+ */
+ String[] transitions() {
+ return transitions.clone();
+ }
+
+ /**
+ * @return The decoded transitions by output index. Never {@code null}.
+ */
+ Transition[] decodedTransitions() {
+ return decodedTransitions.clone();
+ }
+
+ /**
+ * @param transition The transition to look up.
+ * @return Its output index, or {@code -1} if the inventory lacks it.
+ */
+ int transitionIndex(Transition transition) {
+ final Integer id = transitionIds.get(transition);
+ return id == null ? -1 : id;
+ }
+
+ /**
+ * Creates an independent copy of this network: the weights are deep-copied
and the
+ * vocabularies are shared because their maps are immutable.
+ *
+ * @return A copy sharing no mutable state with this network. Never {@code
null}.
+ */
+ FeedforwardNetwork copy() {
+ return new FeedforwardNetwork(wordIds, tagIds, labelIds, transitions,
+ embeddingSize, copyOf(embeddings), copyOf(hiddenWeights),
hiddenBias.clone(),
+ copyOf(outputWeights), outputBias.clone());
+ }
+
+ /**
+ * Deep-copies a matrix, row by row.
+ *
+ * @param matrix The matrix to copy.
+ * @return A copy sharing no rows with {@code matrix}. Never {@code null}.
+ */
+ private static float[][] copyOf(float[][] matrix) {
+ final float[][] copy = new float[matrix.length][];
+ for (int r = 0; r < matrix.length; r++) {
+ copy[r] = matrix[r].clone();
+ }
+ return copy;
+ }
+
+ /**
+ * @param vocabularyRows The combined word, tag, and label row count.
+ * @param transitionCount The output layer width.
+ * @param embeddingSize The embedding dimensionality.
+ * @param hiddenSize The hidden layer width.
+ * @return The number of float values a network of that shape stores.
+ */
+ static long floatValues(long vocabularyRows, long transitionCount, long
embeddingSize,
+ long hiddenSize) {
+ final long inputSize = FeedforwardContext.FEATURE_COUNT * embeddingSize;
+ return vocabularyRows * embeddingSize + hiddenSize * inputSize + hiddenSize
+ + transitionCount * hiddenSize + transitionCount;
+ }
+
+ /**
+ * @return The immutable map from a normalized word to its embedding row.
Never {@code null}.
+ */
+ Map<String, Integer> wordIds() {
+ return wordIds;
+ }
+
+ /**
+ * @return The immutable map from a tag to its embedding row. Never {@code
null}.
+ */
+ Map<String, Integer> tagIds() {
+ return tagIds;
+ }
+
+ /**
+ * @return The immutable map from an arc label to its embedding row. Never
{@code null}.
+ */
+ Map<String, Integer> labelIds() {
+ return labelIds;
+ }
+
+ /**
+ * @return The first embedding row of the word vocabulary block.
+ */
+ int firstWordRow() {
+ return firstWordRow;
+ }
+
+ /**
+ * @return The first embedding row of the tag vocabulary block.
+ */
+ int firstTagRow() {
+ return firstTagRow;
+ }
+
+ /**
+ * @return The first embedding row of the label vocabulary block.
+ */
+ int firstLabelRow() {
+ return firstLabelRow;
+ }
+
+ /**
+ * @return The width of one embedding row.
+ */
+ int embeddingSize() {
+ return embeddingSize;
+ }
+
+ /**
+ * @return The hidden layer width.
+ */
+ int hiddenSize() {
+ return hiddenBias.length;
+ }
+
+ /**
+ * @return The live embedding matrix, one row per vocabulary entry, not a
copy: the
+ * trainer writes its updates into it. Never {@code null}.
+ */
+ float[][] embeddings() {
+ return embeddings;
+ }
+
+ /**
+ * @return The live hidden layer weights, not a copy. Never {@code null}.
+ */
+ float[][] hiddenWeights() {
+ return hiddenWeights;
+ }
+
+ /**
+ * @return The live hidden layer bias, not a copy. Never {@code null}.
+ */
+ float[] hiddenBias() {
+ return hiddenBias;
+ }
+
+ /**
+ * @return The live output layer weights, one row per transition, not a
copy. Never
+ * {@code null}.
+ */
+ float[][] outputWeights() {
+ return outputWeights;
+ }
+
+ /**
+ * @return The live output layer bias, one entry per transition, not a copy.
Never
+ * {@code null}.
+ */
+ float[] outputBias() {
+ return outputBias;
+ }
+
+ /**
+ * Reads and writes a network as a {@link FeedforwardDependencyModel}
artifact: a
+ * four-byte magic, an {@code int} format version, the three vocabularies,
the
+ * transition inventory, the embedding size, and the weight matrices and
vectors.
+ * Symbols are stored as a byte length followed by UTF-8 bytes, so their
length is
Review Comment:
Reworded to state the limit directly. Fixed the two test comments to 70000
bytes.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/FeedforwardDependencyModel.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Serial;
+import java.nio.file.Path;
+import java.util.Map;
+
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ParamChecks;
+import opennlp.tools.util.model.ArtifactSerializer;
+import opennlp.tools.util.model.BaseModel;
+
+/**
+ * The persisted form of a trained {@link FeedforwardDependencyParser}: the
network
+ * weights plus the standard model manifest, serialized and loaded through the
same
+ * {@link BaseModel} machinery as every other tool model.
+ *
+ * <p>The weights are not modified after construction or loading. A model can
be
+ * shared between threads and between parsers; {@link
FeedforwardDependencyTrainer#refine
+ * refine} trains a copy and returns a new model.</p>
+ *
+ * @see FeedforwardDependencyParser
+ * @see FeedforwardDependencyTrainer
+ * @since 3.0.0
+ */
+public class FeedforwardDependencyModel extends BaseModel {
+
+ @Serial
+ private static final long serialVersionUID = 2746091883120417596L;
+
+ /** The component name recorded in the model manifest. */
+ private static final String COMPONENT_NAME = "FeedforwardDependencyParser";
+
+ /** The artifact map entry holding the network. */
+ static final String NETWORK_ENTRY_NAME = "feedforward.network";
+
+ /** The artifact extension the network serializer is registered under. */
+ private static final String NETWORK_EXTENSION = "network";
+
+ /**
+ * Initializes a {@link FeedforwardDependencyModel} from a trained network.
+ *
+ * @param languageCode The ISO language code of the training data. Must not
be
+ * {@code null}.
+ * @param network The trained network. Must not be {@code null}.
+ * @param manifestInfoEntries Additional entries for the manifest, or {@code
null}.
+ * @throws IllegalArgumentException Thrown if {@code languageCode} or {@code
network}
+ * is {@code null}.
+ */
+ FeedforwardDependencyModel(String languageCode, FeedforwardNetwork network,
+ Map<String, String> manifestInfoEntries) {
+ super(COMPONENT_NAME, ParamChecks.requireNonNullArg(languageCode,
"languageCode"),
+ manifestInfoEntries);
+ ParamChecks.requireNonNullArg(network, "network");
+ artifactMap.put(NETWORK_ENTRY_NAME, network);
+ checkArtifactMap();
+ }
+
+ /**
+ * Initializes a {@link FeedforwardDependencyModel} from a serialized model.
+ *
+ * @param in The stream to read the model from. Must not be {@code null}.
+ * @throws IOException Thrown if reading fails or the content is not a valid
model.
+ * @throws IllegalArgumentException Thrown if {@code in} is {@code null}.
Review Comment:
Added @throws to all three constructors for a model from another component.
##########
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/FeedforwardDependencyParserTest.java:
##########
@@ -0,0 +1,1150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import java.util.stream.Stream;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipInputStream;
+import java.util.zip.ZipOutputStream;
+
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.function.Executable;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.ml.BeamSearch;
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ObjectStreamUtils;
+import opennlp.tools.util.Parameters;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.StringUtil;
+import opennlp.tools.util.TrainingParameters;
+import opennlp.tools.util.model.BaseModel;
+
+import static opennlp.tools.depparse.DependencyTestSamples.LANGUAGE;
+import static opennlp.tools.depparse.DependencyTestSamples.MINIMAL_TRANSITIONS;
+import static opennlp.tools.depparse.DependencyTestSamples.SHE_EATS_FISH_TAGS;
+import static
opennlp.tools.depparse.DependencyTestSamples.SHE_EATS_FISH_TOKENS;
+import static opennlp.tools.depparse.DependencyTestSamples.SHE_EATS_FISH_TREE;
+import static opennlp.tools.depparse.DependencyTestSamples.THE_DOG_BARKS_TAGS;
+import static
opennlp.tools.depparse.DependencyTestSamples.THE_DOG_BARKS_TOKENS;
+import static opennlp.tools.depparse.DependencyTestSamples.THE_DOG_BARKS_TREE;
+import static opennlp.tools.depparse.DependencyTestSamples.UNSEEN_RELATION;
+import static opennlp.tools.depparse.DependencyTestSamples.corpus;
+import static
opennlp.tools.depparse.DependencyTestSamples.feedforwardParameters;
+import static opennlp.tools.depparse.DependencyTestSamples.nonProjectiveSample;
+import static
opennlp.tools.depparse.DependencyTestSamples.nonProjectiveSampleWithUnseenRelation;
+import static opennlp.tools.depparse.DependencyTestSamples.sample;
+import static opennlp.tools.depparse.DependencyTestSamples.trainFeedforward;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests feedforward training, parsing, refinement, the model package, and the
network
+ * format.
+ */
+public class FeedforwardDependencyParserTest {
+
+ /** The beam size of the beamed parsers under test. */
+ private static final int BEAM_SIZE = 4;
+
+ /** The refinement parameters: two passes at a small step size. */
+ private static final TrainingParameters REFINE_PARAMETERS =
feedforwardParameters(2, 0.01);
+
+ /** One-pass parameters for tests that only inspect the result shape. */
+ private static final TrainingParameters SINGLE_EPOCH_PARAMETERS =
feedforwardParameters(1, 0.01);
+
+ /** The reserved rows of the hand-written networks, one per reserved symbol.
*/
+ private static final int MINIMAL_ROWS = FeedforwardNetwork.RESERVED_ROWS;
+
+ /** The first tag row of the hand-written networks, after the reserved word
rows. */
+ private static final int FIRST_TAG_ROW =
FeedforwardNetwork.RESERVED_WORD_SYMBOLS.length;
+
+ /** The first label row of the hand-written networks, after the reserved tag
rows. */
+ private static final int FIRST_LABEL_ROW =
+ FIRST_TAG_ROW + FeedforwardNetwork.RESERVED_TAG_SYMBOLS.length;
+
+ /** An embedding width that every hand-written network uses. */
+ private static final int UNIT_WIDTH = 1;
+
+ /** The hidden matrix rows that push the smallest network over the cell
limit. */
+ private static final int OVERSIZED_HIDDEN_ROWS =
Math.min(ResourceLimits.MAX_ENTRIES,
+ ResourceLimits.MAX_MATRIX_CELLS / FeedforwardContext.FEATURE_COUNT + 1);
+
+ /** A symbol longer than the 65535 bytes {@code writeUTF} could store. */
+ private static final String LONG_SYMBOL = "x".repeat(70_000);
+
+ private static FeedforwardDependencyModel model;
+ private static FeedforwardDependencyParser parser;
+
+ /**
+ * @return The distinct training sentences, one per parameterized
invocation. Never
+ * {@code null}.
+ */
+ private static List<DependencySample> trainingSamples() {
+ return DependencyTestSamples.sentences();
+ }
+
+ /**
+ * Trains the shared model once for all tests.
+ *
+ * @throws IOException Thrown if reading the in-memory samples fails.
+ */
+ @BeforeAll
+ static void trainParser() throws IOException {
+ model = DependencyTestSamples.trainFeedforward();
+ parser = new FeedforwardDependencyParser(model);
+ }
+
+ /** Checks that the greedy parser reproduces a training sentence. */
+ @Test
+ void testMemorizesTrainingSentences() {
+ assertEquals(THE_DOG_BARKS_TREE, parser.parse(THE_DOG_BARKS_TOKENS,
THE_DOG_BARKS_TAGS));
+ }
+
+ /** Checks UAS and LAS of 1.0 on the training corpus. */
+ @Test
+ void testEvaluatorScoresPerfectlyOnTrainingData() throws IOException {
+ final DependencyEvaluator evaluator = new DependencyEvaluator(parser);
+ evaluator.evaluate(ObjectStreamUtils.createObjectStream(corpus()));
+ assertEquals(1.0d, evaluator.getUas());
+ assertEquals(1.0d, evaluator.getLas());
+ }
+
+ /** Checks that unseen words still parse to a tree with one root inside the
sentence. */
+ @Test
+ void testUnknownWordsStillYieldASingleRootedTree() {
+ final DependencyTree parsed = parser.parse(new String[] {"unseen",
"words"},
+ new String[] {"JJ", "NNS"});
+ assertEquals(2, parsed.size());
+ assertTrue(parsed.root() >= 0 && parsed.root() < parsed.size(),
parsed.toString());
+ assertEquals(DependencyArc.ROOT_HEAD, parsed.headOf(parsed.root()));
+ }
+
+ /** Checks that the transition inventory is exactly the sorted set the
oracle produced. */
+ @Test
+ void testTransitionInventoryMatchesTheOracle() {
+ final TreeSet<String> expected = new TreeSet<>();
+ for (final DependencySample sample : DependencyTestSamples.sentences()) {
+ for (final Transition transition :
ArcStandardOracle.transitions(sample.getTree())) {
+ expected.add(transition.encode());
+ }
+ }
+ assertEquals(new ArrayList<>(expected),
List.of(model.network().transitions()));
+ }
+
+ /**
+ * Checks that a beam of one yields the greedy parse for every training
sample.
+ *
+ * @param sample The training sample to parse both ways.
+ */
+ @ParameterizedTest(name = "beam of one matches greedy for {0}")
+ @MethodSource("trainingSamples")
+ void testBeamOfOneMatchesGreedy(DependencySample sample) {
+ final FeedforwardDependencyParser beamed = new
FeedforwardDependencyParser(model, 1);
+ assertEquals(parser.parse(sample.getTokens(), sample.getTags()),
+ beamed.parse(sample.getTokens(), sample.getTags()));
+ }
+
+ /** Checks that beam search reproduces a training sentence. */
+ @Test
+ void testBeamedParserReproducesTrainingSentences() {
+ final FeedforwardDependencyParser beamed = new
FeedforwardDependencyParser(model, BEAM_SIZE);
+ assertEquals(THE_DOG_BARKS_TREE, beamed.parse(THE_DOG_BARKS_TOKENS,
THE_DOG_BARKS_TAGS));
+ }
+
+ /** Checks that beam search is deterministic and single-rooted on unseen
words. */
+ @Test
+ void testBeamedParseIsDeterministicAndSingleRooted() {
+ final FeedforwardDependencyParser beamed = new
FeedforwardDependencyParser(model, 8);
+ final String[] tokens = {"unseen", "words", "everywhere"};
+ final String[] tags = {"JJ", "NNS", "RB"};
+ final DependencyTree first = beamed.parse(tokens, tags);
+ assertEquals(first, beamed.parse(tokens, tags));
+ assertEquals(3, first.size());
+ assertEquals(DependencyArc.ROOT_HEAD, first.headOf(first.root()));
+ }
+
+ /** Checks that a cached parser returns the parses of an uncached one. */
+ @Test
+ void testCachedParserMatchesTheUncachedParser() {
+ final FeedforwardDependencyParser cached =
+ new FeedforwardDependencyParser(model, BEAM_SIZE, 1L << 20);
+ final FeedforwardDependencyParser uncached =
+ new FeedforwardDependencyParser(model, BEAM_SIZE);
+ for (final DependencySample sample : DependencyTestSamples.sentences()) {
+ assertEquals(uncached.parse(sample.getTokens(), sample.getTags()),
+ cached.parse(sample.getTokens(), sample.getTags()));
+ }
+ assertNotNull(cached.cache());
+ assertTrue(cached.cache().entries() > 0);
+ assertNull(uncached.cache());
+ }
+
+ /** Checks that refinement keeps perfect scores on the training corpus. */
+ @Test
+ void testRefinementKeepsToyPerformance() throws IOException {
+ final FeedforwardDependencyModel local =
+ trainFeedforward(corpus(), feedforwardParameters(60, 0.05));
+ final TrainingParameters parameters = feedforwardParameters(2, 0.01);
+ parameters.put(BeamSearch.BEAM_SIZE_PARAMETER, 2);
+ final FeedforwardDependencyModel refined =
FeedforwardDependencyTrainer.refine(
+ local, ObjectStreamUtils.createObjectStream(corpus()), parameters);
+
+ final DependencyEvaluator evaluator =
+ new DependencyEvaluator(new FeedforwardDependencyParser(refined, 2));
+ evaluator.evaluate(ObjectStreamUtils.createObjectStream(corpus()));
+ assertEquals(1.0d, evaluator.getUas());
+ assertEquals(1.0d, evaluator.getLas());
+ assertEquals("2",
refined.getManifestProperty(BeamSearch.BEAM_SIZE_PARAMETER));
+ }
+
+ /** Checks that refinement fails loudly on a relation label the model never
saw. */
+ @Test
+ void testRefineRejectsAnUnknownRelation() {
+ final List<DependencySample> unseenRelation = List.of(
+ sample(THE_DOG_BARKS_TOKENS, THE_DOG_BARKS_TAGS,
+ new int[] {1, 2, -1}, new String[] {"det", UNSEEN_RELATION,
"root"}));
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardDependencyTrainer.refine(model,
+ ObjectStreamUtils.createObjectStream(unseenRelation),
REFINE_PARAMETERS));
+ assertEquals("unknown transition in the refinement samples: LEFT_ARC:" +
UNSEEN_RELATION,
+ e.getMessage());
+ }
+
+ /** Checks that refinement returns a distinct model and leaves the input
scores unchanged. */
+ @Test
+ void testRefineReturnsANewModelAndLeavesTheOriginalUntouched() throws
IOException {
+ final FeedforwardDependencyModel local =
+ trainFeedforward(corpus(), feedforwardParameters(60, 0.05));
+ final int[] features = initialFeatures(local.network(),
THE_DOG_BARKS_TOKENS,
+ THE_DOG_BARKS_TAGS);
+ final double[] before = local.network().score(features, null);
+
+ final FeedforwardDependencyModel refined =
FeedforwardDependencyTrainer.refine(
+ local, ObjectStreamUtils.createObjectStream(corpus()),
REFINE_PARAMETERS);
+
+ assertNotSame(local, refined);
+ assertArrayEquals(before, local.network().score(features, null));
+ assertFalse(Arrays.equals(before, refined.network().score(features,
null)));
+ assertEquals(LANGUAGE, refined.getLanguage());
+ }
+
+ /**
+ * Checks that non-projective samples are skipped before transition
validation and
+ * that the refined model still parses the projective sample.
+ */
+ @Test
+ void testNonProjectiveSampleWithUnknownRelationIsSkippedNotFatal() throws
IOException {
+ final List<DependencySample> mixed =
List.of(nonProjectiveSampleWithUnseenRelation(),
+ new DependencySample(THE_DOG_BARKS_TOKENS, THE_DOG_BARKS_TAGS,
THE_DOG_BARKS_TREE));
+ final FeedforwardDependencyModel refined =
FeedforwardDependencyTrainer.refine(
+ model, ObjectStreamUtils.createObjectStream(mixed),
SINGLE_EPOCH_PARAMETERS);
+ assertEquals(THE_DOG_BARKS_TREE, new FeedforwardDependencyParser(refined,
BEAM_SIZE)
+ .parse(THE_DOG_BARKS_TOKENS, THE_DOG_BARKS_TAGS));
+ }
+
+ /** Checks that the inventory of a model trained on mixed data omits
non-projective labels. */
+ @Test
+ void testTrainingOmitsNonProjectiveLabels() throws IOException {
+ final List<DependencySample> mixed = new ArrayList<>(corpus());
+ mixed.add(nonProjectiveSampleWithUnseenRelation());
+ final FeedforwardDependencyModel trained = trainFeedforward(mixed,
SINGLE_EPOCH_PARAMETERS);
+ for (final String transition : trained.network().transitions()) {
+ assertFalse(transition.contains(UNSEEN_RELATION), transition);
+ }
+ }
+
+ /**
+ * Checks that training on samples without an arc-standard derivation fails
with the
+ * documented message instead of producing an empty model.
+ */
+ @Test
+ void testTrainingWithOnlyNonProjectiveSamplesFailsLoud() {
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> trainFeedforward(nonProjectiveCorpus(),
SINGLE_EPOCH_PARAMETERS));
+ assertEquals("no trainable examples in the samples", e.getMessage());
+ }
+
+ /**
+ * Checks that refining on samples without an arc-standard derivation fails
with the
+ * documented message instead of returning the model unchanged.
+ */
+ @Test
+ void testRefiningWithOnlyNonProjectiveSamplesFailsLoud() {
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardDependencyTrainer.refine(model,
+ ObjectStreamUtils.createObjectStream(nonProjectiveCorpus()),
+ SINGLE_EPOCH_PARAMETERS));
+ assertEquals("no trainable samples for refinement", e.getMessage());
+ }
+
+ /**
+ * @return Two non-projective samples, so that none has an arc-standard
derivation.
+ * Never {@code null}.
+ */
+ private static List<DependencySample> nonProjectiveCorpus() {
+ return List.of(nonProjectiveSampleWithUnseenRelation(),
nonProjectiveSample());
+ }
+
+ /** Checks per-code-point lowering and reserved-symbol pass-through. */
+ @Test
+ void testNormalizeUsesTheUnicodeDataCaseMapping() {
+ // StringUtil maps per code point via UnicodeData, so no character
expands; the JDK's
+ // String.toLowerCase would render this word as "i" + COMBINING DOT ABOVE
instead.
+ assertEquals(StringUtil.toLowerCase("İSTANBUL"),
+ FeedforwardNetwork.normalize("İSTANBUL"));
+ assertEquals("istanbul", FeedforwardNetwork.normalize("İSTANBUL"));
+ assertEquals("*hello", FeedforwardNetwork.normalize("*HELLO"));
+ // supplementary code points lower too: DESERET CAPITAL LETTER LONG I
+ assertEquals("𐐨", FeedforwardNetwork.normalize("𐐀"));
+ assertEquals(FeedforwardNetwork.UNKNOWN,
FeedforwardNetwork.normalize(FeedforwardNetwork.UNKNOWN));
+ assertNull(FeedforwardNetwork.normalize(null));
+ assertEquals("barks", FeedforwardNetwork.normalize("barks"));
+ // Greek capital sigma lowers to final sigma at the end of a word
+ assertEquals("\u03BF\u03B4\u03BF\u03C2",
FeedforwardNetwork.normalize("\u039F\u0394\u039F\u03A3"));
+ }
+
+ /**
+ * Checks that embedding rows follow the sorted symbol order after the
reserved rows,
+ * so the trained weights depend only on the samples and the seed.
+ */
+ @Test
+ void testVocabularyRowsFollowSortedSymbolOrder() {
+ final FeedforwardNetwork network = model.network();
+ assertSortedAfterReserved(network.wordIds(),
FeedforwardNetwork.RESERVED_WORD_SYMBOLS);
+ assertSortedAfterReserved(network.tagIds(),
FeedforwardNetwork.RESERVED_TAG_SYMBOLS);
+ assertSortedAfterReserved(network.labelIds(),
FeedforwardNetwork.RESERVED_LABEL_SYMBOLS);
+ assertEquals(0, network.firstWordRow());
+ assertEquals(network.wordIds().size(), network.firstTagRow());
+ assertEquals(network.wordIds().size() + network.tagIds().size(),
network.firstLabelRow());
+ }
+
+ /**
+ * Asserts that the reserved symbols take the first rows of a vocabulary
block in
+ * order and the remaining symbols follow in ascending string order.
+ *
+ * @param ids The vocabulary.
+ * @param reserved The reserved symbols.
+ */
+ private static void assertSortedAfterReserved(Map<String, Integer> ids,
String[] reserved) {
+ final String[] byRow = new String[ids.size()];
+ final int first =
ids.values().stream().mapToInt(Integer::intValue).min().orElseThrow();
+ for (final Map.Entry<String, Integer> entry : ids.entrySet()) {
+ byRow[entry.getValue() - first] = entry.getKey();
+ }
+ assertArrayEquals(reserved, Arrays.copyOf(byRow, reserved.length));
+ final String[] rest = Arrays.copyOfRange(byRow, reserved.length,
byRow.length);
+ final String[] sorted = rest.clone();
+ Arrays.sort(sorted);
+ assertArrayEquals(sorted, rest);
+ }
+
+ /** Checks that the same samples and parameters give byte-identical
networks. */
+ @Test
+ void testTrainingIsDeterministic() throws IOException {
+ final byte[] first = networkBytes(trainFeedforward(corpus(),
SINGLE_EPOCH_PARAMETERS));
+ final byte[] second = networkBytes(trainFeedforward(corpus(),
SINGLE_EPOCH_PARAMETERS));
+ assertArrayEquals(first, second);
+ }
+
+ /**
+ * Pins which reserved rows training touches: the tag and label unknown rows
keep
+ * their initialization, while the word unknown row is trained once words
fall below
+ * the cutoff.
+ */
+ @Test
+ void testUnknownRowsAreTrainedOnlyForWordsBelowTheCutoff() throws
IOException {
+ final TrainingParameters slow = feedforwardParameters(1, 0.001);
+ final TrainingParameters fast = feedforwardParameters(1, 0.1);
+ final FeedforwardNetwork slowNetwork = trainFeedforward(corpus(),
slow).network();
+ final FeedforwardNetwork fastNetwork = trainFeedforward(corpus(),
fast).network();
+ assertArrayEquals(slowNetwork.embeddings()[slowNetwork.firstTagRow()],
+ fastNetwork.embeddings()[fastNetwork.firstTagRow()]);
+ assertArrayEquals(slowNetwork.embeddings()[slowNetwork.firstLabelRow()],
+ fastNetwork.embeddings()[fastNetwork.firstLabelRow()]);
+ // every word reaches the cutoff of one, so the word unknown row is
untouched too
+ assertArrayEquals(slowNetwork.embeddings()[slowNetwork.firstWordRow()],
+ fastNetwork.embeddings()[fastNetwork.firstWordRow()]);
+
+ slow.put(Parameters.CUTOFF_PARAM, Integer.MAX_VALUE);
+ fast.put(Parameters.CUTOFF_PARAM, Integer.MAX_VALUE);
+ final FeedforwardNetwork slowUnknown = trainFeedforward(corpus(),
slow).network();
+ final FeedforwardNetwork fastUnknown = trainFeedforward(corpus(),
fast).network();
+ assertEquals(FeedforwardNetwork.RESERVED_WORD_SYMBOLS.length,
slowUnknown.wordIds().size());
+
assertFalse(Arrays.equals(slowUnknown.embeddings()[slowUnknown.firstWordRow()],
+ fastUnknown.embeddings()[fastUnknown.firstWordRow()]));
+ }
+
+ /** Checks that the manifest records the component, language, and training
run. */
+ @Test
+ void testManifestRecordsTheTrainingRun() {
+ assertEquals(LANGUAGE, model.getLanguage());
+ assertEquals("120",
model.getManifestProperty(BaseModel.TRAINING_ITERATIONS_PROPERTY));
+ assertEquals("1",
model.getManifestProperty(BaseModel.TRAINING_CUTOFF_PROPERTY));
+ }
+
+ /** Checks that serialized vocabulary entries follow their numeric ids. */
+ @Test
+ void testSerializedVocabulariesAreWrittenInAscendingIdOrder() throws
IOException {
+ try (DataInputStream data = new DataInputStream(
+ new ByteArrayInputStream(networkBytes(model)))) {
+ data.readFully(new byte[FeedforwardNetwork.Serializer.MAGIC.length]);
+ assertEquals(FeedforwardNetwork.Serializer.FORMAT_VERSION,
data.readInt());
+ for (int vocabulary = 0; vocabulary < 3; vocabulary++) {
+ final int size = data.readInt();
+ int previous = Integer.MIN_VALUE;
+ for (int entry = 0; entry < size; entry++) {
+ data.readFully(new byte[data.readInt()]);
+ final int id = data.readInt();
+ assertTrue(id > previous,
+ "vocabulary " + vocabulary + " must be written in ascending id
order");
+ previous = id;
+ }
+ }
+ }
+ }
+
+ /** Checks that a serialized and reloaded model parses like the original. */
+ @Test
+ void testModelRoundTripThroughSerialization() throws IOException {
+ final ByteArrayOutputStream out = new ByteArrayOutputStream();
+ model.serialize(out);
+ final FeedforwardDependencyModel reloaded =
+ new FeedforwardDependencyModel(new
ByteArrayInputStream(out.toByteArray()));
+ assertEquals(LANGUAGE, reloaded.getLanguage());
+ assertEquals(SHE_EATS_FISH_TREE, new FeedforwardDependencyParser(reloaded)
+ .parse(SHE_EATS_FISH_TOKENS, SHE_EATS_FISH_TAGS));
+ assertArrayEquals(networkBytes(model), networkBytes(reloaded));
+ }
+
+ /** Checks that a model file reloads and parses every corpus sentence
identically. */
+ @Test
+ void testModelFileRoundTripParsesIdentically(@TempDir Path dir) throws
IOException {
+ final Path file = dir.resolve("depparse-ff.bin");
+ model.serialize(file);
+ final FeedforwardDependencyParser reloaded =
+ new FeedforwardDependencyParser(new FeedforwardDependencyModel(file));
+ for (final DependencySample sample : corpus()) {
+ assertEquals(parser.parse(sample.getTokens(), sample.getTags()),
+ reloaded.parse(sample.getTokens(), sample.getTags()),
+ Arrays.toString(sample.getTokens()));
+ }
+ assertEquals(SHE_EATS_FISH_TREE, reloaded.parse(SHE_EATS_FISH_TOKENS,
SHE_EATS_FISH_TAGS));
+ }
+
+ /** Checks that a model file reloads through the {@link java.io.File}
constructor. */
+ @Test
+ void testModelFileConstructorParsesIdentically(@TempDir Path dir) throws
IOException {
+ final Path file = dir.resolve("depparse-ff-file.bin");
+ model.serialize(file);
+ final FeedforwardDependencyModel reloaded = new
FeedforwardDependencyModel(file.toFile());
+ assertEquals(LANGUAGE, reloaded.getLanguage());
+ assertArrayEquals(networkBytes(model), networkBytes(reloaded));
+ final FeedforwardDependencyParser reloadedParser = new
FeedforwardDependencyParser(reloaded);
+ for (final DependencySample sample : corpus()) {
+ assertEquals(parser.parse(sample.getTokens(), sample.getTags()),
+ reloadedParser.parse(sample.getTokens(), sample.getTags()),
+ Arrays.toString(sample.getTokens()));
+ }
+ }
+
+ /** Checks that a model package with a corrupt network entry is rejected. */
+ @Test
+ void testCorruptNetworkEntryIsRejected() throws IOException {
+ final ByteArrayOutputStream valid = new ByteArrayOutputStream();
+ model.serialize(valid);
+ final ByteArrayOutputStream corrupt = new ByteArrayOutputStream();
+ try (ZipInputStream in = new ZipInputStream(new
ByteArrayInputStream(valid.toByteArray()));
+ ZipOutputStream out = new ZipOutputStream(corrupt)) {
+ ZipEntry entry;
+ while ((entry = in.getNextEntry()) != null) {
+ out.putNextEntry(new ZipEntry(entry.getName()));
+
out.write(FeedforwardDependencyModel.NETWORK_ENTRY_NAME.equals(entry.getName())
+ ? "not a network".getBytes(StandardCharsets.UTF_8) :
in.readAllBytes());
+ out.closeEntry();
+ }
+ }
+ assertThrows(InvalidFormatException.class, () -> new
FeedforwardDependencyModel(
+ new ByteArrayInputStream(corrupt.toByteArray())));
+ }
+
+ /**
+ * Checks that a model package of another component is rejected the way
+ * {@link BaseModel} rejects every foreign package.
+ */
+ @Test
+ void testForeignModelPackageIsRejected() throws IOException {
+ final ByteArrayOutputStream out = new ByteArrayOutputStream();
+ DependencyTestSamples.train(corpus()).serialize(out);
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> new FeedforwardDependencyModel(new
ByteArrayInputStream(out.toByteArray())));
+ assertInstanceOf(InvalidFormatException.class, e.getCause());
+ }
+
+ /**
+ * @return Named invalid network payloads. Never {@code null}.
+ * @throws IOException Thrown if building a payload fails.
+ */
+ private static Stream<Arguments> invalidNetworks() throws IOException {
+ final NetworkBytes defaults = new NetworkBytes();
+ return Stream.of(
+ Arguments.of("wrong magic",
+ new
NetworkBytes().magic("ONLP".getBytes(StandardCharsets.US_ASCII)).bytes()),
+ Arguments.of("unsupported version", new
NetworkBytes().version(2).bytes()),
+ Arguments.of("negative vocabulary size", new
NetworkBytes().wordCount(-1).bytes()),
+ Arguments.of("vocabulary size above the limit",
+ new NetworkBytes().wordCount(ResourceLimits.MAX_ENTRIES +
1).bytes()),
+ Arguments.of("symbol length above the limit",
+ new NetworkBytes().firstWordLength(ResourceLimits.MAX_ENTRIES +
1).bytes()),
+ Arguments.of("truncated body", Arrays.copyOf(defaults.bytes(),
defaults.bytes().length / 2)),
+ Arguments.of("duplicate symbol", new NetworkBytes().tagSymbols(
+ FeedforwardNetwork.UNKNOWN, FeedforwardNetwork.UNKNOWN,
+ DependencyContextGenerator.ROOT_VALUE).bytes()),
+ Arguments.of("missing reserved symbol", new NetworkBytes().tagSymbols(
+ FeedforwardNetwork.UNKNOWN, DependencyContextGenerator.NONE_VALUE,
"NN").bytes()),
+ Arguments.of("overlapping vocabulary ids", new
NetworkBytes().firstTagRow(0).bytes()),
+ Arguments.of("vocabulary block with a gap", new
NetworkBytes().firstTagRow(4).bytes()),
+ Arguments.of("hidden matrix of the wrong width", new
NetworkBytes().hiddenColumns(1).bytes()),
+ Arguments.of("hidden matrix above the cell limit",
+ new NetworkBytes().hiddenRows(OVERSIZED_HIDDEN_ROWS).bytes()),
+ Arguments.of("non-finite weight", new
NetworkBytes().outputBias(Float.NaN).bytes()),
+ Arguments.of("trailing data", concat(defaults.bytes(), new byte[]
{1})),
+ Arguments.of("no SHIFT", new
NetworkBytes().transitions("RIGHT_ARC:root").bytes()),
+ Arguments.of("no RIGHT_ARC", new
NetworkBytes().transitions("SHIFT").bytes()),
+ Arguments.of("invalid transition", new
NetworkBytes().transitions("SHIFT", "JUMP").bytes()),
+ Arguments.of("duplicate transition",
+ new NetworkBytes().transitions("SHIFT", "SHIFT",
"RIGHT_ARC:root").bytes()));
+ }
+
+ /**
+ * Checks that each invalid payload is rejected with an {@link
InvalidFormatException}.
+ *
+ * @param name The defect.
+ * @param payload The network bytes.
+ */
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("invalidNetworks")
+ void testInvalidNetworkIsRejected(String name, byte[] payload) {
+ assertThrows(InvalidFormatException.class,
+ () -> new FeedforwardNetwork.Serializer().create(new
ByteArrayInputStream(payload)));
+ }
+
+ /** Checks that a well-formed minimal payload loads, with a symbol beyond
the old UTF limit. */
+ @Test
+ void testMinimalNetworkWithALongSymbolLoads() throws IOException {
+ final byte[] payload = new
NetworkBytes().tagSymbols(FeedforwardNetwork.UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE,
+ LONG_SYMBOL).bytes();
+ final FeedforwardNetwork network =
+ new FeedforwardNetwork.Serializer().create(new
ByteArrayInputStream(payload));
+ assertEquals(MINIMAL_ROWS + 1, network.embeddings().length);
+ assertEquals(FIRST_LABEL_ROW, network.tagIds().get(LONG_SYMBOL));
+ assertArrayEquals(payload, networkBytes(network));
+ }
+
+ /**
+ * Checks that a symbol of exactly {@link ResourceLimits#MAX_ENTRIES} UTF-8
bytes loads
+ * and reserializes unchanged, while one byte more is refused by the same
check the
+ * trainer applies. The limit is inclusive.
+ *
+ * @throws IOException Thrown if building the payload fails.
+ */
+ @Test
+ void testSymbolOfExactlyTheLimitLoads() throws IOException {
+ final String atLimit = "x".repeat(ResourceLimits.MAX_ENTRIES);
+ final byte[] payload = new
NetworkBytes().tagSymbols(FeedforwardNetwork.UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE,
+ atLimit).bytes();
+ final FeedforwardNetwork network =
+ new FeedforwardNetwork.Serializer().create(new
ByteArrayInputStream(payload));
+ assertEquals(FIRST_LABEL_ROW, network.tagIds().get(atLimit));
+ assertArrayEquals(payload, networkBytes(network));
+ FeedforwardNetwork.requireStorable(atLimit, "tag vocabulary");
+ assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardNetwork.requireStorable(atLimit + "x", "tag
vocabulary"));
+ }
+
+ /**
+ * Checks the limit on the encoded length rather than the character count: a
symbol of
+ * two-byte characters that fills the limit is accepted, and one more byte
is refused.
+ * Such a symbol is too long in bytes for the length shortcut and takes the
encoding path.
+ */
+ @Test
+ void testMultiByteSymbolIsMeasuredInBytes() {
+ final String twoByte = "\u00e9".repeat(ResourceLimits.MAX_ENTRIES / 2);
+ FeedforwardNetwork.requireStorable(twoByte, "word vocabulary");
+ assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardNetwork.requireStorable(twoByte + "x", "word
vocabulary"));
+ }
+
+ /**
+ * @return Named parameter sets that must be rejected. Never {@code null}.
+ */
+ private static Stream<Arguments> invalidParameters() {
+ return Stream.of(
+ Arguments.of("embedding size 0",
FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM, 0),
+ Arguments.of("hidden size 0",
FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM, 0),
+ Arguments.of("epochs 0", Parameters.ITERATIONS_PARAM, 0),
+ Arguments.of("batch size 0",
FeedforwardDependencyTrainer.BATCH_SIZE_PARAM, 0),
+ Arguments.of("negative learning rate",
FeedforwardDependencyTrainer.LEARNING_RATE_PARAM, -1.0),
+ Arguments.of("NaN learning rate",
FeedforwardDependencyTrainer.LEARNING_RATE_PARAM, Double.NaN),
+ Arguments.of("infinite l2", FeedforwardDependencyTrainer.L2_PARAM,
Double.POSITIVE_INFINITY),
+ Arguments.of("dropout 1", FeedforwardDependencyTrainer.DROPOUT_PARAM,
1.0),
+ Arguments.of("dropout -0.1",
FeedforwardDependencyTrainer.DROPOUT_PARAM, -0.1),
+ Arguments.of("word cutoff -1", Parameters.CUTOFF_PARAM, -1),
+ Arguments.of("embedding size above the limit",
+ FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM,
+ FeedforwardDependencyTrainer.MAX_EMBEDDING_SIZE + 1),
+ Arguments.of("hidden size above the limit",
+ FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM,
+ FeedforwardDependencyTrainer.MAX_HIDDEN_SIZE + 1));
+ }
+
+ /**
+ * Checks that an out-of-range hyperparameter is rejected before any sample
is read.
+ *
+ * @param name The defect.
+ * @param key The parameter key.
+ * @param value The invalid value.
+ */
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("invalidParameters")
+ void testInvalidParametersAreRejected(String name, String key, Object value)
{
+ final TrainingParameters parameters = feedforwardParameters(1, 0.01);
+ if (value instanceof Integer i) {
+ parameters.put(key, i);
+ } else {
+ parameters.put(key, (Double) value);
+ }
+ assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardDependencyTrainer.Settings.from(parameters));
+ }
+
+ /** Checks that the largest widths and the largest network together are
rejected or accepted. */
+ @Test
+ void testParameterLimitsAreInclusive() {
+ final TrainingParameters wide = feedforwardParameters(1, 0.01);
+ wide.put(FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM,
+ FeedforwardDependencyTrainer.MAX_EMBEDDING_SIZE);
+ wide.put(FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM, 1);
+ assertEquals(FeedforwardDependencyTrainer.MAX_EMBEDDING_SIZE,
+ FeedforwardDependencyTrainer.Settings.from(wide).embeddingSize());
+
+ final TrainingParameters deep = feedforwardParameters(1, 0.01);
+ deep.put(FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM, 1);
+ deep.put(FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM,
+ FeedforwardDependencyTrainer.MAX_HIDDEN_SIZE);
+ assertEquals(FeedforwardDependencyTrainer.MAX_HIDDEN_SIZE,
+ FeedforwardDependencyTrainer.Settings.from(deep).hiddenSize());
+
+ final TrainingParameters dropout = feedforwardParameters(1, 0.01);
+ dropout.put(FeedforwardDependencyTrainer.DROPOUT_PARAM,
Math.nextDown(1.0));
+ assertEquals(Math.nextDown(1.0),
+ FeedforwardDependencyTrainer.Settings.from(dropout).dropout());
+
+ final TrainingParameters oversized = feedforwardParameters(1, 0.01);
+ oversized.put(FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM,
+ FeedforwardDependencyTrainer.MAX_EMBEDDING_SIZE);
+ oversized.put(FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM,
+ FeedforwardDependencyTrainer.MAX_HIDDEN_SIZE);
+ assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardDependencyTrainer.Settings.from(oversized));
+ }
+
+ /**
+ * @return Single-epoch parameters with a beam size of one. Never {@code
null}.
+ */
+ private static TrainingParameters greedyBeamParameters() {
+ final TrainingParameters parameters = feedforwardParameters(1, 0.01);
+ parameters.put(BeamSearch.BEAM_SIZE_PARAMETER, 1);
+ return parameters;
+ }
+
+ /**
+ * Checks that the beam size only constrains refinement: local training
ignores it,
+ * so parameters shared with a greedy parser train without an error.
+ *
+ * @throws IOException Thrown if reading the in-memory samples fails.
+ */
+ @Test
+ void testTrainAcceptsAGreedyBeamSize() throws IOException {
+ final FeedforwardDependencyModel trained =
+ trainFeedforward(corpus(), greedyBeamParameters());
+ assertNotNull(new FeedforwardDependencyParser(trained)
+ .parse(THE_DOG_BARKS_TOKENS, THE_DOG_BARKS_TAGS));
+ }
+
+ /** Checks that absent keys take the documented defaults. */
+ @Test
+ void testDefaultsApplyToAbsentKeys() {
+ final FeedforwardDependencyTrainer.Settings settings =
+ FeedforwardDependencyTrainer.Settings.from(new TrainingParameters());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_EMBEDDING_SIZE,
settings.embeddingSize());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_HIDDEN_SIZE,
settings.hiddenSize());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_EPOCHS,
settings.epochs());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_BATCH_SIZE,
settings.batchSize());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_LEARNING_RATE,
settings.learningRate());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_L2, settings.l2());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_DROPOUT,
settings.dropout());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_WORD_CUTOFF,
settings.wordCutoff());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_SEED, settings.seed());
+ assertEquals(FeedforwardDependencyTrainer.DEFAULT_BEAM_SIZE,
settings.beamSize());
+ }
+
+ /**
+ * Checks that training rejects a symbol the network loader could not read
back,
+ * instead of producing a model that does not load.
+ */
+ @Test
+ void testTrainingRejectsASymbolAboveTheLoaderLimit() {
+ final String oversized = "x".repeat(ResourceLimits.MAX_ENTRIES + 1);
+ final List<DependencySample> samples = List.of(sample(new String[]
{oversized, "barks"},
+ new String[] {"NN", "VBZ"}, new int[] {1, -1}, new String[] {"nsubj",
"root"}));
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> trainFeedforward(samples, SINGLE_EPOCH_PARAMETERS));
+ assertEquals("word vocabulary symbol length " +
(ResourceLimits.MAX_ENTRIES + 1L)
+ + " exceeds the limit of " + ResourceLimits.MAX_ENTRIES + " UTF-8
bytes",
+ e.getMessage());
+ }
+
+ /** Checks that every reserved symbol, and nothing else, is recognized as
reserved. */
+ @Test
+ void testReservedSymbolsAreRecognized() {
+ for (final String[] reserved :
List.of(FeedforwardNetwork.RESERVED_WORD_SYMBOLS,
+ FeedforwardNetwork.RESERVED_TAG_SYMBOLS,
FeedforwardNetwork.RESERVED_LABEL_SYMBOLS)) {
+ for (final String symbol : reserved) {
+ assertTrue(FeedforwardNetwork.isReservedSymbol(symbol), symbol);
+ }
+ }
+ assertFalse(FeedforwardNetwork.isReservedSymbol("barks"));
+ assertFalse(FeedforwardNetwork.isReservedSymbol(null));
+ }
+
+ /** Checks that a diverging run fails with an IllegalStateException. */
+ @Test
+ void testTrainingRejectsNonFiniteWeights() {
+ final TrainingParameters parameters = feedforwardParameters(2,
Double.MAX_VALUE);
+ parameters.put(FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM, 4);
+ parameters.put(FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM, 4);
+ parameters.put(FeedforwardDependencyTrainer.BATCH_SIZE_PARAM, 1);
+ assertThrows(IllegalStateException.class, () -> trainFeedforward(corpus(),
parameters));
+ }
+
+ /** Checks the feature length guards of the network. */
+ @Test
+ void testNetworkRejectsInvalidFeatureArrays() {
+ assertThrows(IllegalArgumentException.class, () ->
model.network().score(new int[0], null));
+ final int[] outOfRange = new int[FeedforwardContext.FEATURE_COUNT];
+ outOfRange[0] = model.network().embeddings().length;
+ assertThrows(IllegalArgumentException.class, () ->
model.network().score(outOfRange, null));
+ }
+
+ /** Checks that pretrained vectors seed rows and malformed vectors are
rejected. */
+ @Test
+ void testPretrainedSeedingAppliesAndValidates() throws IOException {
+ // near-zero learning keeps the seeded row observable after one epoch
+ final TrainingParameters parameters = feedforwardParameters(1, 1e-9);
+ parameters.put(FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM, 4);
+ parameters.put(FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM, 8);
+ final float[] vector = {0.25f, -0.5f, 0.75f, -1.0f};
+ final FeedforwardNetwork seeded =
FeedforwardDependencyTrainer.train(LANGUAGE,
+ ObjectStreamUtils.createObjectStream(corpus()), parameters,
+ word -> "dog".equals(word) ? vector.clone() : null).network();
+ final int row = seeded.wordIds().get("dog");
+ for (int d = 0; d < vector.length; d++) {
+ assertEquals(vector[d], seeded.embeddings()[row][d], 1e-4);
+ }
+ assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardDependencyTrainer.train(LANGUAGE,
+ ObjectStreamUtils.createObjectStream(corpus()), parameters,
+ word -> new float[] {1.0f}));
+ assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardDependencyTrainer.train(LANGUAGE,
+ ObjectStreamUtils.createObjectStream(corpus()), parameters,
+ word -> new float[] {Float.NaN, 0.0f, 0.0f, 0.0f}));
+ }
+
+ /**
+ * @return Named invalid calls with their expected messages. Never {@code
null}.
+ */
+ private static Stream<Arguments> invalidArguments() {
+ final TrainingParameters parameters = new TrainingParameters();
+ return Stream.of(
+ Arguments.of("parser without model", "model must not be null",
+ (Executable) () -> new FeedforwardDependencyParser(null)),
+ Arguments.of("parser with beam 0", "beamSize must be positive",
+ (Executable) () -> new FeedforwardDependencyParser(model, 0)),
+ Arguments.of("parser with negative cache", "scoringCacheBytes must not
be negative",
+ (Executable) () -> new FeedforwardDependencyParser(model, 1, -1L)),
+ Arguments.of("parser with a cache too small for the tables", null,
+ (Executable) () -> new FeedforwardDependencyParser(model, 1, 1L)),
+ Arguments.of("train without language", "languageCode must not be null",
+ (Executable) () -> FeedforwardDependencyTrainer.train(null,
+ ObjectStreamUtils.createObjectStream(corpus()), parameters)),
+ Arguments.of("train without samples", "samples must not be null",
+ (Executable) () -> FeedforwardDependencyTrainer.train(LANGUAGE,
null, parameters)),
+ Arguments.of("train without parameters", "parameters must not be null",
+ (Executable) () -> FeedforwardDependencyTrainer.train(LANGUAGE,
+ ObjectStreamUtils.createObjectStream(corpus()), null)),
+ Arguments.of("refine without model", "model must not be null",
+ (Executable) () -> FeedforwardDependencyTrainer.refine(null,
+ ObjectStreamUtils.createObjectStream(corpus()), parameters)),
+ Arguments.of("refine without samples", "samples must not be null",
+ (Executable) () -> FeedforwardDependencyTrainer.refine(model,
null, parameters)),
+ Arguments.of("refine without parameters", "parameters must not be
null",
+ (Executable) () -> FeedforwardDependencyTrainer.refine(model,
+ ObjectStreamUtils.createObjectStream(corpus()), null)),
+ Arguments.of("refine with beam 1", BeamSearch.BEAM_SIZE_PARAMETER
+ + " must be at least 2: 1",
+ (Executable) () -> FeedforwardDependencyTrainer.refine(model,
+ ObjectStreamUtils.createObjectStream(corpus()),
greedyBeamParameters())),
+ Arguments.of("parse without tokens", "tokens must not be null",
+ (Executable) () -> parser.parse(null, new String[] {"DT"})),
+ Arguments.of("parse without tags", "tags must not be null",
+ (Executable) () -> parser.parse(new String[] {"the"}, null)),
+ Arguments.of("parse with misaligned tags", null,
+ (Executable) () -> parser.parse(new String[] {"the"}, new String[]
{"DT", "NN"})),
+ Arguments.of("score without features", "features must not be null",
+ (Executable) () -> model.network().score(null, null)),
+ Arguments.of("model without stream", "in must not be null",
+ (Executable) () -> new FeedforwardDependencyModel((InputStream)
null)),
+ Arguments.of("model without path", "modelPath must not be null",
+ (Executable) () -> new FeedforwardDependencyModel((Path) null)));
+ }
+
+ /**
+ * Checks the argument guards of the parser, trainer, network, and model.
+ *
+ * @param name The call.
+ * @param message The expected message, or {@code null} to check only the
type.
+ * @param call The invalid call.
+ */
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("invalidArguments")
+ void testInvalidArgumentsAreRejected(String name, String message, Executable
call) {
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class, call);
+ if (message != null) {
+ assertEquals(message, e.getMessage());
+ }
+ }
+
+ /**
+ * Verifies that training data using the reserved symbols as ordinary tags,
labels,
+ * or tokens does not displace the reserved rows: the model reloads and the
colliding
+ * symbols share the reserved rows the way unknown symbols do.
+ *
+ * @throws IOException Thrown if serialization or loading fails.
+ */
+ @Test
+ void testReservedSymbolsInTrainingDataSurviveReload() throws IOException {
+ final List<DependencySample> colliding = List.of(
+ sample(new String[] {"*ROOT*", "*UNK*", "barks"}, new String[]
{"*UNK*", "NN", "VBZ"},
+ new int[] {1, 2, -1}, new String[] {"*NULL*", "nsubj", "root"}),
+ sample(new String[] {"dogs", "bark"}, new String[] {"*NULL*",
"*ROOT*"},
+ new int[] {1, -1}, new String[] {"*UNK*", "root"}));
+ final FeedforwardDependencyModel trained = trainFeedforward(colliding,
SINGLE_EPOCH_PARAMETERS);
+ final ByteArrayOutputStream out = new ByteArrayOutputStream();
+ trained.serialize(out);
+ final FeedforwardDependencyModel reloaded =
+ new FeedforwardDependencyModel(new
ByteArrayInputStream(out.toByteArray()));
+ final String[] tokens = {"*ROOT*", "*UNK*", "barks"};
+ final String[] tags = {"*UNK*", "NN", "VBZ"};
+ assertEquals(new FeedforwardDependencyParser(trained).parse(tokens, tags),
+ new FeedforwardDependencyParser(reloaded).parse(tokens, tags));
+ }
+
+ /**
+ * @param network The network whose vocabularies map the features.
+ * @param tokens The sentence tokens.
+ * @param tags The tags aligned with {@code tokens}.
+ * @return The features of the initial configuration. Never {@code null}.
+ */
+ private static int[] initialFeatures(FeedforwardNetwork network, String[]
tokens,
+ String[] tags) {
+ return new FeedforwardContext(network, tokens, tags)
+ .featureIds(new ArcStandardState(tokens.length));
+ }
+
+ /**
+ * @param model The model whose network is serialized.
+ * @return The network bytes. Never {@code null}.
+ * @throws IOException Thrown if writing fails.
+ */
+ private static byte[] networkBytes(FeedforwardDependencyModel model) throws
IOException {
+ return networkBytes(model.network());
+ }
+
+ /**
+ * @param network The network to serialize.
+ * @return The network bytes. Never {@code null}.
+ * @throws IOException Thrown if writing fails.
+ */
+ private static byte[] networkBytes(FeedforwardNetwork network) throws
IOException {
Review Comment:
Moved shared helpers to DependencyTestSamples and removed the duplicates.
The 60 epoch test model is now trained once there.
##########
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/FeedforwardDependencyParserTest.java:
##########
@@ -0,0 +1,1150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.depparse;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import java.util.stream.Stream;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipInputStream;
+import java.util.zip.ZipOutputStream;
+
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.function.Executable;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.ml.BeamSearch;
+import opennlp.tools.util.InvalidFormatException;
+import opennlp.tools.util.ObjectStreamUtils;
+import opennlp.tools.util.Parameters;
+import opennlp.tools.util.ResourceLimits;
+import opennlp.tools.util.StringUtil;
+import opennlp.tools.util.TrainingParameters;
+import opennlp.tools.util.model.BaseModel;
+
+import static opennlp.tools.depparse.DependencyTestSamples.LANGUAGE;
+import static opennlp.tools.depparse.DependencyTestSamples.MINIMAL_TRANSITIONS;
+import static opennlp.tools.depparse.DependencyTestSamples.SHE_EATS_FISH_TAGS;
+import static
opennlp.tools.depparse.DependencyTestSamples.SHE_EATS_FISH_TOKENS;
+import static opennlp.tools.depparse.DependencyTestSamples.SHE_EATS_FISH_TREE;
+import static opennlp.tools.depparse.DependencyTestSamples.THE_DOG_BARKS_TAGS;
+import static
opennlp.tools.depparse.DependencyTestSamples.THE_DOG_BARKS_TOKENS;
+import static opennlp.tools.depparse.DependencyTestSamples.THE_DOG_BARKS_TREE;
+import static opennlp.tools.depparse.DependencyTestSamples.UNSEEN_RELATION;
+import static opennlp.tools.depparse.DependencyTestSamples.corpus;
+import static
opennlp.tools.depparse.DependencyTestSamples.feedforwardParameters;
+import static opennlp.tools.depparse.DependencyTestSamples.nonProjectiveSample;
+import static
opennlp.tools.depparse.DependencyTestSamples.nonProjectiveSampleWithUnseenRelation;
+import static opennlp.tools.depparse.DependencyTestSamples.sample;
+import static opennlp.tools.depparse.DependencyTestSamples.trainFeedforward;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests feedforward training, parsing, refinement, the model package, and the
network
+ * format.
+ */
+public class FeedforwardDependencyParserTest {
+
+ /** The beam size of the beamed parsers under test. */
+ private static final int BEAM_SIZE = 4;
+
+ /** The refinement parameters: two passes at a small step size. */
+ private static final TrainingParameters REFINE_PARAMETERS =
feedforwardParameters(2, 0.01);
+
+ /** One-pass parameters for tests that only inspect the result shape. */
+ private static final TrainingParameters SINGLE_EPOCH_PARAMETERS =
feedforwardParameters(1, 0.01);
+
+ /** The reserved rows of the hand-written networks, one per reserved symbol.
*/
+ private static final int MINIMAL_ROWS = FeedforwardNetwork.RESERVED_ROWS;
+
+ /** The first tag row of the hand-written networks, after the reserved word
rows. */
+ private static final int FIRST_TAG_ROW =
FeedforwardNetwork.RESERVED_WORD_SYMBOLS.length;
+
+ /** The first label row of the hand-written networks, after the reserved tag
rows. */
+ private static final int FIRST_LABEL_ROW =
+ FIRST_TAG_ROW + FeedforwardNetwork.RESERVED_TAG_SYMBOLS.length;
+
+ /** An embedding width that every hand-written network uses. */
+ private static final int UNIT_WIDTH = 1;
+
+ /** The hidden matrix rows that push the smallest network over the cell
limit. */
+ private static final int OVERSIZED_HIDDEN_ROWS =
Math.min(ResourceLimits.MAX_ENTRIES,
+ ResourceLimits.MAX_MATRIX_CELLS / FeedforwardContext.FEATURE_COUNT + 1);
+
+ /** A symbol longer than the 65535 bytes {@code writeUTF} could store. */
+ private static final String LONG_SYMBOL = "x".repeat(70_000);
+
+ private static FeedforwardDependencyModel model;
+ private static FeedforwardDependencyParser parser;
+
+ /**
+ * @return The distinct training sentences, one per parameterized
invocation. Never
+ * {@code null}.
+ */
+ private static List<DependencySample> trainingSamples() {
+ return DependencyTestSamples.sentences();
+ }
+
+ /**
+ * Trains the shared model once for all tests.
+ *
+ * @throws IOException Thrown if reading the in-memory samples fails.
+ */
+ @BeforeAll
+ static void trainParser() throws IOException {
+ model = DependencyTestSamples.trainFeedforward();
+ parser = new FeedforwardDependencyParser(model);
+ }
+
+ /** Checks that the greedy parser reproduces a training sentence. */
+ @Test
+ void testMemorizesTrainingSentences() {
+ assertEquals(THE_DOG_BARKS_TREE, parser.parse(THE_DOG_BARKS_TOKENS,
THE_DOG_BARKS_TAGS));
+ }
+
+ /** Checks UAS and LAS of 1.0 on the training corpus. */
+ @Test
+ void testEvaluatorScoresPerfectlyOnTrainingData() throws IOException {
+ final DependencyEvaluator evaluator = new DependencyEvaluator(parser);
+ evaluator.evaluate(ObjectStreamUtils.createObjectStream(corpus()));
+ assertEquals(1.0d, evaluator.getUas());
+ assertEquals(1.0d, evaluator.getLas());
+ }
+
+ /** Checks that unseen words still parse to a tree with one root inside the
sentence. */
+ @Test
+ void testUnknownWordsStillYieldASingleRootedTree() {
+ final DependencyTree parsed = parser.parse(new String[] {"unseen",
"words"},
+ new String[] {"JJ", "NNS"});
+ assertEquals(2, parsed.size());
+ assertTrue(parsed.root() >= 0 && parsed.root() < parsed.size(),
parsed.toString());
+ assertEquals(DependencyArc.ROOT_HEAD, parsed.headOf(parsed.root()));
+ }
+
+ /** Checks that the transition inventory is exactly the sorted set the
oracle produced. */
+ @Test
+ void testTransitionInventoryMatchesTheOracle() {
+ final TreeSet<String> expected = new TreeSet<>();
+ for (final DependencySample sample : DependencyTestSamples.sentences()) {
+ for (final Transition transition :
ArcStandardOracle.transitions(sample.getTree())) {
+ expected.add(transition.encode());
+ }
+ }
+ assertEquals(new ArrayList<>(expected),
List.of(model.network().transitions()));
+ }
+
+ /**
+ * Checks that a beam of one yields the greedy parse for every training
sample.
+ *
+ * @param sample The training sample to parse both ways.
+ */
+ @ParameterizedTest(name = "beam of one matches greedy for {0}")
+ @MethodSource("trainingSamples")
+ void testBeamOfOneMatchesGreedy(DependencySample sample) {
+ final FeedforwardDependencyParser beamed = new
FeedforwardDependencyParser(model, 1);
+ assertEquals(parser.parse(sample.getTokens(), sample.getTags()),
+ beamed.parse(sample.getTokens(), sample.getTags()));
+ }
+
+ /** Checks that beam search reproduces a training sentence. */
+ @Test
+ void testBeamedParserReproducesTrainingSentences() {
+ final FeedforwardDependencyParser beamed = new
FeedforwardDependencyParser(model, BEAM_SIZE);
+ assertEquals(THE_DOG_BARKS_TREE, beamed.parse(THE_DOG_BARKS_TOKENS,
THE_DOG_BARKS_TAGS));
+ }
+
+ /** Checks that beam search is deterministic and single-rooted on unseen
words. */
+ @Test
+ void testBeamedParseIsDeterministicAndSingleRooted() {
+ final FeedforwardDependencyParser beamed = new
FeedforwardDependencyParser(model, 8);
+ final String[] tokens = {"unseen", "words", "everywhere"};
+ final String[] tags = {"JJ", "NNS", "RB"};
+ final DependencyTree first = beamed.parse(tokens, tags);
+ assertEquals(first, beamed.parse(tokens, tags));
+ assertEquals(3, first.size());
+ assertEquals(DependencyArc.ROOT_HEAD, first.headOf(first.root()));
+ }
+
+ /** Checks that a cached parser returns the parses of an uncached one. */
+ @Test
+ void testCachedParserMatchesTheUncachedParser() {
+ final FeedforwardDependencyParser cached =
+ new FeedforwardDependencyParser(model, BEAM_SIZE, 1L << 20);
+ final FeedforwardDependencyParser uncached =
+ new FeedforwardDependencyParser(model, BEAM_SIZE);
+ for (final DependencySample sample : DependencyTestSamples.sentences()) {
+ assertEquals(uncached.parse(sample.getTokens(), sample.getTags()),
+ cached.parse(sample.getTokens(), sample.getTags()));
+ }
+ assertNotNull(cached.cache());
+ assertTrue(cached.cache().entries() > 0);
+ assertNull(uncached.cache());
+ }
+
+ /** Checks that refinement keeps perfect scores on the training corpus. */
+ @Test
+ void testRefinementKeepsToyPerformance() throws IOException {
+ final FeedforwardDependencyModel local =
+ trainFeedforward(corpus(), feedforwardParameters(60, 0.05));
+ final TrainingParameters parameters = feedforwardParameters(2, 0.01);
+ parameters.put(BeamSearch.BEAM_SIZE_PARAMETER, 2);
+ final FeedforwardDependencyModel refined =
FeedforwardDependencyTrainer.refine(
+ local, ObjectStreamUtils.createObjectStream(corpus()), parameters);
+
+ final DependencyEvaluator evaluator =
+ new DependencyEvaluator(new FeedforwardDependencyParser(refined, 2));
+ evaluator.evaluate(ObjectStreamUtils.createObjectStream(corpus()));
+ assertEquals(1.0d, evaluator.getUas());
+ assertEquals(1.0d, evaluator.getLas());
+ assertEquals("2",
refined.getManifestProperty(BeamSearch.BEAM_SIZE_PARAMETER));
+ }
+
+ /** Checks that refinement fails loudly on a relation label the model never
saw. */
+ @Test
+ void testRefineRejectsAnUnknownRelation() {
+ final List<DependencySample> unseenRelation = List.of(
+ sample(THE_DOG_BARKS_TOKENS, THE_DOG_BARKS_TAGS,
+ new int[] {1, 2, -1}, new String[] {"det", UNSEEN_RELATION,
"root"}));
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardDependencyTrainer.refine(model,
+ ObjectStreamUtils.createObjectStream(unseenRelation),
REFINE_PARAMETERS));
+ assertEquals("unknown transition in the refinement samples: LEFT_ARC:" +
UNSEEN_RELATION,
+ e.getMessage());
+ }
+
+ /** Checks that refinement returns a distinct model and leaves the input
scores unchanged. */
+ @Test
+ void testRefineReturnsANewModelAndLeavesTheOriginalUntouched() throws
IOException {
+ final FeedforwardDependencyModel local =
+ trainFeedforward(corpus(), feedforwardParameters(60, 0.05));
+ final int[] features = initialFeatures(local.network(),
THE_DOG_BARKS_TOKENS,
+ THE_DOG_BARKS_TAGS);
+ final double[] before = local.network().score(features, null);
+
+ final FeedforwardDependencyModel refined =
FeedforwardDependencyTrainer.refine(
+ local, ObjectStreamUtils.createObjectStream(corpus()),
REFINE_PARAMETERS);
+
+ assertNotSame(local, refined);
+ assertArrayEquals(before, local.network().score(features, null));
+ assertFalse(Arrays.equals(before, refined.network().score(features,
null)));
+ assertEquals(LANGUAGE, refined.getLanguage());
+ }
+
+ /**
+ * Checks that non-projective samples are skipped before transition
validation and
+ * that the refined model still parses the projective sample.
+ */
+ @Test
+ void testNonProjectiveSampleWithUnknownRelationIsSkippedNotFatal() throws
IOException {
+ final List<DependencySample> mixed =
List.of(nonProjectiveSampleWithUnseenRelation(),
+ new DependencySample(THE_DOG_BARKS_TOKENS, THE_DOG_BARKS_TAGS,
THE_DOG_BARKS_TREE));
+ final FeedforwardDependencyModel refined =
FeedforwardDependencyTrainer.refine(
+ model, ObjectStreamUtils.createObjectStream(mixed),
SINGLE_EPOCH_PARAMETERS);
+ assertEquals(THE_DOG_BARKS_TREE, new FeedforwardDependencyParser(refined,
BEAM_SIZE)
+ .parse(THE_DOG_BARKS_TOKENS, THE_DOG_BARKS_TAGS));
+ }
+
+ /** Checks that the inventory of a model trained on mixed data omits
non-projective labels. */
+ @Test
+ void testTrainingOmitsNonProjectiveLabels() throws IOException {
+ final List<DependencySample> mixed = new ArrayList<>(corpus());
+ mixed.add(nonProjectiveSampleWithUnseenRelation());
+ final FeedforwardDependencyModel trained = trainFeedforward(mixed,
SINGLE_EPOCH_PARAMETERS);
+ for (final String transition : trained.network().transitions()) {
+ assertFalse(transition.contains(UNSEEN_RELATION), transition);
+ }
+ }
+
+ /**
+ * Checks that training on samples without an arc-standard derivation fails
with the
+ * documented message instead of producing an empty model.
+ */
+ @Test
+ void testTrainingWithOnlyNonProjectiveSamplesFailsLoud() {
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> trainFeedforward(nonProjectiveCorpus(),
SINGLE_EPOCH_PARAMETERS));
+ assertEquals("no trainable examples in the samples", e.getMessage());
+ }
+
+ /**
+ * Checks that refining on samples without an arc-standard derivation fails
with the
+ * documented message instead of returning the model unchanged.
+ */
+ @Test
+ void testRefiningWithOnlyNonProjectiveSamplesFailsLoud() {
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardDependencyTrainer.refine(model,
+ ObjectStreamUtils.createObjectStream(nonProjectiveCorpus()),
+ SINGLE_EPOCH_PARAMETERS));
+ assertEquals("no trainable samples for refinement", e.getMessage());
+ }
+
+ /**
+ * @return Two non-projective samples, so that none has an arc-standard
derivation.
+ * Never {@code null}.
+ */
+ private static List<DependencySample> nonProjectiveCorpus() {
+ return List.of(nonProjectiveSampleWithUnseenRelation(),
nonProjectiveSample());
+ }
+
+ /** Checks per-code-point lowering and reserved-symbol pass-through. */
+ @Test
+ void testNormalizeUsesTheUnicodeDataCaseMapping() {
+ // StringUtil maps per code point via UnicodeData, so no character
expands; the JDK's
+ // String.toLowerCase would render this word as "i" + COMBINING DOT ABOVE
instead.
+ assertEquals(StringUtil.toLowerCase("İSTANBUL"),
+ FeedforwardNetwork.normalize("İSTANBUL"));
+ assertEquals("istanbul", FeedforwardNetwork.normalize("İSTANBUL"));
+ assertEquals("*hello", FeedforwardNetwork.normalize("*HELLO"));
+ // supplementary code points lower too: DESERET CAPITAL LETTER LONG I
+ assertEquals("𐐨", FeedforwardNetwork.normalize("𐐀"));
+ assertEquals(FeedforwardNetwork.UNKNOWN,
FeedforwardNetwork.normalize(FeedforwardNetwork.UNKNOWN));
+ assertNull(FeedforwardNetwork.normalize(null));
+ assertEquals("barks", FeedforwardNetwork.normalize("barks"));
+ // Greek capital sigma lowers to final sigma at the end of a word
+ assertEquals("\u03BF\u03B4\u03BF\u03C2",
FeedforwardNetwork.normalize("\u039F\u0394\u039F\u03A3"));
+ }
+
+ /**
+ * Checks that embedding rows follow the sorted symbol order after the
reserved rows,
+ * so the trained weights depend only on the samples and the seed.
+ */
+ @Test
+ void testVocabularyRowsFollowSortedSymbolOrder() {
+ final FeedforwardNetwork network = model.network();
+ assertSortedAfterReserved(network.wordIds(),
FeedforwardNetwork.RESERVED_WORD_SYMBOLS);
+ assertSortedAfterReserved(network.tagIds(),
FeedforwardNetwork.RESERVED_TAG_SYMBOLS);
+ assertSortedAfterReserved(network.labelIds(),
FeedforwardNetwork.RESERVED_LABEL_SYMBOLS);
+ assertEquals(0, network.firstWordRow());
+ assertEquals(network.wordIds().size(), network.firstTagRow());
+ assertEquals(network.wordIds().size() + network.tagIds().size(),
network.firstLabelRow());
+ }
+
+ /**
+ * Asserts that the reserved symbols take the first rows of a vocabulary
block in
+ * order and the remaining symbols follow in ascending string order.
+ *
+ * @param ids The vocabulary.
+ * @param reserved The reserved symbols.
+ */
+ private static void assertSortedAfterReserved(Map<String, Integer> ids,
String[] reserved) {
+ final String[] byRow = new String[ids.size()];
+ final int first =
ids.values().stream().mapToInt(Integer::intValue).min().orElseThrow();
+ for (final Map.Entry<String, Integer> entry : ids.entrySet()) {
+ byRow[entry.getValue() - first] = entry.getKey();
+ }
+ assertArrayEquals(reserved, Arrays.copyOf(byRow, reserved.length));
+ final String[] rest = Arrays.copyOfRange(byRow, reserved.length,
byRow.length);
+ final String[] sorted = rest.clone();
+ Arrays.sort(sorted);
+ assertArrayEquals(sorted, rest);
+ }
+
+ /** Checks that the same samples and parameters give byte-identical
networks. */
+ @Test
+ void testTrainingIsDeterministic() throws IOException {
+ final byte[] first = networkBytes(trainFeedforward(corpus(),
SINGLE_EPOCH_PARAMETERS));
+ final byte[] second = networkBytes(trainFeedforward(corpus(),
SINGLE_EPOCH_PARAMETERS));
+ assertArrayEquals(first, second);
+ }
+
+ /**
+ * Pins which reserved rows training touches: the tag and label unknown rows
keep
+ * their initialization, while the word unknown row is trained once words
fall below
+ * the cutoff.
+ */
+ @Test
+ void testUnknownRowsAreTrainedOnlyForWordsBelowTheCutoff() throws
IOException {
+ final TrainingParameters slow = feedforwardParameters(1, 0.001);
+ final TrainingParameters fast = feedforwardParameters(1, 0.1);
+ final FeedforwardNetwork slowNetwork = trainFeedforward(corpus(),
slow).network();
+ final FeedforwardNetwork fastNetwork = trainFeedforward(corpus(),
fast).network();
+ assertArrayEquals(slowNetwork.embeddings()[slowNetwork.firstTagRow()],
+ fastNetwork.embeddings()[fastNetwork.firstTagRow()]);
+ assertArrayEquals(slowNetwork.embeddings()[slowNetwork.firstLabelRow()],
+ fastNetwork.embeddings()[fastNetwork.firstLabelRow()]);
+ // every word reaches the cutoff of one, so the word unknown row is
untouched too
+ assertArrayEquals(slowNetwork.embeddings()[slowNetwork.firstWordRow()],
+ fastNetwork.embeddings()[fastNetwork.firstWordRow()]);
+
+ slow.put(Parameters.CUTOFF_PARAM, Integer.MAX_VALUE);
+ fast.put(Parameters.CUTOFF_PARAM, Integer.MAX_VALUE);
+ final FeedforwardNetwork slowUnknown = trainFeedforward(corpus(),
slow).network();
+ final FeedforwardNetwork fastUnknown = trainFeedforward(corpus(),
fast).network();
+ assertEquals(FeedforwardNetwork.RESERVED_WORD_SYMBOLS.length,
slowUnknown.wordIds().size());
+
assertFalse(Arrays.equals(slowUnknown.embeddings()[slowUnknown.firstWordRow()],
+ fastUnknown.embeddings()[fastUnknown.firstWordRow()]));
+ }
+
+ /** Checks that the manifest records the component, language, and training
run. */
+ @Test
+ void testManifestRecordsTheTrainingRun() {
+ assertEquals(LANGUAGE, model.getLanguage());
+ assertEquals("120",
model.getManifestProperty(BaseModel.TRAINING_ITERATIONS_PROPERTY));
+ assertEquals("1",
model.getManifestProperty(BaseModel.TRAINING_CUTOFF_PROPERTY));
+ }
+
+ /** Checks that serialized vocabulary entries follow their numeric ids. */
+ @Test
+ void testSerializedVocabulariesAreWrittenInAscendingIdOrder() throws
IOException {
+ try (DataInputStream data = new DataInputStream(
+ new ByteArrayInputStream(networkBytes(model)))) {
+ data.readFully(new byte[FeedforwardNetwork.Serializer.MAGIC.length]);
+ assertEquals(FeedforwardNetwork.Serializer.FORMAT_VERSION,
data.readInt());
+ for (int vocabulary = 0; vocabulary < 3; vocabulary++) {
+ final int size = data.readInt();
+ int previous = Integer.MIN_VALUE;
+ for (int entry = 0; entry < size; entry++) {
+ data.readFully(new byte[data.readInt()]);
+ final int id = data.readInt();
+ assertTrue(id > previous,
+ "vocabulary " + vocabulary + " must be written in ascending id
order");
+ previous = id;
+ }
+ }
+ }
+ }
+
+ /** Checks that a serialized and reloaded model parses like the original. */
+ @Test
+ void testModelRoundTripThroughSerialization() throws IOException {
+ final ByteArrayOutputStream out = new ByteArrayOutputStream();
+ model.serialize(out);
+ final FeedforwardDependencyModel reloaded =
+ new FeedforwardDependencyModel(new
ByteArrayInputStream(out.toByteArray()));
+ assertEquals(LANGUAGE, reloaded.getLanguage());
+ assertEquals(SHE_EATS_FISH_TREE, new FeedforwardDependencyParser(reloaded)
+ .parse(SHE_EATS_FISH_TOKENS, SHE_EATS_FISH_TAGS));
+ assertArrayEquals(networkBytes(model), networkBytes(reloaded));
+ }
+
+ /** Checks that a model file reloads and parses every corpus sentence
identically. */
+ @Test
+ void testModelFileRoundTripParsesIdentically(@TempDir Path dir) throws
IOException {
+ final Path file = dir.resolve("depparse-ff.bin");
+ model.serialize(file);
+ final FeedforwardDependencyParser reloaded =
+ new FeedforwardDependencyParser(new FeedforwardDependencyModel(file));
+ for (final DependencySample sample : corpus()) {
+ assertEquals(parser.parse(sample.getTokens(), sample.getTags()),
+ reloaded.parse(sample.getTokens(), sample.getTags()),
+ Arrays.toString(sample.getTokens()));
+ }
+ assertEquals(SHE_EATS_FISH_TREE, reloaded.parse(SHE_EATS_FISH_TOKENS,
SHE_EATS_FISH_TAGS));
+ }
+
+ /** Checks that a model file reloads through the {@link java.io.File}
constructor. */
+ @Test
+ void testModelFileConstructorParsesIdentically(@TempDir Path dir) throws
IOException {
+ final Path file = dir.resolve("depparse-ff-file.bin");
+ model.serialize(file);
+ final FeedforwardDependencyModel reloaded = new
FeedforwardDependencyModel(file.toFile());
+ assertEquals(LANGUAGE, reloaded.getLanguage());
+ assertArrayEquals(networkBytes(model), networkBytes(reloaded));
+ final FeedforwardDependencyParser reloadedParser = new
FeedforwardDependencyParser(reloaded);
+ for (final DependencySample sample : corpus()) {
+ assertEquals(parser.parse(sample.getTokens(), sample.getTags()),
+ reloadedParser.parse(sample.getTokens(), sample.getTags()),
+ Arrays.toString(sample.getTokens()));
+ }
+ }
+
+ /** Checks that a model package with a corrupt network entry is rejected. */
+ @Test
+ void testCorruptNetworkEntryIsRejected() throws IOException {
+ final ByteArrayOutputStream valid = new ByteArrayOutputStream();
+ model.serialize(valid);
+ final ByteArrayOutputStream corrupt = new ByteArrayOutputStream();
+ try (ZipInputStream in = new ZipInputStream(new
ByteArrayInputStream(valid.toByteArray()));
+ ZipOutputStream out = new ZipOutputStream(corrupt)) {
+ ZipEntry entry;
+ while ((entry = in.getNextEntry()) != null) {
+ out.putNextEntry(new ZipEntry(entry.getName()));
+
out.write(FeedforwardDependencyModel.NETWORK_ENTRY_NAME.equals(entry.getName())
+ ? "not a network".getBytes(StandardCharsets.UTF_8) :
in.readAllBytes());
+ out.closeEntry();
+ }
+ }
+ assertThrows(InvalidFormatException.class, () -> new
FeedforwardDependencyModel(
+ new ByteArrayInputStream(corrupt.toByteArray())));
+ }
+
+ /**
+ * Checks that a model package of another component is rejected the way
+ * {@link BaseModel} rejects every foreign package.
+ */
+ @Test
+ void testForeignModelPackageIsRejected() throws IOException {
+ final ByteArrayOutputStream out = new ByteArrayOutputStream();
+ DependencyTestSamples.train(corpus()).serialize(out);
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> new FeedforwardDependencyModel(new
ByteArrayInputStream(out.toByteArray())));
+ assertInstanceOf(InvalidFormatException.class, e.getCause());
+ }
+
+ /**
+ * @return Named invalid network payloads. Never {@code null}.
+ * @throws IOException Thrown if building a payload fails.
+ */
+ private static Stream<Arguments> invalidNetworks() throws IOException {
+ final NetworkBytes defaults = new NetworkBytes();
+ return Stream.of(
+ Arguments.of("wrong magic",
+ new
NetworkBytes().magic("ONLP".getBytes(StandardCharsets.US_ASCII)).bytes()),
+ Arguments.of("unsupported version", new
NetworkBytes().version(2).bytes()),
+ Arguments.of("negative vocabulary size", new
NetworkBytes().wordCount(-1).bytes()),
+ Arguments.of("vocabulary size above the limit",
+ new NetworkBytes().wordCount(ResourceLimits.MAX_ENTRIES +
1).bytes()),
+ Arguments.of("symbol length above the limit",
+ new NetworkBytes().firstWordLength(ResourceLimits.MAX_ENTRIES +
1).bytes()),
+ Arguments.of("truncated body", Arrays.copyOf(defaults.bytes(),
defaults.bytes().length / 2)),
+ Arguments.of("duplicate symbol", new NetworkBytes().tagSymbols(
+ FeedforwardNetwork.UNKNOWN, FeedforwardNetwork.UNKNOWN,
+ DependencyContextGenerator.ROOT_VALUE).bytes()),
+ Arguments.of("missing reserved symbol", new NetworkBytes().tagSymbols(
+ FeedforwardNetwork.UNKNOWN, DependencyContextGenerator.NONE_VALUE,
"NN").bytes()),
+ Arguments.of("overlapping vocabulary ids", new
NetworkBytes().firstTagRow(0).bytes()),
+ Arguments.of("vocabulary block with a gap", new
NetworkBytes().firstTagRow(4).bytes()),
+ Arguments.of("hidden matrix of the wrong width", new
NetworkBytes().hiddenColumns(1).bytes()),
+ Arguments.of("hidden matrix above the cell limit",
+ new NetworkBytes().hiddenRows(OVERSIZED_HIDDEN_ROWS).bytes()),
+ Arguments.of("non-finite weight", new
NetworkBytes().outputBias(Float.NaN).bytes()),
+ Arguments.of("trailing data", concat(defaults.bytes(), new byte[]
{1})),
+ Arguments.of("no SHIFT", new
NetworkBytes().transitions("RIGHT_ARC:root").bytes()),
+ Arguments.of("no RIGHT_ARC", new
NetworkBytes().transitions("SHIFT").bytes()),
+ Arguments.of("invalid transition", new
NetworkBytes().transitions("SHIFT", "JUMP").bytes()),
+ Arguments.of("duplicate transition",
+ new NetworkBytes().transitions("SHIFT", "SHIFT",
"RIGHT_ARC:root").bytes()));
+ }
+
+ /**
+ * Checks that each invalid payload is rejected with an {@link
InvalidFormatException}.
+ *
+ * @param name The defect.
+ * @param payload The network bytes.
+ */
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("invalidNetworks")
+ void testInvalidNetworkIsRejected(String name, byte[] payload) {
+ assertThrows(InvalidFormatException.class,
+ () -> new FeedforwardNetwork.Serializer().create(new
ByteArrayInputStream(payload)));
+ }
+
+ /** Checks that a well-formed minimal payload loads, with a symbol beyond
the old UTF limit. */
+ @Test
+ void testMinimalNetworkWithALongSymbolLoads() throws IOException {
+ final byte[] payload = new
NetworkBytes().tagSymbols(FeedforwardNetwork.UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE,
+ LONG_SYMBOL).bytes();
+ final FeedforwardNetwork network =
+ new FeedforwardNetwork.Serializer().create(new
ByteArrayInputStream(payload));
+ assertEquals(MINIMAL_ROWS + 1, network.embeddings().length);
+ assertEquals(FIRST_LABEL_ROW, network.tagIds().get(LONG_SYMBOL));
+ assertArrayEquals(payload, networkBytes(network));
+ }
+
+ /**
+ * Checks that a symbol of exactly {@link ResourceLimits#MAX_ENTRIES} UTF-8
bytes loads
+ * and reserializes unchanged, while one byte more is refused by the same
check the
+ * trainer applies. The limit is inclusive.
+ *
+ * @throws IOException Thrown if building the payload fails.
+ */
+ @Test
+ void testSymbolOfExactlyTheLimitLoads() throws IOException {
+ final String atLimit = "x".repeat(ResourceLimits.MAX_ENTRIES);
+ final byte[] payload = new
NetworkBytes().tagSymbols(FeedforwardNetwork.UNKNOWN,
+ DependencyContextGenerator.NONE_VALUE,
DependencyContextGenerator.ROOT_VALUE,
+ atLimit).bytes();
+ final FeedforwardNetwork network =
+ new FeedforwardNetwork.Serializer().create(new
ByteArrayInputStream(payload));
+ assertEquals(FIRST_LABEL_ROW, network.tagIds().get(atLimit));
+ assertArrayEquals(payload, networkBytes(network));
+ FeedforwardNetwork.requireStorable(atLimit, "tag vocabulary");
+ assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardNetwork.requireStorable(atLimit + "x", "tag
vocabulary"));
+ }
+
+ /**
+ * Checks the limit on the encoded length rather than the character count: a
symbol of
+ * two-byte characters that fills the limit is accepted, and one more byte
is refused.
+ * Such a symbol is too long in bytes for the length shortcut and takes the
encoding path.
+ */
+ @Test
+ void testMultiByteSymbolIsMeasuredInBytes() {
+ final String twoByte = "\u00e9".repeat(ResourceLimits.MAX_ENTRIES / 2);
+ FeedforwardNetwork.requireStorable(twoByte, "word vocabulary");
+ assertThrows(IllegalArgumentException.class,
+ () -> FeedforwardNetwork.requireStorable(twoByte + "x", "word
vocabulary"));
+ }
+
+ /**
+ * @return Named parameter sets that must be rejected. Never {@code null}.
+ */
+ private static Stream<Arguments> invalidParameters() {
+ return Stream.of(
+ Arguments.of("embedding size 0",
FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM, 0),
+ Arguments.of("hidden size 0",
FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM, 0),
+ Arguments.of("epochs 0", Parameters.ITERATIONS_PARAM, 0),
+ Arguments.of("batch size 0",
FeedforwardDependencyTrainer.BATCH_SIZE_PARAM, 0),
+ Arguments.of("negative learning rate",
FeedforwardDependencyTrainer.LEARNING_RATE_PARAM, -1.0),
+ Arguments.of("NaN learning rate",
FeedforwardDependencyTrainer.LEARNING_RATE_PARAM, Double.NaN),
+ Arguments.of("infinite l2", FeedforwardDependencyTrainer.L2_PARAM,
Double.POSITIVE_INFINITY),
+ Arguments.of("dropout 1", FeedforwardDependencyTrainer.DROPOUT_PARAM,
1.0),
+ Arguments.of("dropout -0.1",
FeedforwardDependencyTrainer.DROPOUT_PARAM, -0.1),
+ Arguments.of("word cutoff -1", Parameters.CUTOFF_PARAM, -1),
+ Arguments.of("embedding size above the limit",
+ FeedforwardDependencyTrainer.EMBEDDING_SIZE_PARAM,
+ FeedforwardDependencyTrainer.MAX_EMBEDDING_SIZE + 1),
+ Arguments.of("hidden size above the limit",
+ FeedforwardDependencyTrainer.HIDDEN_SIZE_PARAM,
+ FeedforwardDependencyTrainer.MAX_HIDDEN_SIZE + 1));
+ }
+
+ /**
+ * Checks that an out-of-range hyperparameter is rejected before any sample
is read.
+ *
+ * @param name The defect.
+ * @param key The parameter key.
+ * @param value The invalid value.
+ */
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("invalidParameters")
+ void testInvalidParametersAreRejected(String name, String key, Object value)
{
+ final TrainingParameters parameters = feedforwardParameters(1, 0.01);
+ if (value instanceof Integer i) {
+ parameters.put(key, i);
+ } else {
+ parameters.put(key, (Double) value);
+ }
+ assertThrows(IllegalArgumentException.class,
Review Comment:
Each row in invalidParameters and invalidNetworks now checks its message.
Overlap and gap rows use a single entry since map order is not fixed.
> Add a feedforward neural dependency parser
> ------------------------------------------
>
> Key: OPENNLP-1991
> URL: https://issues.apache.org/jira/browse/OPENNLP-1991
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Add a feedforward neural implementation of the dependency parser API
> introduced by OPENNLP-547. Use OpenNLP’s standard model container, artifact
> serializers, TrainingParameters, factory, and CLI conventions. Include
> deterministic training, greedy and beam decoding, model round-trip tests, and
> held-out Universal Dependencies evaluation. Address the deferred review
> feedback on shared scoring/training logic and parser-owned caches. Implement
> this without new dependencies.
> The list above may still get this ticket to further split. For now this is a
> placeholder to capture the code that was moved out of OPENNLP-547. I'll edit
> after 547 is merged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
