[ https://issues.apache.org/jira/browse/OPENNLP-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705061#comment-17705061 ]
ASF GitHub Bot commented on OPENNLP-1442: ----------------------------------------- jzonthemtn commented on code in PR #523: URL: https://github.com/apache/opennlp/pull/523#discussion_r1148536600 ########## opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java: ########## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.dl; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.HashMap; +import java.util.Map; +import java.util.stream.Stream; + +import ai.onnxruntime.OrtEnvironment; +import ai.onnxruntime.OrtSession; + +import opennlp.tools.tokenize.Tokenizer; + +/** + * Base class for OpenNLP deep-learning classes using ONNX Runtime. + */ +public abstract class AbstractDL { + + public static final String INPUT_IDS = "input_ids"; + public static final String ATTENTION_MASK = "attention_mask"; + public static final String TOKEN_TYPE_IDS = "token_type_ids"; + + protected OrtEnvironment env; + protected OrtSession session; + protected Tokenizer tokenizer; + protected Map<String, Integer> vocab; + + /** + * Loads a vocabulary file from disk. + * @param vocabFile The vocabulary file. + * @return A map of vocabulary words to integer IDs. + * @throws IOException Thrown if the vocabulary file cannot be opened and read. + */ + public Map<String, Integer> loadVocab(final File vocabFile) throws IOException { + + final Map<String, Integer> vocab = new HashMap<>(); + + int counter = 0; + + try (Stream<String> lines = Files.lines(Path.of(vocabFile.getPath()))) { + + lines.forEach(line -> { + vocab.put(line, counter); Review Comment: Thanks for catching that. > Use ONNX Runtime to support sentence-transformers > ------------------------------------------------- > > Key: OPENNLP-1442 > URL: https://issues.apache.org/jira/browse/OPENNLP-1442 > Project: OpenNLP > Issue Type: Task > Components: Deep Learning > Reporter: Jeff Zemerick > Assignee: Jeff Zemerick > Priority: Major > > Use ONNX Runtime to support sentence-transformers. OpenNLP should be able to > generate embeddings using an ONNX model. -- This message was sent by Atlassian Jira (v8.20.10#820010)