krickert opened a new issue, #1471: URL: https://github.com/apache/opennlp/issues/1471
There is no benchmark for the ONNX embeddings, so changes to SentenceVectorsDL cannot be compared before and after. Each inference also copies its tensors through the Java heap: the inputs go from long arrays into heap buffers that ONNX Runtime copies again, and getValue() builds one float array per token of the output. Proposal: - Add SentenceVectorsDLBenchmark (JMH) to opennlp-dl, with the same jmh profile as opennlp-runtime. It times embed per text and embedAll for calls of 1 to 128 texts of mixed length, on the CPU or a CUDA device. - Write the inputs into direct buffers and let ONNX Runtime write the output into a direct buffer that pooling reads in place. Branch: https://github.com/apache/opennlp/tree/feat/embeddings-timings-tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
