krickert opened a new pull request, #1474: URL: https://github.com/apache/opennlp/pull/1474
Draft until #1468 merges; this branch is stacked on it. SentenceVectorsDL.embedAll groups texts by exact tokenized length. On natural text almost every length differs, so most inferences hold one or two texts, and on a GPU a call of mixed lengths is no faster than calling embed in a loop. PaddingStrategy sets how embedAll batches texts of different lengths: - EXACT_LENGTH does not pad and runs one inference per tokenized length. It is the default on the CPU. - LONGEST orders texts by length and pads each batch to its longest text. It is the default on a GPU when the vocabulary has a padding token. - MAX_LENGTH pads every text to the maximum length, for models or accelerators that need a fixed shape. A new constructor takes the strategy, and the onnx provider takes a padding option. The padding id is read from the vocabulary ([PAD], or <pad> for RoBERTa). Padded positions are 0 in the attention mask and mean pooling skips them, so the vectors match up to floating-point rounding when the model honors the mask. Every strategy stays within 16384 token positions per inference. The machine learning chapter documents the option and the GPU default. SentenceVectorsDLBenchmark on all-MiniLM-L6-v2, RTX 4080 SUPER, texts of 4 to 40 words, embedAll time per call with EXACT_LENGTH and with the GPU default: | Texts | EXACT_LENGTH | LONGEST | | ---: | ---: | ---: | | 8 | 3603 us | 862 us | | 32 | 11431 us | 2152 us | | 128 | 23815 us | 7443 us | Tests: batching under each strategy, the token position cap with padding, the padding id from the vocabulary, the device default, and SentenceVectorsDLPaddingEdgesTest for the cap boundaries under every strategy, input order with interleaved lengths, a RoBERTa vocabulary, padding id 0, CLS pooling and unit length scaling, the pooled-output model, single embed under MAX_LENGTH, and CUDA against the CPU (skipped without a CUDA provider or device). tiny-masked.onnx, generated by gen_tiny_masked_model.py next to it, sums the ids times the attention mask, so a test can tell masked padding from unmasked. Fixes #1392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
