Davis-Zhang-Onehouse opened a new issue, #20695:
URL: https://github.com/apache/hudi/issues/20695

   Part of #20683 (RFC-118, #20684).
   
   The text preprocessing framework of the embedding stream: the file 
preprocessor interface, chunks and chunk ids, prep counters, failures and side 
outputs, BLOB byte sources, text normalization, chunk packing under a token 
budget, and tokenizers (word count and an OpenAI BPE tokenizer on jtokkit).
   
   PR: (linked once opened)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to