Davis-Zhang-Onehouse opened a new issue, #20699:
URL: https://github.com/apache/hudi/issues/20699

   Part of #20683 (RFC-118, #20684).
   
   The embedding compute core and the `prep_only` embedding transformer: the 
embedding client factory and provider settings, request rate limiting and a 
provider retry loop, micro-batched embedding calls, compute and prep rounds, 
and `EmbeddingTableTransformer`, which `UnstructuredDataSource` in embedding 
mode puts in its required transformer chain.
   
   Known limitation of this stage: for files that yield no chunk, the prep 
round can queue one side-output record per file before it yields its first row, 
so a long run of such files is held in memory. It is fixed later in this chain, 
before the chain completes.
   
   PR: (linked once opened)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to