Davis-Zhang-Onehouse opened a new issue, #20699: URL: https://github.com/apache/hudi/issues/20699
Part of #20683 (RFC-118, #20684). The embedding compute core and the `prep_only` embedding transformer: the embedding client factory and provider settings, request rate limiting and a provider retry loop, micro-batched embedding calls, compute and prep rounds, and `EmbeddingTableTransformer`, which `UnstructuredDataSource` in embedding mode puts in its required transformer chain. Known limitation of this stage: for files that yield no chunk, the prep round can queue one side-output record per file before it yields its first row, so a long run of such files is held in memory. It is fixed later in this chain, before the chain completes. PR: (linked once opened) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
