Hi, Swapna. Supporting local models has always been a key milestone on our journey toward Flink AI. After an offline discussion with @DianFu, we found that the current API is not well-suited for PyFlink, as PyFlink prefers to provide its own Python operators.
To better accommodate this requirement and improve extensibility, we propose the following enhancements: *Introduce a pluggable Model Provider interface* Extend the model management framework by adding a standardized interface for model providers, enabling custom implementations — such as a PythonModelProvider — to supply native Python operators (e.g., PythonTableFunctionOperator). *Add built-in model providers for PyTorch and TensorFlow in the Flink repository* Implement and integrate first-party model providers for popular frameworks like PyTorch and TensorFlow. These providers will implement the new interface and be maintained within the main Flink codebase. *Optimize resource usage via cached files and batched inference* Leverage `pipeline.cached-files` to manage model weight files efficiently, avoiding redundant downloads across tasks. Additionally, support batched inference to maximize hardware utilization (e.g., GPU/TPU throughput) and improve overall performance. Best, Shengkai Swapna Marru <[email protected]> 于2025年9月22日周一 23:36写道： > Hi Devs, > > > I am interested in learning more about MODEL, ML_PREDICT, and ML_EVALUATE > functionalities added in the following FLIP. > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-437%3A+Support+ML+Models+in+Flink+SQL > > > I see the original FLIP has extensibility to local model providers in > Flink. > > > Is there a way to do pluggable local model providers in Python? Like, say, > generate embeddings using Sentence transformer models running locally in > Flink. > > > An option could be to introduce a Model Provider factory implementation in > Java that internally uses a predict function in Python . But I see this > puts in a lot of work related to Java to Python communication/translation > inside the provider. > > > Something like PythonRuntimeProvider along with PredictRuntimeProvider / > AsyncRuntimeProvider which can handle Java -> Python translations out of > the box would be helpful to de-duplicate that effort. > > > Can you please point to, if there are any discussions related to this > already ? Or any other ways to achieve the same? Please share your > thoughts. > > > -Thanks, > > Swapna Marru >
