Hello Daniel, In my company we use airflow to update our ML models and to predict.
As we use kubernetesOperator to trigger jobs, each ML DAG are similar and ML/Data science engineer can reuse a template and choose which type of machine they needs (highcpu, highmem, GPU or not..etc) We have a process in place describe in the second part of this article (Industrializing machine learning pipeline) : https://medium.com/dailymotion/collaboration-between-data-engineers-data-analysts-and-data-scientists-97c00ab1211f Hope this help. Germain. On 19/02/2020 16:42, "Daniel Imberman" <[email protected]> wrote: Hello everyone! I’m working on a few proposals to make Apache Airflow more friendly for ML/Data science use-cases, and I wanted to reach out in hopes of hearing from people that are using/wish to use Airflow for ML. If you have any opinions on the subject, I’d love to hear what you’re all working on! Current questions I’m looking into: 1. How do you use Airflow for your ML? Has it worked out well for you? 2. Are there any features that would improve your experience of building models on Airflow? 3. Have you built anything on top of airflow/around Airflow to aide you in this process? Thank you so much for your time! via Newton Mail [https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloudmagic.com%2Fk%2Fd%2Fmailapp%3Fct%3Ddx%26cv%3D10.0.32%26pv%3D10.14.6%26source%3Demail_footer_2&data=02%7C01%7Cgermain.tanguy%40dailymotion.com%7C2f6dfaee7bdf467a651108d7b552411d%7C37530da3f7a748f4ba462dc336d55387%7C0%7C0%7C637177237197962425&sdata=s4YovJSTKgLqi%2BAjRXfQFVntaPUyTO%2BTAlJnCIVygYE%3D&reserved=0]
