Hello Everyone, We recently worked on an open source Machine Learning Work Bench, (not released to public yet) where a "github for ML models" is envisioned. It is fundamentally based on the idea that an ML Application a DAG and is specified via an YAML file and under the hood it is parsed and mapped to an airflow DAG.
We wanted this tool to be an open source alternative to the commercial offerings such as a Amazon SageMaker, Google Cloud ML etc.. AirFlow is seen as one of the alternatives to the orchestrators. Others being Dask for example. I see that the compiling/parsing yaml suggested in this threads fits into what we already did, and there is huge scope to take that idea forward. More about the principles and the scope is discussed in my blog here <https://www.linkedin.com/pulse/eleven-commandments-axiomatic-framework-building-machine-soma-dhavala/?lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_recent_activity_details_shares%3BF%2FXPrwiLS0uxVNIqR3Zkrw%3D%3D&licu=urn%3Ali%3Acontrol%3Ad_flagship3_profile_view_base_recent_activity_details_shares-link>. If there is interest, we can push the project to the public and collaborate. Btw, I initiated a baby experimental project to see the application of KanBans as a way capture the nodes in a DAG here. Technically it is possible to spec an ML App as a Kanban and parse it into an airflow DAG. thanks, -soma That specification needs to support CRUD operations an ML App. yet, where we used AirFlow as one of the DAG orchestrators. On Tue, Oct 2, 2018 at 12:10 PM Sai Phanindhra <phani8...@gmail.com> wrote: > Hi Siddarth, > Your idea sounds interesting and challenging. I would like to > contribute to it. > > On Tue, 2 Oct 2018 at 06:08, siddharth gupta <sid8...@gmail.com> wrote: > > > Hello Folks! > > > > I am working on an Airflow Plugin to auto generate DAG code from the UI > and > > YAML (already have initial design on how it can work etc). I am looking > for > > folks interested in collaborating. Please reach out to me if you are > > interested :) > > > > You can reach out to me at sid8...@gmail.com > > > > Thanks > > Sid > > > > > -- > Sai Phanindhra, > Ph: +91 9043258999 >
