>
> I hope ideas in that treads and questions do not interfere with current
> discussions. I have not mentioned anything about `cwl descriptor + job`
> (Michael K probably did :(), but in current implementation CWL Description
> is static and lives long live :) and then one can attach a sensor to the
> top of CWLDAG to check files in a directory or as we do we trigger that
> dag `airflow trigger_dag --conf "{\"job\":$(cat ./hg19.job)}"
> "bowtie-index"` so we starting it with kind of xcom data filled in.
>
I looked at the
https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside
to
understand what CWL is and that's where I took the descriptor + job (in Key
Concepts).
> I never thought about rewriting all the python DAGs code :) into CWL. I
> did say that there are people who decided to use CWL and they already
> limited to environment they have like toil, IBM HPC and others who picked
> up Airflow for them CWL is the solution.
OK. So as I understand finally the problem you want to solve - "To make
Airflow more accessible to people who already use CWL or who will find it
easier to write dags in CWL". I still think this does not necessarily have
to be solved by donating CWL code to Airflow (see below).
I do not believe that it is possible to convert anybody (so I separate
> existing users and new users), but I strongly believe that there is a huge
> biology, chemistry, astronomers, physics, etc community which would
> appreciate CWL in Airflow
>
So possibly the solution is that we make it easy to configure Airflow with
the CWL installed - but not necessarily donating code to Airflow. We have
various ways of installing Airflow and we are just now working on a new
Airflow Website, landing page that describes various ways of installing
Airflow. We are also going to have an official helm chart updated with
production image of Airflow and it already has various ways of installation
(Celery/Kubernetes executor etc). What I see can be done there is to have a
dedicated section on how to install Airflow locally with CWL enabled (you
could provide a description on how to do it) and have an option in helm
chart to support installation with cwl-airflow installed as separate POD in
kubernetes. This way you achieve the goal of making Airflow available for
CWL users without donating CWL code. I am happy to help with this - as I am
involved with both tasks.
> > What's the problem you want to solve by
> > donating the code to Airflow team ?
> >
>
> Because I do it in free time, and at some point I just might have no free
> time.
>
Ok. So what you basically say is that you think Airflow community has more
capacity than CWL community to maintain CWL converter. I am not so sure
about it (precisely because of the lost opportunities). But maybe a better
solution is to ask in the airflow community whether there are people who
could join the CWL-airflow converter and increase the community there.
I would not say for the whole community, but I would not feel comfortable
as a community to take responsibility on the converter without prior
knowledge and understanding CWL in detail. Especially that it is rather for
small group of users (at least initially). But I find CWL as an idea very
interesting and maybe there are some people in the community who would love
to contribute to your project? Suggestion - maybe just ask - here and in
slack - if there is enough interest in contributing to CWL-Airflow, rather
than donating the code to Airflow ? Just promote your project in the
community and ask for help.
I can see this as the best of both worlds - if you find a few people who
would like to help and get familiar with it and they are also part of the
Airflow community and we get collective knowledge about it - then
eventually it might lead to incorporating it to Airflow itself if our
community gets more familiar with CWL. I think this is the best way to
achieve the final goal of finally incorporating CWL as part of Airflow.
In the meantime - I am happy to help to make Airflow more "CWL friendly"
for the users - both from documentation and Helm chart POV.
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
