Hey Andrey, I think you should read the "quotes" in the email. For devlists it's generally good practice to quote the parts of emails you refer to and provide your answer (most mailers support it). I try to do it every time I respond..
On Wed, Nov 13, 2019 at 9:42 PM Andrey Kartashov <por...@porter.st> wrote: > I don't quite get what this example should to prove? I agree XML is too > much for everything but CWL is not XML. And our point is not to bring a new > pipeline manager we don't have any other but Airflow. The XML comment was about Oozie-2-Airflow to explain the principles and assumptions we made (that was a question asked by Misha " I need to get more information about oozie-to-airflow approach.". I provided it so that you get all the context and understand how we've done that. I do not say you have to follow the same patterns - I just explained it for comparison. Oozie has i's own specification of DAG <https://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html> (it is also an executor) and oozie-2-airflow is really converting the specification of Oozie to specification of Airflow (it does not replicate the pipline manager - just converts Oozie specification (XML) to Airflow one (Python). > We do not do conversion because we have to change CWL file to send them to > others Universities/Hospitals/Companies to reproduce result and get > improved CWL back to run it on our lovely Airflow. If we convert CWL to > python and change it to python we later would have to convert it to CWL > back. Our CWLDAG is just an Airflow's DAG and can be used as usual within > any Airflow's pipelines > with task.set_upstream or task.set_downstream to integrate CWL pipeline > into bigger Airflow pipeline. That's fine as long as it is one of the ways to get the DAG, but primary use for Airflow is to use DAGs. If you can produce Airflow DAGs from CWL and you do not expect that Airflow DAGs are converted back to CWL (which might be impossible) that's all fine and nice way to execute CWL dags via Airflow. I think we have no argument here. J. > Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
