Re: Donating code to add Common Workflow Language import to Airflow

2019-11-14 Thread Andrey Kartashov
I looked at the > https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside > to > understand what CWL is and that's where I took the descriptor + job (in Key > Concepts). > Oh this is an old one, but even new one probably does not reflect the real picture. OK. So

Re: Donating code to add Common Workflow Language import to Airflow

2019-11-13 Thread Andrey Kartashov
Hi J, You always start with hello Andrey (so no I feel uncomfortable :) > I'd like to start by saying I'm a big big fan of Airflow, otherwise there > > wouldn’t be CWL-Airflow :). I like everything about Airflow development. > > Especially the way of adding extra packages to Airflow like Kubernete

Re: Donating code to add Common Workflow Language import to Airflow

2019-11-13 Thread Andrey Kartashov
On 2019/11/13 23:28:22, Jarek Potiuk wrote: > On Wed, Nov 13, 2019 at 11:33 PM Andrey Kartashov wrote: > > > Previous email gives more info why I believe CWL should be part of > > Airflow. Or should I elaborate more? > > > > Well yes. I still miss answers to t

Re: Donating code to add Common Workflow Language import to Airflow

2019-11-13 Thread Andrey Kartashov
I'm sorry I have not used quotes. But now, I do not understand at all what we are discussing. The email system mixed time and order. Can we all have a phone/skype/hangouts/jeet.si call where we can quickly reduce the problem and start a new thread? To answer 10 questions from one email and then

Re: Donating code to add Common Workflow Language import to Airflow

2019-11-13 Thread Andrey Kartashov
ckaged with our PyPI > package or live in the Airflow repo? > > Max > > On Wed, Nov 13, 2019 at 12:42 PM Andrey Kartashov wrote: > > > I don't quite get what this example should to prove? I agree XML is too > > much for everything but CWL is not XML. And our po

Re: Donating code to add Common Workflow Language import to Airflow

2019-11-13 Thread Andrey Kartashov
I believe there are just two big questions: - what is CWL/why CWL-Airflow exists/should be CWL or CWL-Airflow be a part of Apache-Airflow -technical implementation of CWL-Airflow I'd like to start by saying I'm a big big fan of Airflow, otherwise there wouldn’t be CWL-Airflow :). I like everyth

Re: Donating code to add Common Workflow Language import to Airflow

2019-11-13 Thread Andrey Kartashov
I don't quite get what this example should to prove? I agree XML is too much for everything but CWL is not XML. And our point is not to bring a new pipeline manager we don't have any other but Airflow. We do not do conversion because we have to change CWL file to send them to others Universities

Re: Donating code to add Common Workflow Language import to Airflow

2019-10-30 Thread Andrey Kartashov
My name is Andrey and I'm developer behind CWL-Airflow. This message is follow up slack conversation. I copy past some messages from there here. >> Slack chat: When I've met CWL team there were no pipeline managers to support it. I've picked up Airflow to just prove the concept that it is poss