On 2019/11/13 23:28:22, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > On Wed, Nov 13, 2019 at 11:33 PM Andrey Kartashov <por...@porter.st> wrote: > > > Previous email gives more info why I believe CWL should be part of > > Airflow. Or should I elaborate more? > > > > Well yes. I still miss answers to the basic questions (we keep on asking > them with Maxime): > > 1) We still need to understand more why do you insist so strongly on the > code being taken over (or written) by Airflow team rather than maintained > by CWL team? >
I probably could not give you the whole answer. However, at least my understanding, which might be wrong :). CWL was invented and developed by teams like airflow team. They have their own pipeline managers. CWL team’s idea was not to write CWL for every existing pipeline manager (in java, go, c/c++, python, other languages). The idea was to create a standard simple and complete enough to satisfy 90% of existing pipelines. I by myself and Michael Kotliar are bioinformaticians in a biology wet lab, main task is data analysis, statistical methods etc. We most likely would support CWL-Airflow till our last breath, but we are not “immortal”, and users who would like to use Airflow with CWL feel that. CWL - has a strong community and followers Airflow - has an excellent community, followers and fast growing CWL-Airflow - two free time developers? > 2) What's wrong with the current model where you provide converters to > Airflow and maintain it as separate packages. > I like current model and we use it in production. We install our solution with sequential executer on laptops/desktops, with local on a server, celery on (workers) cluster. For me with good experience in programming, software architecture, admin skills it is not a big deal to install Airflow, CWL-Airflow then add a dag into dag directory, wrap CWL into CWLDAG put cwl-wrapper python into the same dag directory. For a novice student/PhD/professor :) with biology background it would be difficult, a more natural way to put a CWL file without wrappers into dag directory. > 3) As we wrote before - we are happy to promote CWL as one of the "sources" > of workflow imports through our website and make easy to install > cwl-airflow via [cwl] extra. What do you want to achieve by having code in > Airflow repo/donated to Apache community? > For me it is more like a heritage, the kid will finally join the family.
