As someone who has spent a lot of time acting as a maintainer, a code "donation" seems like dangerous gift to accept.
Personally I like the idea of an ecosystem of packages (and repos) managed and maintained by their specialist. That way they can have their own CI, their own release processes and cycles, and "namespaced" notifications. If anything I'd rather push in the direction of breaking Airflow into many smaller packages (core, scheduler, web, ...) as opposed to tacking other projects on top of it. Also arguably Airflow's DSL may be more "common" than CWL. Clearly CWL has more focussed intentions around creating something universal, but to me that doesn't necessarily make it more legitimate or common than other specs (Oozie, Azkaban , Informatica, ...) and should be treated similarly (would we want to include extensions to all these as part of Airflow?). I also prefer the codegen/migration approach (I think the `oozie-to-airflow` tool does that) to allow a path that resolves the common denominator lmitations. How can this tooling expose features that are proper to Airflow (pools, priority weights, xcoms, callbacks!, ...)? Max On Wed, Oct 30, 2019 at 12:32 PM Andrey Kartashov <por...@porter.st> wrote: > My name is Andrey and I'm developer behind CWL-Airflow. > This message is follow up slack conversation. I copy past some messages > from there here. > > > >> Slack chat: > > When I've met CWL team there were no pipeline managers to support it. I've > picked up Airflow to just prove the concept that it is possible. > > The same time I was looking for a pipeline manager to use for > bioinformatic analysis and asked tons of questions from Airflow team as a > result special note in documentation: "Beyond the Horizon". Nevertheless, I > adopted Airflow for our bioinformatic use > > There are more than 200 different pipeline managers, and to believe that > in nearest future there will the only one and perfect one sounds > impossible. So, to exchange pipeline logic between different pipeline > managers and people it is good to have a standard (CWL is a a perfect fit) > like JavaScript standard and different executers, browsers... > > Apache taverna (pipeline manager) is working on adopting CWL for a while > now, we have code it is already working. > > So yes, CWL-Airflow is developed and the use is simple it extends Airflow > DAG class. However it is still required to put .py file with DAG (CWLDAG in > our case) to the dag directory. I would like just to put .cwl file into DAG > directory to simplify the usage > > I'm ready to develop what is necessary, but I'm not quite sure (I'm not a > big expert in airflow code) which way to go, plugin or some native core > code, or ... > > The project by itself lives https://github.com/Barski-lab/cwl-airflow, > there are tons of CWL tests > https://ci.commonwl.org/job/airflow-conformance/ >
