After all the exploration of this topic here in this thread, I'm a pretty hard -1 on this one.
I think CWL and CWL-Airflow are great projects, but they can't rely on the Airflow community to evolve/maintain/package this integration. Personally I think that generally and *within reason* (winking at the npm communities ;) that smaller, targeted and loosely coupled packages [and their corresponding smaller repositories with their own set of maintainers] is better than bigger monoliths. Some reasons: * separation of concerns * faster, more targeted builds and test suites * independent release cycles * clearer ownership * independent and adapted level of rigor / styling / standards * more targeted notifications for people watching repos * ... Max On Thu, Nov 14, 2019 at 12:33 PM Andrey Kartashov <por...@porter.st> wrote: > > > I looked at the > > > https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside > > to > > understand what CWL is and that's where I took the descriptor + job (in > Key > > Concepts). > > > > Oh this is an old one, but even new one probably does not reflect the real > picture. > > > OK. So as I understand finally the problem you want to solve - "To make > > Airflow more accessible to people who already use CWL or who will find it > > easier to write dags in CWL". I still think this does not necessarily > have > > to be solved by donating CWL code to Airflow (see below). > > > > I think there are many ways. > > > > Ok. So what you basically say is that you think Airflow community has > more > > capacity than CWL community to maintain CWL converter. > > My understanding CWL community just developing common standard (CWL) not > converters or converter :). For me the CWL-Airflow developer definitely > Airflow community has far more capacity that me alone :) > > > I am not so sure > > about it (precisely because of the lost opportunities). But maybe a > better > > solution is to ask in the airflow community whether there are people who > > could join the CWL-airflow converter and increase the community there. > > > > That probably a good start just to check and see the interest > > > I would not say for the whole community, but I would not feel comfortable > > as a community to take responsibility on the converter without prior > > knowledge and understanding CWL in detail. Especially that it is rather > for > > small group of users (at least initially). But I find CWL as an idea very > > interesting and maybe there are some people in the community who would > love > > to contribute to your project? Suggestion - maybe just ask - here and in > > slack - if there is enough interest in contributing to CWL-Airflow, > rather > > than donating the code to Airflow ? Just promote your project in the > > community and ask for help. > > I tried but have not got any feedback :) but I’m not a promoter or seller > > > > > > I can see this as the best of both worlds - if you find a few people who > > would like to help and get familiar with it and they are also part of the > > Airflow community and we get collective knowledge about it - then > > eventually it might lead to incorporating it to Airflow itself if our > > community gets more familiar with CWL. I think this is the best way to > > achieve the final goal of finally incorporating CWL as part of Airflow. > > > > Works for me > > > > In the meantime - I am happy to help to make Airflow more "CWL friendly" > > for the users - both from documentation and Helm chart POV. > > > > Thank you, I appreciate that, how we proceed? > >
