I am also -1. But I am happy to help with surfacing the CWL integration on - both the new package (together with Oozie-2-airflow and maybe other converters) and having it easily installable as external Package. I will talk to Andrey separately about this so that we do not clutter the devlist.
J. On Fri, Nov 15, 2019 at 7:37 AM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > After all the exploration of this topic here in this thread, I'm a pretty > hard -1 on this one. > > I think CWL and CWL-Airflow are great projects, but they can't rely on the > Airflow community to evolve/maintain/package this integration. > > Personally I think that generally and *within reason* (winking at the npm > communities ;) that smaller, targeted and loosely coupled packages [and > their corresponding smaller repositories with their own set of maintainers] > is better than bigger monoliths. Some reasons: > * separation of concerns > * faster, more targeted builds and test suites > * independent release cycles > * clearer ownership > * independent and adapted level of rigor / styling / standards > * more targeted notifications for people watching repos > * ... > > Max > > On Thu, Nov 14, 2019 at 12:33 PM Andrey Kartashov <por...@porter.st> > wrote: > > > > > > > I looked at the > > > > > > https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside > > > to > > > understand what CWL is and that's where I took the descriptor + job (in > > Key > > > Concepts). > > > > > > > Oh this is an old one, but even new one probably does not reflect the > real > > picture. > > > > > > OK. So as I understand finally the problem you want to solve - "To make > > > Airflow more accessible to people who already use CWL or who will find > it > > > easier to write dags in CWL". I still think this does not necessarily > > have > > > to be solved by donating CWL code to Airflow (see below). > > > > > > > I think there are many ways. > > > > > > > Ok. So what you basically say is that you think Airflow community has > > more > > > capacity than CWL community to maintain CWL converter. > > > > My understanding CWL community just developing common standard (CWL) not > > converters or converter :). For me the CWL-Airflow developer definitely > > Airflow community has far more capacity that me alone :) > > > > > I am not so sure > > > about it (precisely because of the lost opportunities). But maybe a > > better > > > solution is to ask in the airflow community whether there are people > who > > > could join the CWL-airflow converter and increase the community there. > > > > > > > That probably a good start just to check and see the interest > > > > > I would not say for the whole community, but I would not feel > comfortable > > > as a community to take responsibility on the converter without prior > > > knowledge and understanding CWL in detail. Especially that it is rather > > for > > > small group of users (at least initially). But I find CWL as an idea > very > > > interesting and maybe there are some people in the community who would > > love > > > to contribute to your project? Suggestion - maybe just ask - here and > in > > > slack - if there is enough interest in contributing to CWL-Airflow, > > rather > > > than donating the code to Airflow ? Just promote your project in the > > > community and ask for help. > > > > I tried but have not got any feedback :) but I’m not a promoter or seller > > > > > > > > > > I can see this as the best of both worlds - if you find a few people > who > > > would like to help and get familiar with it and they are also part of > the > > > Airflow community and we get collective knowledge about it - then > > > eventually it might lead to incorporating it to Airflow itself if our > > > community gets more familiar with CWL. I think this is the best way to > > > achieve the final goal of finally incorporating CWL as part of Airflow. > > > > > > > Works for me > > > > > > > In the meantime - I am happy to help to make Airflow more "CWL > friendly" > > > for the users - both from documentation and Helm chart POV. > > > > > > > Thank you, I appreciate that, how we proceed? > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
