Hi Jarek, Maxime, and Andrey, Since we all (well, mostly you, guys) have already found a consensus on this, I would call it, "political question" :) would you mind giving me some directions where I can find answers on the following technical problems:
1) The more dags I have in a dags folder, the longer time it takes to parse them all. Taking into account that in my case I have also to parse CWL files, it takes even more time for such a simple operation. So I was wondering is there any common solution to approach this issue. Also, I was thinking if I can use your Plugins mechanism to integrate some additional functionality such as parsing CWL files directly without making any changes in the core of Airflow. 2) I'm working on running CWL pipelines in Kubernetes through Airflow and one of the problems that I have to deal with is sharing directories between the PODs. It looks like Kubernetes doesn't provide the direct solution to this problem and mostly relies on the platform where it is installed. I will appreciate if you direct me to the proper discussions/threads where people solve similar problems. Thanks a lot, Michael On 2019/11/15 10:17:30, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > I am also -1. But I am happy to help with surfacing the CWL integration on > - both the new package (together with Oozie-2-airflow and maybe other > converters) and having it easily installable as external Package. I will > talk to Andrey separately about this so that we do not clutter the devlist. > > J. > > On Fri, Nov 15, 2019 at 7:37 AM Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > > > After all the exploration of this topic here in this thread, I'm a pretty > > hard -1 on this one. > > > > I think CWL and CWL-Airflow are great projects, but they can't rely on the > > Airflow community to evolve/maintain/package this integration. > > > > Personally I think that generally and *within reason* (winking at the npm > > communities ;) that smaller, targeted and loosely coupled packages [and > > their corresponding smaller repositories with their own set of maintainers] > > is better than bigger monoliths. Some reasons: > > * separation of concerns > > * faster, more targeted builds and test suites > > * independent release cycles > > * clearer ownership > > * independent and adapted level of rigor / styling / standards > > * more targeted notifications for people watching repos > > * ... > > > > Max > > > > On Thu, Nov 14, 2019 at 12:33 PM Andrey Kartashov <por...@porter.st> > > wrote: > > > > > > > > > > > I looked at the > > > > > > > > > https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside > > > > to > > > > understand what CWL is and that's where I took the descriptor + job (in > > > Key > > > > Concepts). > > > > > > > > > > Oh this is an old one, but even new one probably does not reflect the > > real > > > picture. > > > > > > > > > OK. So as I understand finally the problem you want to solve - "To make > > > > Airflow more accessible to people who already use CWL or who will find > > it > > > > easier to write dags in CWL". I still think this does not necessarily > > > have > > > > to be solved by donating CWL code to Airflow (see below). > > > > > > > > > > I think there are many ways. > > > > > > > > > > Ok. So what you basically say is that you think Airflow community has > > > more > > > > capacity than CWL community to maintain CWL converter. > > > > > > My understanding CWL community just developing common standard (CWL) not > > > converters or converter :). For me the CWL-Airflow developer definitely > > > Airflow community has far more capacity that me alone :) > > > > > > > I am not so sure > > > > about it (precisely because of the lost opportunities). But maybe a > > > better > > > > solution is to ask in the airflow community whether there are people > > who > > > > could join the CWL-airflow converter and increase the community there. > > > > > > > > > > That probably a good start just to check and see the interest > > > > > > > I would not say for the whole community, but I would not feel > > comfortable > > > > as a community to take responsibility on the converter without prior > > > > knowledge and understanding CWL in detail. Especially that it is rather > > > for > > > > small group of users (at least initially). But I find CWL as an idea > > very > > > > interesting and maybe there are some people in the community who would > > > love > > > > to contribute to your project? Suggestion - maybe just ask - here and > > in > > > > slack - if there is enough interest in contributing to CWL-Airflow, > > > rather > > > > than donating the code to Airflow ? Just promote your project in the > > > > community and ask for help. > > > > > > I tried but have not got any feedback :) but I’m not a promoter or seller > > > > > > > > > > > > > > I can see this as the best of both worlds - if you find a few people > > who > > > > would like to help and get familiar with it and they are also part of > > the > > > > Airflow community and we get collective knowledge about it - then > > > > eventually it might lead to incorporating it to Airflow itself if our > > > > community gets more familiar with CWL. I think this is the best way to > > > > achieve the final goal of finally incorporating CWL as part of Airflow. > > > > > > > > > > Works for me > > > > > > > > > > In the meantime - I am happy to help to make Airflow more "CWL > > friendly" > > > > for the users - both from documentation and Helm chart POV. > > > > > > > > > > Thank you, I appreciate that, how we proceed? > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
