(2) Usually to share directories between pods you need a ReadWriteMany (RWX) volume plugin for your persistent volume. The easy answer is "just some kind of NFS". Using GCE Persistent Disk/AWS EBS is ReadWriteOnce (one pod at a time). Of course, you need some kind of NFS deployed to just use some kind of NFS.
Brian > On Nov 15, 2019, at 1:14 PM, Misha Kotliar <misha.kotl...@gmail.com> wrote: > > Hi Jarek, Maxime, and Andrey, > > Since we all (well, mostly you, guys) have already found a consensus on this, > I would call it, "political question" :) would you mind giving me some > directions where I can find answers on the following technical problems: > > 1) The more dags I have in a dags folder, the longer time it takes to parse > them all. Taking into account that in my case I have also to parse CWL files, > it takes even more time for such a simple operation. So I was wondering is > there any common solution to approach this issue. Also, I was thinking if I > can use your Plugins mechanism to integrate some additional functionality > such as parsing CWL files directly without making any changes in the core of > Airflow. > > 2) I'm working on running CWL pipelines in Kubernetes through Airflow and one > of the problems that I have to deal with is sharing directories between the > PODs. It looks like Kubernetes doesn't provide the direct solution to this > problem and mostly relies on the platform where it is installed. I will > appreciate if you direct me to the proper discussions/threads where people > solve similar problems. > > Thanks a lot, > Michael > > > > > > > > On 2019/11/15 10:17:30, Jarek Potiuk <jarek.pot...@polidea.com> wrote: >> I am also -1. But I am happy to help with surfacing the CWL integration on >> - both the new package (together with Oozie-2-airflow and maybe other >> converters) and having it easily installable as external Package. I will >> talk to Andrey separately about this so that we do not clutter the devlist. >> >> J. >> >> On Fri, Nov 15, 2019 at 7:37 AM Maxime Beauchemin < >> maximebeauche...@gmail.com> wrote: >> >>> After all the exploration of this topic here in this thread, I'm a pretty >>> hard -1 on this one. >>> >>> I think CWL and CWL-Airflow are great projects, but they can't rely on the >>> Airflow community to evolve/maintain/package this integration. >>> >>> Personally I think that generally and *within reason* (winking at the npm >>> communities ;) that smaller, targeted and loosely coupled packages [and >>> their corresponding smaller repositories with their own set of maintainers] >>> is better than bigger monoliths. Some reasons: >>> * separation of concerns >>> * faster, more targeted builds and test suites >>> * independent release cycles >>> * clearer ownership >>> * independent and adapted level of rigor / styling / standards >>> * more targeted notifications for people watching repos >>> * ... >>> >>> Max >>> >>> On Thu, Nov 14, 2019 at 12:33 PM Andrey Kartashov <por...@porter.st> >>> wrote: >>> >>>> >>>> >>>> I looked at the >>>>> >>>> >>> https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside >>>>> to >>>>> understand what CWL is and that's where I took the descriptor + job (in >>>> Key >>>>> Concepts). >>>>> >>>> >>>> Oh this is an old one, but even new one probably does not reflect the >>> real >>>> picture. >>>> >>>> >>>> OK. So as I understand finally the problem you want to solve - "To make >>>>> Airflow more accessible to people who already use CWL or who will find >>> it >>>>> easier to write dags in CWL". I still think this does not necessarily >>>> have >>>>> to be solved by donating CWL code to Airflow (see below). >>>>> >>>> >>>> I think there are many ways. >>>> >>>> >>>>> Ok. So what you basically say is that you think Airflow community has >>>> more >>>>> capacity than CWL community to maintain CWL converter. >>>> >>>> My understanding CWL community just developing common standard (CWL) not >>>> converters or converter :). For me the CWL-Airflow developer definitely >>>> Airflow community has far more capacity that me alone :) >>>> >>>>> I am not so sure >>>>> about it (precisely because of the lost opportunities). But maybe a >>>> better >>>>> solution is to ask in the airflow community whether there are people >>> who >>>>> could join the CWL-airflow converter and increase the community there. >>>>> >>>> >>>> That probably a good start just to check and see the interest >>>> >>>>> I would not say for the whole community, but I would not feel >>> comfortable >>>>> as a community to take responsibility on the converter without prior >>>>> knowledge and understanding CWL in detail. Especially that it is rather >>>> for >>>>> small group of users (at least initially). But I find CWL as an idea >>> very >>>>> interesting and maybe there are some people in the community who would >>>> love >>>>> to contribute to your project? Suggestion - maybe just ask - here and >>> in >>>>> slack - if there is enough interest in contributing to CWL-Airflow, >>>> rather >>>>> than donating the code to Airflow ? Just promote your project in the >>>>> community and ask for help. >>>> >>>> I tried but have not got any feedback :) but I’m not a promoter or seller >>>> >>>> >>>>> >>>>> I can see this as the best of both worlds - if you find a few people >>> who >>>>> would like to help and get familiar with it and they are also part of >>> the >>>>> Airflow community and we get collective knowledge about it - then >>>>> eventually it might lead to incorporating it to Airflow itself if our >>>>> community gets more familiar with CWL. I think this is the best way to >>>>> achieve the final goal of finally incorporating CWL as part of Airflow. >>>>> >>>> >>>> Works for me >>>> >>>> >>>>> In the meantime - I am happy to help to make Airflow more "CWL >>> friendly" >>>>> for the users - both from documentation and Helm chart POV. >>>>> >>>> >>>> Thank you, I appreciate that, how we proceed? >>>> >>>> >>> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >>
