Indeed - I think it is a recurring pattern I see learning from different
users. We saw a number of people writing specific solution to their
problems and using Airflow underneath, Airflow's big strengths are:
a) distributed execution model built-in
b) solid and well tested integrations with multip
On Sat, Nov 16, 2019 at 2:25 PM Jarek Potiuk
wrote:
> >
> > 1) The more dags I have in a dags folder, the longer time it takes to
> > parse them all. Taking into account that in my case I have also to parse
> > CWL files, it takes even more time for such a simple operation. So I was
> > wondering
Thanks a lot Jarek and Brian for your answers.
I will learn from the links you provided and when I have other questions I will
post them in the correspondent discussion threads. As for the main topic of
this thread, I agree that currently, it might be more reasonable to keep it in
as a separate
>
> 1) The more dags I have in a dags folder, the longer time it takes to
> parse them all. Taking into account that in my case I have also to parse
> CWL files, it takes even more time for such a simple operation. So I was
> wondering is there any common solution to approach this issue. Also, I wa
(2) Usually to share directories between pods you need a ReadWriteMany (RWX)
volume plugin for your persistent volume. The easy answer is "just some kind of
NFS". Using GCE Persistent Disk/AWS EBS is ReadWriteOnce (one pod at a time).
Of course, you need some kind of NFS deployed to just use som
Hi Jarek, Maxime, and Andrey,
Since we all (well, mostly you, guys) have already found a consensus on this, I
would call it, "political question" :) would you mind giving me some directions
where I can find answers on the following technical problems:
1) The more dags I have in a dags folder, t
I am also -1. But I am happy to help with surfacing the CWL integration on
- both the new package (together with Oozie-2-airflow and maybe other
converters) and having it easily installable as external Package. I will
talk to Andrey separately about this so that we do not clutter the devlist.
J.
After all the exploration of this topic here in this thread, I'm a pretty
hard -1 on this one.
I think CWL and CWL-Airflow are great projects, but they can't rely on the
Airflow community to evolve/maintain/package this integration.
Personally I think that generally and *within reason* (winking a
I looked at the
> https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside
> to
> understand what CWL is and that's where I took the descriptor + job (in Key
> Concepts).
>
Oh this is an old one, but even new one probably does not reflect the real
picture.
OK. So
>
> I hope ideas in that treads and questions do not interfere with current
> discussions. I have not mentioned anything about `cwl descriptor + job`
> (Michael K probably did :(), but in current implementation CWL Description
> is static and lives long live :) and then one can attach a sensor to t
On Thu, Nov 14, 2019 at 2:58 AM Andrey Kartashov wrote:
> I'm sorry I have not used quotes. But now, I do not understand at all what
> we are discussing. The email system mixed time and order. Can we all have a
> phone/skype/hangouts/jeet.si call where we can quickly reduce the problem
> and star
Hi J,
You always start with hello Andrey (so no I feel uncomfortable :)
> I'd like to start by saying I'm a big big fan of Airflow, otherwise there
> > wouldn’t be CWL-Airflow :). I like everything about Airflow development.
> > Especially the way of adding extra packages to Airflow like Kubernete
On 2019/11/13 23:28:22, Jarek Potiuk wrote:
> On Wed, Nov 13, 2019 at 11:33 PM Andrey Kartashov wrote:
>
> > Previous email gives more info why I believe CWL should be part of
> > Airflow. Or should I elaborate more?
> >
>
> Well yes. I still miss answers to the basic questions (we keep on ask
I'm sorry I have not used quotes. But now, I do not understand at all what we
are discussing. The email system mixed time and order. Can we all have a
phone/skype/hangouts/jeet.si call where we can quickly reduce the problem and
start a new thread? To answer 10 questions from one email and then
On Wed, Nov 13, 2019 at 11:33 PM Andrey Kartashov wrote:
> Previous email gives more info why I believe CWL should be part of
> Airflow. Or should I elaborate more?
>
Well yes. I still miss answers to the basic questions (we keep on asking
them with Maxime):
1) We still need to understand more
Hello Andrey,
I have nothing about emotions, I actually think emotions are super
important and drive our passion to do stuff, but I'd love if you can answer
the questions we ask by quoting and answering them.
I'd like to start by saying I'm a big big fan of Airflow, otherwise there
> wouldn’t be
Hey Andrey,
I think you should read the "quotes" in the email. For devlists it's
generally good practice to quote the parts of emails you refer to and
provide your answer (most mailers support it). I try to do it every time I
respond..
On Wed, Nov 13, 2019 at 9:42 PM Andrey Kartashov wrote:
> I
Previous email gives more info why I believe CWL should be part of Airflow. Or
should I elaborate more?
On 2019/11/13 21:09:52, Maxime Beauchemin wrote:
> The big question is why can't it just be on its own Github repository and
> in its own PyPI package? Why does it have to be packaged with ou
I believe there are just two big questions:
- what is CWL/why CWL-Airflow exists/should be CWL or CWL-Airflow be a part of
Apache-Airflow
-technical implementation of CWL-Airflow
I'd like to start by saying I'm a big big fan of Airflow, otherwise there
wouldn’t be CWL-Airflow :). I like everyth
The big question is why can't it just be on its own Github repository and
in its own PyPI package? Why does it have to be packaged with our PyPI
package or live in the Airflow repo?
Max
On Wed, Nov 13, 2019 at 12:42 PM Andrey Kartashov wrote:
> I don't quite get what this example should to prov
I don't quite get what this example should to prove? I agree XML is too much
for everything but CWL is not XML. And our point is not to bring a new pipeline
manager we don't have any other but Airflow. We do not do conversion because we
have to change CWL file to send them to others Universities
I think it would be great to hear from others what they think - but here
are my thoughts:
1) CWL-Airflow package is more of a converter. We created a special class
> CWLDAG (inherited from DAG), which allows us to set the path to the CWL
> file (a separate YAML file). Once parsed by Airflow the ne
Hi everyone,
Sorry for the late response:) I'm Michael. I wrote the post that Michael R.
Crusoe mentioned at the very beginning of this conversation. Both Andrey and I
work on CWL-Airflow. I will try to answer some of your questions.
1) CWL-Airflow package is more of a converter. We created a s
Hello Andrey,
I think both myself and Maxime - we asked some important questions. If you
want to proceed with the donation, I think it would be great if you let us
know what do you think about the issues we mentioned. I know also Michael
whom I met at the workshops in Berlin - was very interested
As someone who has spent a lot of time acting as a maintainer, a code
"donation" seems like dangerous gift to accept.
Personally I like the idea of an ecosystem of packages (and repos) managed
and maintained by their specialist. That way they can have their own CI,
their own release processes and
My name is Andrey and I'm developer behind CWL-Airflow.
This message is follow up slack conversation. I copy past some messages from
there here.
>> Slack chat:
When I've met CWL team there were no pipeline managers to support it. I've
picked up Airflow to just prove the concept that it is poss
I have not looked at the details of the code yet, it I would like to
understand few things:
1) is the CWL package more of a converter of CWL to Python DAG files (that
can then be scheduled as usual) or whether it is running alongside of the
scheduler and schedules tasks and operators separately us
Dear Airflow People,
Following up to
http://mail-archives.apache.org/mod_mbox/airflow-dev/201812.mbox/%3c9a85db58-cef7-4b9a-9cfb-20cd5bfce...@gmail.com%3e
The CWL AIrflow paper has been published!
Michael Kotliar, Andrey V Kartashov, Artem Barski, CWL-Airflow: a
lightweight pipeline manager supp
28 matches
Mail list logo
