Re: Donating code to add Common Workflow Language import to Airflow

Hi J, You always start with hello Andrey (so no I feel uncomfortable :) > I'd like to start by saying I'm a big big fan of Airflow, otherwise there > > wouldn’t be CWL-Airflow :). I like everything about Airflow development. > > Especially the way of adding extra packages to Airflow like Kubernete

Re: Donating code to add Common Workflow Language import to Airflow

On 2019/11/13 23:28:22, Jarek Potiuk wrote: > On Wed, Nov 13, 2019 at 11:33 PM Andrey Kartashov wrote: > > > Previous email gives more info why I believe CWL should be part of > > Airflow. Or should I elaborate more? > > > > Well yes. I still miss answers to the basic questions (we keep on ask

Re: Donating code to add Common Workflow Language import to Airflow

I'm sorry I have not used quotes. But now, I do not understand at all what we are discussing. The email system mixed time and order. Can we all have a phone/skype/hangouts/jeet.si call where we can quickly reduce the problem and start a new thread? To answer 10 questions from one email and then

Re: Donating code to add Common Workflow Language import to Airflow

On Wed, Nov 13, 2019 at 11:33 PM Andrey Kartashov wrote: > Previous email gives more info why I believe CWL should be part of > Airflow. Or should I elaborate more? > Well yes. I still miss answers to the basic questions (we keep on asking them with Maxime): 1) We still need to understand more

Re: Donating code to add Common Workflow Language import to Airflow

Hello Andrey, I have nothing about emotions, I actually think emotions are super important and drive our passion to do stuff, but I'd love if you can answer the questions we ask by quoting and answering them. I'd like to start by saying I'm a big big fan of Airflow, otherwise there > wouldn’t be

Re: Donating code to add Common Workflow Language import to Airflow

Hey Andrey, I think you should read the "quotes" in the email. For devlists it's generally good practice to quote the parts of emails you refer to and provide your answer (most mailers support it). I try to do it every time I respond.. On Wed, Nov 13, 2019 at 9:42 PM Andrey Kartashov wrote: > I

Re: [Final review] Airflow website + need your input to the web copy

I was not involved in the design :) - I am just a "user" really for the website design :). So I will let the designers to answer it :). J. On Wed, Nov 13, 2019 at 7:38 PM Chris Hendrixson wrote: > Hey Jarek, that makes sense. I trust your instincts. Did you ever try > multiple shapes? Not just

Re: Donating code to add Common Workflow Language import to Airflow

Previous email gives more info why I believe CWL should be part of Airflow. Or should I elaborate more? On 2019/11/13 21:09:52, Maxime Beauchemin wrote: > The big question is why can't it just be on its own Github repository and > in its own PyPI package? Why does it have to be packaged with ou

Re: Donating code to add Common Workflow Language import to Airflow

I believe there are just two big questions: - what is CWL/why CWL-Airflow exists/should be CWL or CWL-Airflow be a part of Apache-Airflow -technical implementation of CWL-Airflow I'd like to start by saying I'm a big big fan of Airflow, otherwise there wouldn’t be CWL-Airflow :). I like everyth

Re: Donating code to add Common Workflow Language import to Airflow

The big question is why can't it just be on its own Github repository and in its own PyPI package? Why does it have to be packaged with our PyPI package or live in the Airflow repo? Max On Wed, Nov 13, 2019 at 12:42 PM Andrey Kartashov wrote: > I don't quite get what this example should to prov

Re: Donating code to add Common Workflow Language import to Airflow

I don't quite get what this example should to prove? I agree XML is too much for everything but CWL is not XML. And our point is not to bring a new pipeline manager we don't have any other but Airflow. We do not do conversion because we have to change CWL file to send them to others Universities

Re: [Final review] Airflow website + need your input to the web copy

Hey Jarek, that makes sense. I trust your instincts. Did you ever try multiple shapes? Not just squares but circles and triangles too? —Chris On Wed, Nov 13, 2019 at 10:24 AM Jarek Potiuk wrote: > Hey Chris, > > I looked at both and i like the "confetti" one much more than the "boxy" > one (Afte

Re: New operator emr_cluster_name_to_id viability

Sure. While most EMR clusters are ephemeral, some of our use cases required persistent EMR clusters since the apps they run are short and run on a short interval so the overhead of creating a new EMR cluster is too high. In these cases I want to make sure that if the cluster dies and is replaced

Re: New operator emr_cluster_name_to_id viability

My initial thought is that doesn't quite sound like a whole operator, but a useful function to add to the EmrHook. Could you describe in a little bit more detail how you use it? -a > On 13 Nov 2019, at 15:40, Aviem Zur wrote: > > Hi, > > I've created a new operator and want to check viabilit

New operator emr_cluster_name_to_id viability

Hi, I've created a new operator and want to check viability to contribute it to airflow/contrib The operator is called: emr_cluster_name_to_id Given an EMR cluster name will return id of the first live cluster found with a matching name. This is useful for users with persistent EMR clusters they

Re: [Final review] Airflow website + need your input to the web copy

Hey Chris, I looked at both and i like the "confetti" one much more than the "boxy" one (After the "shuriken" logo, I got used to naming our design proposals - makes it much easier to talk about them). One reason is that the "boxy" one looks far more cluttered and distracting. But I also have a b

Re: [Final review] Airflow website + need your input to the web copy

I wanted to see how this box idea would play out on the actual website so I mocked it up here: https://imgur.com/iOTBLuX Hmmm, not sure about it. What does everyone think? Hopefully, it can at least spark an idea from you Polidea folks. I also don't hate the original confetti idea — it just feels

Re: [Final review] Airflow website + need your input to the web copy

I made a quick sketch of the idea that I mentioned a few minutes ago. I basically just created boxes at various widths, used the Airflow colors as the border and used the same color at 10% opacity inside the box. Here are two options, one with rounded corners and one with fully rounded edges. This

Re: [Final review] Airflow website + need your input to the web copy

Can we reuse the animation for a new Airflow 404 page? ;-) On 13 Nov 2019, at 15:01, Chris Hendrixson mailto:ch...@astronomer.io>> wrote: I like the concept of the floating squares, though, as-is, they seem a bit like confetti. I think the Slack implementation works really well because, with th

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

Hello Gaetan, Just to let you know (and keep you informed) - the "AIP-26 - Production-ready Airflow Docker Image and helm chart " is now approved and I am working on it. I split it in

Re: [Final review] Airflow website + need your input to the web copy

I like the concept of the floating squares, though, as-is, they seem a bit like confetti. I think the Slack implementation works really well because, with the recognizable logos, it still feels relevant to Slack. An idea: What if they were a different shape? Something a bit more relevant to Airflo

Re: Donating code to add Common Workflow Language import to Airflow

I think it would be great to hear from others what they think - but here are my thoughts: 1) CWL-Airflow package is more of a converter. We created a special class > CWLDAG (inherited from DAG), which allows us to set the path to the CWL > file (a separate YAML file). Once parsed by Airflow the ne

Re: Drop Python 3.5 support?

+1 On Tue, Nov 12, 2019 at 11:54 PM Bolke de Bruin wrote: > Yeah, I was a bit lazy ;-). Cheers Jarek. > > Sent from my iPhone > > > On 13 Nov 2019, at 08:16, Driesprong, Fokko > wrote: > > > > +1 > > > > Thanks Jarek for the background information :-) > > > > Cheers, Fokko > > > > Op wo 13 nov