Re: Airflow Kubernetes pod Operator debug library

Yeah. This seems like a really good feature (I saw that you added it to the Awesome Apache Airflow :). And I will also be happy to help with getting it in the CLI. It's rather simple task (especially after some of the refactorings we've done in

Re: Airflow Kubernetes pod Operator debug library

Hi, Thank you! For the detail explanation. I will read everything and create a ticket with all the expectations. Cheers, Javier El jue., 26 mar. 2020 a las 23:46, Kamil Breguła () escribió: > Hello, > > Can you create a ticket in Github and describe your expectations? This > will allow you

Re: [VOTE] AIP-31: Airflow functional DAG definition

+1 (binding) -- this will be a nice little polish to writing DAGs. -a On Thu Mar 26, 2020 at 3:46 PM, Kamil Breguła wrote: > +1 binding > > > On Thu, Mar 26, 2020 at 3:41 PM Daniel Imberman > wrote: > > > > +1 binding. I volunteer to help implement as well :D > > On Mar 26, 2020, 7:15 AM

Re: Airflow Kubernetes pod Operator debug library

Hello, Can you create a ticket in Github and describe your expectations? This will allow you to discuss this topic more easily. https://github.com/apache/airflow/issues/new/choose First of all, you should read the contribution guide, which contains a very detailed description of how to propose

Re: Airflow Kubernetes pod Operator debug library

Hi, Yes, I missed this feature in the Airflow CLI. For me, the ideal scenario is to have something similar to the helm install --debug --dry-run command ( https://helm.sh/docs/chart_template_guide/debugging/). So the same that for a helm chart you generate multiple k8s templates, for an airflow

Re: Airflow Kubernetes pod Operator debug library

Hello, The idea is fantastic. I like it very much and it will facilitate the work with Kubernethes. I'm just afraid that this tool will not be available when I need it. Did you think to add this tool to CLI of Airflow? This is the best place to share useful tools. When I missed the DAG preview

Airflow Kubernetes pod Operator debug library

Hi all! it's Javier. I am a Data Engineer, I have been working with Airflow and it's Kubernetes integration for almost one year and a half. It's great and it has helped a lot to me and the data engineering team I have been working with. Although I and my colleagues find a recurring issue

Re: How about switching from Celery to RQ?

Clear! Thanks for the explanation! On Thu, Mar 26, 2020 at 8:08 PM Alexandre Vermeerbergen < avermeerber...@gmail.com> wrote: > We prefer ActiveMQ over RabbitMQ because we are used it deploy it in > production at scale, in HA way, with the upgrade stuff and so on. > Rebuilding all this knowledge

Re: How about switching from Celery to RQ?

We prefer ActiveMQ over RabbitMQ because we are used it deploy it in production at scale, in HA way, with the upgrade stuff and so on. Rebuilding all this knowledge with RabbitMQ is an hassle for us... Le jeu. 26 mars 2020 à 19:30, Jarek Potiuk a écrit : > > Out of curiosity why ActiveMQ rather

Re: How about switching from Celery to RQ?

Out of curiosity why ActiveMQ rather than RabbitMQ (except that it is an Apache project that is)? J, On Thu, Mar 26, 2020 at 6:40 PM Alexandre Vermeerbergen < avermeerber...@gmail.com> wrote: > Hi, > > To be honest I found RQ while searching why Airflow isn't currently > supporting ActiveMQ

Re: How about switching from Celery to RQ?

Hi, To be honest I found RQ while searching why Airflow isn't currently supporting ActiveMQ instead of RabbitMQ. I found that RabbitMQ come from a dependency of Celery on Kombu; and I found an incident in Kombu discussing about ActiveMQ support: https://github.com/celery/kombu/issues/548 I

Re: How about switching from Celery to RQ?

I think we once tried to abandon Celery for other stuff (kombu, knative)... If there's no reasonable advantage of RQ I see no reason in switching to it. However, rewriting the "Queue worker" can help a lot :) Also a few days ago one of Celery maintainers expressed interest in contributing to

Re: How about switching from Celery to RQ?

I mean… we’re not planning it (kind of an “if it’s not broke don’t fix it” situation), but I don’t think we’re super set on Celery. Would you be interested in making an AIP to discuss potential benefits? On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen , wrote: > Hi there, > > Looks

How about switching from Celery to RQ?

Hi there, Looks like couple of projects have moved from Celery to RQ: * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq or are planning such move: * https://github.com/getredash/redash/issues/4092 Is such move

Re: Requesting to give details on FacebookAdsHook project

Welcome Chetan. What I also can recommend - we have an "#outreachy" channel in our slack (helpfully pointed out by Michał) and right now there are quite a few applicants for the outreachy that are applying for an internship in airflow. You will find a Pinned item there where I gave all the

Re: Requesting to give details on FacebookAdsHook project

Hello Ben. Here is the documentation about the mailing lists from Apache: https://apache.org/foundation/mailinglists.html you will find all the instructions there. And for folders. I recommend Gmail + their auto-filter. I use Gmail for 10+ years now and with its keyboard shortcuts and

Re: Helm Chart for Airflow

Absolutely! Please do :). The more eyes on this the better! On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman wrote: > @Jarek Steven and I will look into getting KinD to work on GitHub actions > today. If we can get that working, we can donate the helm chart and > possibly even deprecate the old

Re: Helm Chart for Airflow

@Jarek Steven and I will look into getting KinD to work on GitHub actions today. If we can get that working, we can donate the helm chart and possibly even deprecate the old Kubernetes tests in the process. On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk , wrote: > We have A POC for Github Actions

Re: Helm Chart for Airflow

We have A POC for Github Actions done by Tomek, and the blocker there was precisely Kubernetes tests and specifically running kind cluster in the Github Actions https://github.com/PolideaInternal/airflow/pull/542 . In the meantime Jiaje Zhong is trying to make it works on GA we discussed this very

Re: Helm Chart for Airflow

<3 On Thu, Mar 26, 2020 at 3:59 PM Daniel Imberman wrote: > @Jarek I think with the helm chart + prod image we can go even further > than that :). We can test CeleryExecutor, with KEDA autoscaling, and a > bunch of other configurations. > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk , > wrote:

Re: Helm Chart for Airflow

@Steven glad to help you out on that. For the moment we only use travis, but I believe @jarek has been looking to move to GitHub CI/CD. Perhaps we can use this as our first GitHub tests? On Mar 26, 2020, 8:17 AM -0700, Steven Miller , wrote: > Hey I’m happy to PR in the chart and get the CI

Re: Helm Chart for Airflow

Hey I’m happy to PR in the chart and get the CI working to test it. Where do we want to run it? Do you all just use Travis only? Steven On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman wrote: > @Jarek I think with the helm chart + prod image we can go even further > than that :). We can test

Re: Helm Chart for Airflow

@Jarek I think with the helm chart + prod image we can go even further than that :). We can test CeleryExecutor, with KEDA autoscaling, and a bunch of other configurations. On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk , wrote: > Yeah. I meant Custom Resources  not CRDs in my original email :) >

Re: [VOTE] AIP-31: Airflow functional DAG definition

+1 binding On Thu, Mar 26, 2020 at 3:41 PM Daniel Imberman wrote: > > +1 binding. I volunteer to help implement as well :D > On Mar 26, 2020, 7:15 AM -0700, Jarek Potiuk , > wrote: > > +1 binding > > > > On Thu, Mar 26, 2020 at 8:59 AM Sumit Maheshwari > > wrote: > > > > > +1 binding > > > > >

Re: Helm Chart for Airflow

Yeah. I meant Custom Resources not CRDs in my original email :) On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman wrote: > We’re not using CRDs for the tests at the moment. We just have deployment > files. If anything having the helm chart as a part of the airflow repo > could mean that the helm

Re: Helm Chart for Airflow

Cool! Everything is here: https://github.com/apache/airflow/tree/master/scripts/ci/in_container/kubernetes In short: In "kubernetes" dir: setup_kind_cluster.sh + kind-cluster-conf.yaml -> they are use to create a kind Kubernetes cluster. In "docker" subdir:

Re: [VOTE] AIP-31: Airflow functional DAG definition

+1 binding. I volunteer to help implement as well :D On Mar 26, 2020, 7:15 AM -0700, Jarek Potiuk , wrote: > +1 binding > > On Thu, Mar 26, 2020 at 8:59 AM Sumit Maheshwari > wrote: > > > +1 binding > > > > On Thu, Mar 26, 2020 at 9:50 AM Chao-Han Tsai > > wrote: > > > > > +1 binding > > > > > >

Re: Helm Chart for Airflow

We’re not using CRDs for the tests at the moment. We just have deployment files. If anything having the helm chart as a part of the airflow repo could mean that the helm chart becomes the defacto system for testing airflow on kubernetes (we can get rid of all the yams files and run multiple k8s

Re: Helm Chart for Airflow

Yep, we can absolutely pull it into the airflow repo. We've also been building up a test suite that currently runs on CircleCI and uses kind (Kubernetes in Docker) to test several kubernetes versions with some different settings. Right now we're mostly testing the different executors since that

Re: [VOTE] AIP-31: Airflow functional DAG definition

+1 binding On Thu, Mar 26, 2020 at 8:59 AM Sumit Maheshwari wrote: > +1 binding > > On Thu, Mar 26, 2020 at 9:50 AM Chao-Han Tsai > wrote: > > > +1 binding > > > > On Wed, Mar 25, 2020 at 1:56 PM Kaxil Naik wrote: > > > > > +1 binding > > > > > > On Wed, Mar 25, 2020 at 8:55 PM Dan Davydov >

Re: Requesting to give details on FacebookAdsHook project

Hey everyone, How can I get removed from this email list? Or is there a way to have it always go to a specific folder in my inbox? The discussions are great, they are just clogging up my work email and I have missed a handful of important emails because of it. Thank you On Thu, Mar 26, 2020 at

Re: Requesting to give details on FacebookAdsHook project

Hi Chetan, we also have a slack channel. https://apache-airflow-slack.herokuapp.com/ Come on and join. Best regards Michał On Wed, Mar 25, 2020 at 7:16 PM Daniel Imberman wrote: > Hi Chetan, > > Please see our documentation here https://airflow.apache.org/. Please > feel free to reach out

Re: [VOTE] AIP-31: Airflow functional DAG definition

+1 binding On Thu, Mar 26, 2020 at 9:50 AM Chao-Han Tsai wrote: > +1 binding > > On Wed, Mar 25, 2020 at 1:56 PM Kaxil Naik wrote: > > > +1 binding > > > > On Wed, Mar 25, 2020 at 8:55 PM Dan Davydov > > > wrote: > > > > > Sorry +1 (binding) :). > > > > > > On Wed, Mar 25, 2020 at 4:53 PM Dan