why airflow create tmp config files in Worker's /tmp dir

I just found out that airflow create some tmp config files in /tmp dir on servers that run Celery workers. All the files have name tmpxxx, and the same content like: {"core": {"dags_folder": "/data/data366/airflow/dags", "base_log_folder": "/data/data366/airflow/logs", "remote_logging": "Fal

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc2

+1 (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Apr 3, 2020 at 11:10 AM, Kaxil Naik wrote: +1 (binding) On Fri, Apr 3, 2020 at 6:55 PM Kaxil Naik wrote: > Hello Airflow Community, > > This is a call for the vote to

Re: [CANCELLED][VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Yeah. You can run full Airflow with Breeze this way: ./breeze --install-airflow-version 1.10.10rc1 --python 3.6 restart --backend postgres --db-reset You are dropped into a shell ... then ... tmux Ctrl+B " (or Ctrl + B + %) several times `airflow webserver` in one terminal `airflow scheduler`

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc2

+1 (binding) On Fri, Apr 3, 2020 at 6:55 PM Kaxil Naik wrote: > Hello Airflow Community, > > This is a call for the vote to release Apache Airflow version 1.10.10. > > The release candidate: > https://dist.apache.org/repos/dist/dev/airflow/1.10.10rc2/ > > *apache-airflow-1.10.10rc2-source.tar.gz

Re: [CANCELLED][VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Hi Jarek Did you mean I could check out to `v1-10-test` with git command and start with Breeze to test RC version, instead of using pip install command? And maybe we could add this to RELEASE.rst doc to help Breeze users more easier to test pre-release package? Best Wish — Jiajie

[VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc2

Hello Airflow Community, This is a call for the vote to release Apache Airflow version 1.10.10. The release candidate: https://dist.apache.org/repos/dist/dev/airflow/1.10.10rc2/ *apache-airflow-1.10.10rc2-source.tar.gz* is a source release that comes with INSTALL instructions. *apache-airflow-1.

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Might be time for another community video hang out on breeze, Jarek - i'd like to watch you using it again. On Fri, Apr 3, 2020 at 1:47 PM Daniel Imberman wrote: > @jarek when you’re running on breeze how do you actually run airflow s.t. > I can see the UI/run DAGs? It seems like it downloads ev

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

@jarek when you’re running on breeze how do you actually run airflow s.t. I can see the UI/run DAGs? It seems like it downloads everything, but it doesn’t seem like a full airflow deployment. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2]

Re: [CANCELLED][VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

N . no worries with Breeze installation it's really fast. I can test almost all combinations in very little time. The previous time it was much more difficult. J. On Fri, Apr 3, 2020 at 7:08 PM Ash Berlin-Taylor wrote: > Sorry, my fault :) > > Most of your functional tests should apply to R

Re: [CANCELLED][VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Sorry, my fault :) Most of your functional tests should apply to RC2 as well: https://github.com/apache/airflow/compare/1.10.10rc1...1dbd50c

Re: [CANCELLED][VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Aaaargh. I was Just in the middle of my tests :) On Fri, Apr 3, 2020 at 6:50 PM Kaxil Naik wrote: > Canceling this vote to fix a bug discovered with Timezone and Tooltips. > > RC2 will be out shortly with the fix > > On Fri, Apr 3, 2020 at 3:53 PM Anita Fronczak wrote: > > > Casting a vote (

[CANCELLED][VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Canceling this vote to fix a bug discovered with Timezone and Tooltips. RC2 will be out shortly with the fix On Fri, Apr 3, 2020 at 3:53 PM Anita Fronczak wrote: > Casting a vote (non binding) +1 > > pt., 3 kwi 2020 o 16:35 Kaxil Naik napisał(a): > > > Casting my own vote: +1 (binding) > > >

Re: [VOTE] AIP-32: Airflow REST API

Looking forward to it! +1! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Apr 3, 2020 at 4:06 AM, Kaxil Naik wrote: I had the same concern with the Auth model (Had commented on the AIP on confluence: https://cwiki.apache.org/conflu

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Casting a vote (non binding) +1 pt., 3 kwi 2020 o 16:35 Kaxil Naik napisał(a): > Casting my own vote: +1 (binding) > > Regards, > Kaxil > > On Fri, Apr 3, 2020 at 2:59 PM Daniel Imberman > wrote: > > > @jarek once we merge the helm chart we can add a breeze command to do the > > same with the k

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Casting my own vote: +1 (binding) Regards, Kaxil On Fri, Apr 3, 2020 at 2:59 PM Daniel Imberman wrote: > @jarek once we merge the helm chart we can add a breeze command to do the > same with the k8sexecutor on a kubernetes cluster :) > > via Newton Mail > [ > https://cloudmagic.com/k/d/mailapp?

Re: Helm Chart for Airflow

Hi Jarek, Congrats on getting the image out! This is going to make a lot of things easier :). We’ve actually been working on separating out KinD as well (and in fact moving it out of travis and into GitHub issues). Let me circle back with Steven (the engineer who’s been working on that) and get

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

@jarek once we merge the helm chart we can add a breeze command to do the same with the k8sexecutor on a kubernetes cluster :) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Apr 3, 2020 at 6:41 AM, Jarek Potiuk wrote: Yep. And t

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Yep. And the breeze command actually uses PyPi to install it :) On Fri, Apr 3, 2020 at 3:37 PM Kaxil Naik wrote: > For convenience, RC is also uploaded on *PyPI* so you can install it using > *pip > install apache-airflow==1.10.10rc1* > > Regards, > Kaxil > > On Fri, Apr 3, 2020 at 1:49 PM Jarek

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

For convenience, RC is also uploaded on *PyPI* so you can install it using *pip install apache-airflow==1.10.10rc1* Regards, Kaxil On Fri, Apr 3, 2020 at 1:49 PM Jarek Potiuk wrote: > One more comment. > > I believe one of the slow-downs with testing was that it was a few steps to > install - a

Reminder about CFP for the Online Airflow Summit 2020 - just 2 days left!

Just a reminder that there are still 2 days left to submit for the CFP for Online Airflow Summit this summer! The CFP is available at https://pretalx.com/apache-airflow-summit-bay-area-2020/cfp and will be open until the end of Saturday 4th of April PST -- Jarek Potiuk Polidea

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

One more comment. I believe one of the slow-downs with testing was that it was a few steps to install - and taking into account that not all PMCs are actively developing Airflow they could have refrained from testing it. But this problem is gone now. If you checkout the latest master, it should b

[VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

Hello Airflow Community, This is a call for the vote to release Apache Airflow version 1.10.10. The release candidate: https://dist.apache.org/repos/dist/dev/airflow/1.10.10rc1/ *apache-airflow-1.10.10rc1-source.tar.gz* is a source release that comes with INSTALL instructions. *apache-airflow-1.

Re: [VOTE] AIP-32: Airflow REST API

I had the same concern with the Auth model (Had commented on the AIP on confluence: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-32%3A+Airflow+REST+API?focusedCommentId=148641141#comment-148641141) on 9 Mar :-D But happy if we would re-use the Auth from Fab to begin with. +1 On Fri,

Re: [VOTE] AIP-32: Airflow REST API

Cool! On Fri, Apr 3, 2020 at 12:44 PM Kamil Breguła wrote: > The vote was open for more than 72 hours, so I hope we have a common > consensus and we don't need to vote again. > > This vote has passed with 4 +1 votes and no -1 votes. > > +1 votes: > * Kamil Breguła > * Tomek Urbaszek > * Jarek Po

Re: [VOTE] AIP-32: Airflow REST API

The vote was open for more than 72 hours, so I hope we have a common consensus and we don't need to vote again. This vote has passed with 4 +1 votes and no -1 votes. +1 votes: * Kamil Breguła * Tomek Urbaszek * Jarek Potiuk * Ash Berlin-Taylor -1 votes: N/A The adventure will start soon. 😺 Kam

Re: [VOTE] AIP-32: Airflow REST API

I think in the future we can treat them as code change I think. I cannot imagine us voting on PRs to merge :). That would be quite a burden. What I really like about the voting process that there is also -0.9, -0.5, -0. +0 .. I feel we should start using those more often. I'd say the -1 from Dan w

Re: [VOTE] AIP-32: Airflow REST API

It's not clear to me if votes on AIPs count as a code change or not -- for example we passed the Dag serialzation AIP with a -1 from Dan. Anyway, with the change to the proposed permissions model I convert my vote to a +1 anyway :) (Thanks for doing making that change Kamil. We can always revisit

Apache Airflow Production Image in "Alpha" - master, v1-10-test and upcoming 1.10.10

Hello Everyone, In the current master, v1-10-test (and soon upcoming 1.10.10 release), we have an Alpha-quality production-ready docker image that should be ready for testing. All the details of the image are described in https://github.com/apache/airflow/blob/master/IMAGES.rst You can either bu

Re: [VOTE] AIP-32: Airflow REST API

I believe -1 from Ash suspended the vote (actually the wors used in the Apache Voting rules is "kill the proposal") and he must withdraw it in order to continue. See https://www.apache.org/foundation/voting.html#votes-on-code-modification J. On Fri, Apr 3, 2020 at 10:59 AM Kamil Breguła wrote:

Re: [VOTE] AIP-32: Airflow REST API

I don't think it will ever be suspended. We have a lower limit of 72 hours. This allows anyone to speak regardless of their time zone and prevents changes that are not widely recognized. However, if we want to talk longer, I do not think that it will be necessary to suspend voting. On Fri, Apr 3,

Re: [VOTE] AIP-32: Airflow REST API

Looks good to me. +1. Binding Can we resume the vote? J. On Fri, Apr 3, 2020 at 9:17 AM Kamil Breguła wrote: > Hello, > > That's right The authentication will be based on Connexion. However, > this will not be necessary. Users will be able to add a new > authentication method if necessary. Fo

Re: Helm Chart for Airflow

Hello Greg and Daniel, Production image is merged and it's also going to be released in 1.10.10. For now, I will manually tag the release in DockerHub when we release but we will automate that in the future. Currently, we have *airflow:master* and *airflow:v1-10-test* images (for all supported pyt

Re: Let's talk Airflow 2.0

6. Prod image is there :) -> alpha quality for now and ready to be tested. And I have plans to add more tests and with Daniel/Greg integrate it with the helm chart and make both - the image and helm the "official" ones. Alpha version of the image will also be released with 1.10.10 so that people ca

Re: Let's talk Airflow 2.0

> > Move (tested) components out of contrib folder > > https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E AIP-21 is done. We have a follow-up in the form of system tests and backport packages, but this is not a blocker. We ca

Re: Let's talk Airflow 2.0

4. Improve Scheduler performance and reliability The performance was improved by my work. https://lists.apache.org/thread.html/r4b0a6385d263defaf0fe733a48fb291e3a763da2f172ca1931a652b7%40%3Cdev.airflow.apache.org%3E In Q2, my team will continue to work on performance and reliability. We want to p

Re: [VOTE] AIP-32: Airflow REST API

Hello, That's right The authentication will be based on Connexion. However, this will not be necessary. Users will be able to add a new authentication method if necessary. For example, you can easily integrate Airflow with your own identity proxy, which will provide permissions using the JWT token