Re: [DISCUSS] AIP-12 Persist DAG into DB

2019-07-17 Thread Tao Feng
Do we reach any consensus on this topic /AIP? I think persisting DAG is pretty important actually. -Tao On Tue, Mar 12, 2019 at 3:01 AM Kevin Yang wrote: > Hi Fokko, > > As a large cluster maintainer, I’m not a big fan of large DAG files > neither. But I’m not sure if I’ll consider this bad pra

Re: Running Pylint/Flake/Mypy/Docs/Licence checks locally

2019-07-17 Thread Maxime Beauchemin
`pre-commit` is pretty great if someone wants to set it up. https://pre-commit.com/ Here's what our configuration looks like for Superset: https://github.com/apache/incubator-superset/blob/master/.pre-commit-config.yaml It's super fast too as it only executes for the files that have been touched

Running Pylint/Flake/Mypy/Docs/Licence checks locally

2019-07-17 Thread Jarek Potiuk
Hello everyone, Following merging AIP-10, I have a kind request to you. In order to limit failed checks on Travis CI (and make your development workflow faster). You should be able to run all static code analysis tests very easily - and they will be run consistently with Travis CI runs. You can r

Big change in the way we run tests (AIP-10 completed)

2019-07-17 Thread Jarek Potiuk
Hello everyone, I would like to announce that I just merged the last (3/3) part of AIP-10 - big change in the way we run CI tests for Airflow. Nothing changes in the tests, but the way we run them in Travis CI and locally is quite different. You should be able to reproduce Travis CI tests super-e

Re: [VOTE] Release Airflow 1.10.4 from RC3

2019-07-17 Thread Bolke de Bruin
+1, binding. Thanks ash! Verstuurd vanaf mijn iPad > Op 17 jul. 2019 om 20:45 heeft Kaxil Naik het volgende > geschreven: > > +1 (binding) Ran DAGs in both UIs. LGTM > > Great Job Ash. Appreciate all the effort you put into this. > > Regards, > Kaxil > > On Wed, Jul 17, 2019 at 10:30 PM F

Re: [VOTE] Release Airflow 1.10.4 from RC3

2019-07-17 Thread Kaxil Naik
+1 (binding) Ran DAGs in both UIs. LGTM Great Job Ash. Appreciate all the effort you put into this. Regards, Kaxil On Wed, Jul 17, 2019 at 10:30 PM Felix Uellendall wrote: > +1 (non-binding) flawlessly passed all my tests via rbac and classic ui. > Also tested it on production-level dags. Grea

Re: [VOTE] Release Airflow 1.10.4 from RC3

2019-07-17 Thread Felix Uellendall
+1 (non-binding) flawlessly passed all my tests via rbac and classic ui. Also tested it on production-level dags. Great job Ash, thanks :) Kind regards, Felix Sent from ProtonMail mobile Original Message On Jul 17, 2019, 16:41, Ash Berlin-Taylor wrote: > Thanks Andrii - I mis

Re: [VOTE] Release Airflow 1.10.4 from RC3

2019-07-17 Thread Ash Berlin-Taylor
Thanks Andrii - I missed it to when reviewing the PR so it's my fault too. Make the PR against the v1-10-stable branch please. -ash > On 17 Jul 2019, at 14:25, Andrii Soldatenko > wrote: > > @Ash, i'll fix wrong section bug. > > Sorry about that. > > On Wed, Jul 17, 2019 at 4:11 PM Robin Ed

Re: [VOTE] Release Airflow 1.10.4 from RC3

2019-07-17 Thread Philippe Gagnon
+1 (non-binding) On Mon, Jul 15, 2019 at 10:17 AM Ash Berlin-Taylor wrote: > Hello Airflow community, > > This email is calling a vote on the release, which will last for 72 hours > (2019-07-08 15:15 Z), and until three binding votes have been cast. > Consider this my (binding) +1. > > Airflow 1

Re: [VOTE] Release Airflow 1.10.4 from RC3

2019-07-17 Thread Andrii Soldatenko
@Ash, i'll fix wrong section bug. Sorry about that. On Wed, Jul 17, 2019 at 4:11 PM Robin Edwards wrote: > +1 (none binding) - been running in production since RC2. > > Thanks for all your hard work > > R > > On Tue, 16 Jul 2019 at 21:15, Ash Berlin-Taylor wrote: > > > > Thanks for testing. >

Re: [VOTE] Release Airflow 1.10.4 from RC3

2019-07-17 Thread Robin Edwards
+1 (none binding) - been running in production since RC2. Thanks for all your hard work R On Tue, 16 Jul 2019 at 21:15, Ash Berlin-Taylor wrote: > > Thanks for testing. > > On 1) everyone should run upgradedb on every upgrade. The behaviour of not > running it wasn't great. > > 2) I thought we

Re: Best practises for debugging Macros or Plugins?

2019-07-17 Thread Jarek Potiuk
Ah cool Ash. I Did not know about the 'render' command :). J. On Wed, Jul 17, 2019 at 12:49 PM Ash Berlin-Taylor wrote: > And as for ways you can get the macro in a way you debug it - the `airflow > render` subcommand would help here. > > The other option would be to create a unit test file som

Re: Best practises for debugging Macros or Plugins?

2019-07-17 Thread Ash Berlin-Taylor
And as for ways you can get the macro in a way you debug it - the `airflow render` subcommand would help here. The other option would be to create a unit test file something like what I've put below. (This just tests the default `{{ ds }}` macro, but this could be extended to test a custom mac

Re: Best practises for debugging Macros or Plugins?

2019-07-17 Thread Jarek Potiuk
The best way to debug is to use ipdb debugger or remote debugging feature from PyCharm/IntelliJ. * For the first one ipdb - there is a note in upcoming Docker CI image environment: https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#running-individual-tests . You can