I’d be happy to help any anyway I can! Andrew
On Thu, Jul 18, 2019 at 11:12 AM Jarek Potiuk <[email protected]> wrote: > Hey Andrew, > > Fantastic! Would you mind if I use some of this in the docs for the > upcoming Simplified Development Workflow - AIP-7 (look out for the webinar > next week on Thursday) ? > Maybe you would also like to help with testing of the new "Breeze" > simplified development environment in Windows + WSL environment? > > One of the aims of Breeze is to make a consistent environment for > development and debugging (including IDE) but Windows was so far not in > scope. I did a lot of testing for MacOS and some for Linux (with other's > help) but Windows + WSL is a bit out of my reach for now. Also some of the > upcoming improvements like pre-commit checks are Linux/MacOS tested for > now. > > That would be awesome to also target Windows. So far we targeted mostly > MacOS/Linux Developers - however now after AIP-10 pretty much all of the > heavy lifting is done in Docker, and the only integration points you have > with local development are bash scripts. SO maybe using WSL on Windows and > having someone who can test it and use it daily, is a good idea to bring it > as "officially supported" environment and have all Breeze features > available there as well ? > > Some more information about the upcoming Breeze in AIP-7 > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow > > J. > > On Thu, Jul 18, 2019 at 3:36 PM Andrew Harmon <[email protected]> > wrote: > > > Here's a doc I created on using Airflow with Windows/WSL. You can ignore > > most of it, but towards the end I describe how I setup PyCharm to debug. > > https://www.astronomer.io/guides/airflow-wsl/ > > > > Andrew > > > > On Thu, Jul 18, 2019 at 9:33 AM Andrew Harmon <[email protected]> > > wrote: > > > > > I use PyCharm as well to drop in break points. I usually run "airflow > > > test" or "airflow render" in the script path of the debug config, then > > step > > > through my code in the IDE. > > > > > > Andrew > > > > > > On Wed, Jul 17, 2019 at 7:48 AM Jarek Potiuk <[email protected] > > > > > wrote: > > > > > >> Ah cool Ash. I Did not know about the 'render' command :). > > >> > > >> J. > > >> > > >> On Wed, Jul 17, 2019 at 12:49 PM Ash Berlin-Taylor <[email protected]> > > >> wrote: > > >> > > >> > And as for ways you can get the macro in a way you debug it - the > > >> `airflow > > >> > render` subcommand would help here. > > >> > > > >> > The other option would be to create a unit test file something like > > what > > >> > I've put below. > > >> > > > >> > (This just tests the default `{{ ds }}` macro, but this could be > > >> extended > > >> > to test a custom macro) > > >> > > > >> > -ash > > >> > > > >> > __unittest__ > > >> > from datetime import datetime > > >> > from unittest import TestCase > > >> > > > >> > from airflow import DAG > > >> > from airflow.models import DagRun, TaskInstance > > >> > from airflow.utils.db import create_session > > >> > from airflow.utils.timezone import make_aware > > >> > from airflow.utils.state import State > > >> > from airflow.operators.bash_operator import BashOperator > > >> > > > >> > > > >> > class MacroTest(TestCase): > > >> > def setUp(self): > > >> > self.dag = DAG( > > >> > dag_id="unittest", > > >> > start_date=datetime(2019, 1, 1) > > >> > ) > > >> > > > >> > def tearDown(self): > > >> > with create_session() as session: > > >> > session.query(DagRun).filter(DagRun.dag_id == > > >> > self.dag.dag_id).delete() > > >> > session.query(TaskInstance).filter(TaskInstance.dag_id > == > > >> > self.dag.dag_id).delete() > > >> > > > >> > def test_macro(self): > > >> > # Use BashOperator as it's got easy macros to render > > >> > op = BashOperator( > > >> > task_id="test", > > >> > dag=self.dag, > > >> > bash_command='true', > > >> > env={'KEY': '{{ ds }}'}, > > >> > ) > > >> > > > >> > with create_session() as session: > > >> > # Set up the object to get a full template context > > >> > dr = self.dag.create_dagrun( > > >> > run_id="test", > > >> > execution_date=make_aware(datetime(2019, 1, 1)), > > >> > state=State.SCHEDULED, > > >> > session=session, > > >> > ) > > >> > > > >> > ti = dr.get_task_instance(op.task_id) > > >> > ti.task = op > > >> > > > >> > ti.render_templates() > > >> > > > >> > self.assertEqual(op.env['KEY'], '2019-01-01') > > >> > > > >> > > > >> > > On 17 Jul 2019, at 11:07, Jarek Potiuk <[email protected]> > > >> wrote: > > >> > > > > >> > > The best way to debug is to use ipdb debugger or remote debugging > > >> feature > > >> > > from PyCharm/IntelliJ. > > >> > > > > >> > > * For the first one ipdb - there is a note in upcoming Docker CI > > image > > >> > > environment: > > >> > > > > >> > > > >> > > > https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#running-individual-tests > > >> > > . > > >> > > You can set the ipdb trace method in any place of the python code > > you > > >> > have > > >> > > and you should be able to debug from the console. > > >> > > > > >> > > * For remote debugging it is a bit more difficult and requires > paid > > >> > version > > >> > > of PyCharm/IntelliJ - I have another PR that I am going to propose > > >> soon - > > >> > > the Breeze environment and you can find description on how to > setup > > >> > remote > > >> > > debugging here: > > >> > > > > >> > > > >> > > > https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst#debugging-airflow-breeze-tests-in-ide > > >> > > > > >> > > > > >> > > Additionally Pycharm/IntelliJ allow you to debug various templates > > >> > > (including Jinja templates). You can likely combine this with > remote > > >> > > debugging but I have never used it this way): > > >> > > https://blog.jetbrains.com/pycharm/2017/06/template-debugging/ > > >> > > > > >> > > J. > > >> > > > > >> > > On Tue, Jul 16, 2019 at 9:23 PM Shaw, Damian P. < > > >> > > [email protected]> wrote: > > >> > > > > >> > >> Hi all, > > >> > >> > > >> > >> I've just been working on updating many of my macros to include > > >> logic to > > >> > >> handle holiday calendars. There was a small mistake in one of the > > >> macros > > >> > >> and I found it very difficult to debug, I'm not sure where a > > default > > >> > logger > > >> > >> in the plugins would log out to and there was no obvious way to > run > > >> the > > >> > >> code outside of airflow or somewhere I could add a breakpoint. > > >> > >> > > >> > >> My issue is solved now, but does anyone have an advise on how to > > >> debug > > >> > >> Macros or Plugins for future cases? > > >> > >> > > >> > >> Thanks, > > >> > >> Damian > > >> > >> > > >> > >> > > >> > > > >> > > > =============================================================================== > > >> > >> > > >> > >> Please access the attached hyperlink for an important electronic > > >> > >> communications disclaimer: > > >> > >> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > >> > >> > > >> > > > >> > > > =============================================================================== > > >> > >> > > >> > >> > > >> > > > > >> > > > > >> > > -- > > >> > > > > >> > > Jarek Potiuk > > >> > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > > > >> > > M: +48 660 796 129 <+48660796129> > > >> > > [image: Polidea] <https://www.polidea.com/> > > >> > > > >> > > > >> > > >> -- > > >> > > >> Jarek Potiuk > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > >> M: +48 660 796 129 <+48660796129> > > >> [image: Polidea] <https://www.polidea.com/> > > >> > > > > > > > > > -- > > > Andrew Harmon > > > (202) 615-6433 > > > > > > > > > -- > > Andrew Harmon > > (202) 615-6433 > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Sent from Gmail Mobile
