We do! We developed our own little in-house DAG test framework which we could share insights on/would love to hear what other folks are up to. Basically we use mock a DAG's input data, use the BackfillJob API directly to call a DAG in a test, and compare its outputs to the intended result given the inputs. We use docker/docker-compose to manage services, and split our dev and test stack locally so that the tests have their own scheduler and metadata database and so that our CI tool knows how to construct the test stack as well.
We co-opted the BackfillJob API for our own purposes here, but it seemed overly complicated and fragile to start and interact with our own in-test-process executor like we saw in a few of the tests in the Airflow test suite. So I'd be really interested on finding a way to streamline how to describe a test executor for both the Airflow test suite and people's own DAG testing and make that a first class type of API. Laura On Fri, May 5, 2017 at 11:46 AM, Sam Elamin <[email protected]> wrote: > Hi All > > A few people in the Spark community are interested in writing a testing > library for Airflow. We would love anyone who uses Airflow heavily in > production to be involved > > At the moment (AFAIK) testing your DAGs is a bit of a pain, especially if > you want to run them in a CI server > > Is anyone interested in being involved in the discussion? > > Kind Regards > Sam >
