Sounds A+ to me. By “both of you” did you include me? My first response was just to your email address.
On May 5, 2017, 11:58 AM -0700, Sam Elamin <[email protected]>, wrote: > Ok sounds great folks > > Thanks for the detailed response laura! I'll invite both of you to the > group if you are happy and we can schedule a call for next week? > > How does that sound? > On Fri, 5 May 2017 at 17:41, Laura Lorenz <[email protected]> wrote: > > > We do! We developed our own little in-house DAG test framework which we > > could share insights on/would love to hear what other folks are up to. > > Basically we use mock a DAG's input data, use the BackfillJob API directly > > to call a DAG in a test, and compare its outputs to the intended result > > given the inputs. We use docker/docker-compose to manage services, and > > split our dev and test stack locally so that the tests have their own > > scheduler and metadata database and so that our CI tool knows how to > > construct the test stack as well. > > > > We co-opted the BackfillJob API for our own purposes here, but it seemed > > overly complicated and fragile to start and interact with our own > > in-test-process executor like we saw in a few of the tests in the Airflow > > test suite. So I'd be really interested on finding a way to streamline how > > to describe a test executor for both the Airflow test suite and people's > > own DAG testing and make that a first class type of API. > > > > Laura > > > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin <[email protected] > > wrote: > > > > > Hi All > > > > > > A few people in the Spark community are interested in writing a testing > > > library for Airflow. We would love anyone who uses Airflow heavily in > > > production to be involved > > > > > > At the moment (AFAIK) testing your DAGs is a bit of a pain, especially if > > > you want to run them in a CI server > > > > > > Is anyone interested in being involved in the discussion? > > > > > > Kind Regards > > > Sam > > > > >
