đź‘Ť On Fri, May 5, 2017 at 3:07 PM, Scott Halgrim <[email protected]> wrote:
> Sounds A+ to me. By “both of you” did you include me? My first response > was just to your email address. > > On May 5, 2017, 11:58 AM -0700, Sam Elamin <[email protected]>, > wrote: > > Ok sounds great folks > > > > Thanks for the detailed response laura! I'll invite both of you to the > > group if you are happy and we can schedule a call for next week? > > > > How does that sound? > > On Fri, 5 May 2017 at 17:41, Laura Lorenz <[email protected]> > wrote: > > > > > We do! We developed our own little in-house DAG test framework which we > > > could share insights on/would love to hear what other folks are up to. > > > Basically we use mock a DAG's input data, use the BackfillJob API > directly > > > to call a DAG in a test, and compare its outputs to the intended result > > > given the inputs. We use docker/docker-compose to manage services, and > > > split our dev and test stack locally so that the tests have their own > > > scheduler and metadata database and so that our CI tool knows how to > > > construct the test stack as well. > > > > > > We co-opted the BackfillJob API for our own purposes here, but it > seemed > > > overly complicated and fragile to start and interact with our own > > > in-test-process executor like we saw in a few of the tests in the > Airflow > > > test suite. So I'd be really interested on finding a way to streamline > how > > > to describe a test executor for both the Airflow test suite and > people's > > > own DAG testing and make that a first class type of API. > > > > > > Laura > > > > > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin <[email protected] > > > wrote: > > > > > > > Hi All > > > > > > > > A few people in the Spark community are interested in writing a > testing > > > > library for Airflow. We would love anyone who uses Airflow heavily in > > > > production to be involved > > > > > > > > At the moment (AFAIK) testing your DAGs is a bit of a pain, > especially if > > > > you want to run them in a CI server > > > > > > > > Is anyone interested in being involved in the discussion? > > > > > > > > Kind Regards > > > > Sam > > > > > > > >
