Bit late notice but the call is happening today at 9 15 utc so in about 30 mins or so
It will be recorded but if anyone would like to join in on the discussion the hangout link is https://hangouts.google.com/hangouts/_/mbkr6xassnahjjonpuvrirxbnae Regards Sam On Fri, 5 May 2017 at 21:35, Ali Uz <[email protected]> wrote: > I am also very interested in seeing how this turns out. Even though we > don't have a testing framework in-place on the project I am working on, I > would very much like to contribute to some general framework for testing > DAGs. > > As of now we are just implementing dummy tasks that test our actual tasks > and verify if the given input produces the expected output. Nothing crazy > and certainly not flexible in the long run. > > > On Fri, 5 May 2017 at 22:59, Sam Elamin <[email protected]> wrote: > > > Haha yes Scott you are in! > > On Fri, 5 May 2017 at 20:07, Scott Halgrim <[email protected]> > > wrote: > > > > > Sounds A+ to me. By “both of you” did you include me? My first response > > > was just to your email address. > > > > > > On May 5, 2017, 11:58 AM -0700, Sam Elamin <[email protected]>, > > > wrote: > > > > Ok sounds great folks > > > > > > > > Thanks for the detailed response laura! I'll invite both of you to > the > > > > group if you are happy and we can schedule a call for next week? > > > > > > > > How does that sound? > > > > On Fri, 5 May 2017 at 17:41, Laura Lorenz <[email protected]> > > > wrote: > > > > > > > > > We do! We developed our own little in-house DAG test framework > which > > we > > > > > could share insights on/would love to hear what other folks are up > > to. > > > > > Basically we use mock a DAG's input data, use the BackfillJob API > > > directly > > > > > to call a DAG in a test, and compare its outputs to the intended > > result > > > > > given the inputs. We use docker/docker-compose to manage services, > > and > > > > > split our dev and test stack locally so that the tests have their > own > > > > > scheduler and metadata database and so that our CI tool knows how > to > > > > > construct the test stack as well. > > > > > > > > > > We co-opted the BackfillJob API for our own purposes here, but it > > > seemed > > > > > overly complicated and fragile to start and interact with our own > > > > > in-test-process executor like we saw in a few of the tests in the > > > Airflow > > > > > test suite. So I'd be really interested on finding a way to > > streamline > > > how > > > > > to describe a test executor for both the Airflow test suite and > > > people's > > > > > own DAG testing and make that a first class type of API. > > > > > > > > > > Laura > > > > > > > > > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin < > [email protected] > > > > > wrote: > > > > > > > > > > > Hi All > > > > > > > > > > > > A few people in the Spark community are interested in writing a > > > testing > > > > > > library for Airflow. We would love anyone who uses Airflow > heavily > > in > > > > > > production to be involved > > > > > > > > > > > > At the moment (AFAIK) testing your DAGs is a bit of a pain, > > > especially if > > > > > > you want to run them in a CI server > > > > > > > > > > > > Is anyone interested in being involved in the discussion? > > > > > > > > > > > > Kind Regards > > > > > > Sam > > > > > > > > > > > > > > > > >
