Thanks Sam! Is there a part 2 to the video? If not, can you post the "next steps" notes you took whenever you have a chance?
Cheers, Arnie On Mon, May 8, 2017 at 3:08 PM Sam Elamin <[email protected]> wrote: > Hi Folks > > For those of you who missed it, you can catch the discussion from the link > on this tweet <https://twitter.com/samelamin/status/861703888298225670> > > Please do share and feel free to get involved as the more feedback we get > the better the library we create is :) > > Regards > Sam > > On Mon, May 8, 2017 at 9:43 PM, Sam Elamin <[email protected]> > wrote: > > > Bit late notice but the call is happening today at 9 15 utc so in about > > 30 mins or so > > > > It will be recorded but if anyone would like to join in on the discussion > > the hangout link is https://hangouts.google.com/hangouts/_/ > > mbkr6xassnahjjonpuvrirxbnae > > > > Regards > > Sam > > > > On Fri, 5 May 2017 at 21:35, Ali Uz <[email protected]> wrote: > > > >> I am also very interested in seeing how this turns out. Even though we > >> don't have a testing framework in-place on the project I am working on, > I > >> would very much like to contribute to some general framework for testing > >> DAGs. > >> > >> As of now we are just implementing dummy tasks that test our actual > tasks > >> and verify if the given input produces the expected output. Nothing > crazy > >> and certainly not flexible in the long run. > >> > >> > >> On Fri, 5 May 2017 at 22:59, Sam Elamin <[email protected]> > wrote: > >> > >> > Haha yes Scott you are in! > >> > On Fri, 5 May 2017 at 20:07, Scott Halgrim <[email protected]> > >> > wrote: > >> > > >> > > Sounds A+ to me. By “both of you” did you include me? My first > >> response > >> > > was just to your email address. > >> > > > >> > > On May 5, 2017, 11:58 AM -0700, Sam Elamin <[email protected] > >, > >> > > wrote: > >> > > > Ok sounds great folks > >> > > > > >> > > > Thanks for the detailed response laura! I'll invite both of you to > >> the > >> > > > group if you are happy and we can schedule a call for next week? > >> > > > > >> > > > How does that sound? > >> > > > On Fri, 5 May 2017 at 17:41, Laura Lorenz < > [email protected] > >> > > >> > > wrote: > >> > > > > >> > > > > We do! We developed our own little in-house DAG test framework > >> which > >> > we > >> > > > > could share insights on/would love to hear what other folks are > up > >> > to. > >> > > > > Basically we use mock a DAG's input data, use the BackfillJob > API > >> > > directly > >> > > > > to call a DAG in a test, and compare its outputs to the intended > >> > result > >> > > > > given the inputs. We use docker/docker-compose to manage > services, > >> > and > >> > > > > split our dev and test stack locally so that the tests have > their > >> own > >> > > > > scheduler and metadata database and so that our CI tool knows > how > >> to > >> > > > > construct the test stack as well. > >> > > > > > >> > > > > We co-opted the BackfillJob API for our own purposes here, but > it > >> > > seemed > >> > > > > overly complicated and fragile to start and interact with our > own > >> > > > > in-test-process executor like we saw in a few of the tests in > the > >> > > Airflow > >> > > > > test suite. So I'd be really interested on finding a way to > >> > streamline > >> > > how > >> > > > > to describe a test executor for both the Airflow test suite and > >> > > people's > >> > > > > own DAG testing and make that a first class type of API. > >> > > > > > >> > > > > Laura > >> > > > > > >> > > > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin < > >> [email protected] > >> > > > > wrote: > >> > > > > > >> > > > > > Hi All > >> > > > > > > >> > > > > > A few people in the Spark community are interested in writing > a > >> > > testing > >> > > > > > library for Airflow. We would love anyone who uses Airflow > >> heavily > >> > in > >> > > > > > production to be involved > >> > > > > > > >> > > > > > At the moment (AFAIK) testing your DAGs is a bit of a pain, > >> > > especially if > >> > > > > > you want to run them in a CI server > >> > > > > > > >> > > > > > Is anyone interested in being involved in the discussion? > >> > > > > > > >> > > > > > Kind Regards > >> > > > > > Sam > >> > > > > > > >> > > > > > >> > > > >> > > >> > > >
