Hi Folks For those of you who missed it, you can catch the discussion from the link on this tweet <https://twitter.com/samelamin/status/861703888298225670>
Please do share and feel free to get involved as the more feedback we get the better the library we create is :) Regards Sam On Mon, May 8, 2017 at 9:43 PM, Sam Elamin <[email protected]> wrote: > Bit late notice but the call is happening today at 9 15 utc so in about > 30 mins or so > > It will be recorded but if anyone would like to join in on the discussion > the hangout link is https://hangouts.google.com/hangouts/_/ > mbkr6xassnahjjonpuvrirxbnae > > Regards > Sam > > On Fri, 5 May 2017 at 21:35, Ali Uz <[email protected]> wrote: > >> I am also very interested in seeing how this turns out. Even though we >> don't have a testing framework in-place on the project I am working on, I >> would very much like to contribute to some general framework for testing >> DAGs. >> >> As of now we are just implementing dummy tasks that test our actual tasks >> and verify if the given input produces the expected output. Nothing crazy >> and certainly not flexible in the long run. >> >> >> On Fri, 5 May 2017 at 22:59, Sam Elamin <[email protected]> wrote: >> >> > Haha yes Scott you are in! >> > On Fri, 5 May 2017 at 20:07, Scott Halgrim <[email protected]> >> > wrote: >> > >> > > Sounds A+ to me. By “both of you” did you include me? My first >> response >> > > was just to your email address. >> > > >> > > On May 5, 2017, 11:58 AM -0700, Sam Elamin <[email protected]>, >> > > wrote: >> > > > Ok sounds great folks >> > > > >> > > > Thanks for the detailed response laura! I'll invite both of you to >> the >> > > > group if you are happy and we can schedule a call for next week? >> > > > >> > > > How does that sound? >> > > > On Fri, 5 May 2017 at 17:41, Laura Lorenz <[email protected] >> > >> > > wrote: >> > > > >> > > > > We do! We developed our own little in-house DAG test framework >> which >> > we >> > > > > could share insights on/would love to hear what other folks are up >> > to. >> > > > > Basically we use mock a DAG's input data, use the BackfillJob API >> > > directly >> > > > > to call a DAG in a test, and compare its outputs to the intended >> > result >> > > > > given the inputs. We use docker/docker-compose to manage services, >> > and >> > > > > split our dev and test stack locally so that the tests have their >> own >> > > > > scheduler and metadata database and so that our CI tool knows how >> to >> > > > > construct the test stack as well. >> > > > > >> > > > > We co-opted the BackfillJob API for our own purposes here, but it >> > > seemed >> > > > > overly complicated and fragile to start and interact with our own >> > > > > in-test-process executor like we saw in a few of the tests in the >> > > Airflow >> > > > > test suite. So I'd be really interested on finding a way to >> > streamline >> > > how >> > > > > to describe a test executor for both the Airflow test suite and >> > > people's >> > > > > own DAG testing and make that a first class type of API. >> > > > > >> > > > > Laura >> > > > > >> > > > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin < >> [email protected] >> > > > > wrote: >> > > > > >> > > > > > Hi All >> > > > > > >> > > > > > A few people in the Spark community are interested in writing a >> > > testing >> > > > > > library for Airflow. We would love anyone who uses Airflow >> heavily >> > in >> > > > > > production to be involved >> > > > > > >> > > > > > At the moment (AFAIK) testing your DAGs is a bit of a pain, >> > > especially if >> > > > > > you want to run them in a CI server >> > > > > > >> > > > > > Is anyone interested in being involved in the discussion? >> > > > > > >> > > > > > Kind Regards >> > > > > > Sam >> > > > > > >> > > > > >> > > >> > >> >
