Hi @Lucas I certainly would love to write an integration testing library for workflows, I have a few ideas I would love to share with others and they are focused around Airflow since that is what we use
As promised here <https://samelamin.github.io/2017/04/27/Building-A-Datapipeline-part1/> is the first blog post in a series of posts I hope to write on how we build data pipelines Please feel free to retweet my original tweet <https://twitter.com/samelamin/status/857546231492612096> and share because the more ideas we have the better! Feedback is always welcome! Regards Sam On Tue, Apr 25, 2017 at 10:32 PM, lucas.g...@gmail.com <lucas.g...@gmail.com > wrote: > Hi all, whoever (Sam I think) was going to do some work on doing a > template testing pipeline. I'd love to be involved, I have a current task > in my day job (data engineer) to flesh out our testing how-to / best > practices for Spark jobs and I think I'll be doing something very similar > for the next week or 2. > > I'll scrape out what i have now in the next day or so and put it up in a > gist that I can share too. > > G > > On 25 April 2017 at 13:04, Holden Karau <hol...@pigscanfly.ca> wrote: > >> Urgh hangouts did something frustrating, updated link >> https://hangouts.google.com/hangouts/_/ha6kusycp5fvzei2trhay4uhhqe >> >> On Mon, Apr 24, 2017 at 12:13 AM, Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> The (tentative) link for those interested is https://hangouts.google.com >>> /hangouts/_/oyjvcnffejcjhi6qazf3lysypue . >>> >>> On Mon, Apr 24, 2017 at 12:02 AM, Holden Karau <hol...@pigscanfly.ca> >>> wrote: >>> >>>> So 14 people have said they are available on Tuesday the 25th at 1PM >>>> pacific so we will do this meeting then ( https://doodle.com/poll/69y6 >>>> yab4pyf7u8bn ). >>>> >>>> Since hangouts tends to work ok on the Linux distro I'm running my >>>> default is to host this as a "hangouts-on-air" unless there are alternative >>>> ideas. >>>> >>>> I'll record the hangout and if it isn't terrible I'll post it for those >>>> who weren't able to make it (and for next time I'll include more European >>>> friendly time options - Doodle wouldn't let me update it once posted). >>>> >>>> On Fri, Apr 14, 2017 at 11:17 AM, Holden Karau <hol...@pigscanfly.ca> >>>> wrote: >>>> >>>>> Hi Spark Users (+ Some Spark Testing Devs on BCC), >>>>> >>>>> Awhile back on one of the many threads about testing in Spark there >>>>> was some interest in having a chat about the state of Spark testing and >>>>> what people want/need. >>>>> >>>>> So if you are interested in joining an online (with maybe an IRL >>>>> component if enough people are SF based) chat about Spark testing please >>>>> fill out this doodle - https://doodle.com/poll/69y6yab4pyf7u8bn >>>>> >>>>> I think reasonable topics of discussion could be: >>>>> >>>>> 1) What is the state of the different Spark testing libraries in the >>>>> different core (Scala, Python, R, Java) and extended languages (C#, >>>>> Javascript, etc.)? >>>>> 2) How do we make these more easily discovered by users? >>>>> 3) What are people looking for in their testing libraries that we are >>>>> missing? (can be functionality, documentation, etc.) >>>>> 4) Are there any examples of well tested open source Spark projects >>>>> and where are they? >>>>> >>>>> If you have other topics that's awesome. >>>>> >>>>> To clarify this about libraries and best practices for people testing >>>>> their Spark applications, and less about testing Spark's internals >>>>> (although as illustrated by some of the libraries there is some strong >>>>> overlap in what is required to make that work). >>>>> >>>>> Cheers, >>>>> >>>>> Holden :) >>>>> >>>>> -- >>>>> Cell : 425-233-8271 <(425)%20233-8271> >>>>> Twitter: https://twitter.com/holdenkarau >>>>> >>>> >>>> >>>> >>>> -- >>>> Cell : 425-233-8271 <(425)%20233-8271> >>>> Twitter: https://twitter.com/holdenkarau >>>> >>> >>> >>> >>> -- >>> Cell : 425-233-8271 <(425)%20233-8271> >>> Twitter: https://twitter.com/holdenkarau >>> >> >> >> >> -- >> Cell : 425-233-8271 <(425)%20233-8271> >> Twitter: https://twitter.com/holdenkarau >> > >
