Re: Spark Testing Library Discussion

Oh interesting. I did send a PR or thought I did will check this eve. On Apr 29, 2017 10:04 AM, "Sam Elamin" wrote: > Hi lucas > > > Thanks for the detailed feedback, that's really useful! > > I did suggest Github but my colleague asked for an email > > You raise a good

Re: Spark Testing Library Discussion

Hi lucas Thanks for the detailed feedback, that's really useful! I did suggest Github but my colleague asked for an email You raise a good point with the grammar, sure I will rephrase it. I am more than happy to merge in the PR if you send it Th at said I know you can make BDD tests using

Re: Spark Testing Library Discussion

Awesome, thanks. Just reading your post A few observations: 1) You're giving out Marius's email: "I have been lucky enough to build this pipeline with the amazing Marius Feteanu". A linked or github link might be more helpful. 2) "If you are in Pyspark world sadly Holden’s test base wont work

Re: Spark Testing Library Discussion

Hi @Lucas I certainly would love to write an integration testing library for workflows, I have a few ideas I would love to share with others and they are focused around Airflow since that is what we use As promised here is

Re: Spark Testing Library Discussion

Sorry about that, hangouts on air broke in the first one :( On Wed, Apr 26, 2017 at 8:41 AM, Marco Mistroni wrote: > Uh i stayed online in the other link but nobody joinedWill follow > transcript > Kr > > On 26 Apr 2017 9:35 am, "Holden Karau"

Re: Spark Testing Library Discussion

Uh i stayed online in the other link but nobody joinedWill follow transcript Kr On 26 Apr 2017 9:35 am, "Holden Karau" wrote: > And the recording of our discussion is at https://www.youtube.com/ > watch?v=2q0uAldCQ8M > A few of us have follow up things and we will try

Re: Spark Testing Library Discussion

And the recording of our discussion is at https://www.youtube.com/watch?v=2q0uAldCQ8M A few of us have follow up things and we will try and do another meeting in about a month or two :) On Tue, Apr 25, 2017 at 1:04 PM, Holden Karau wrote: > Urgh hangouts did something

Re: Spark Testing Library Discussion

Hi all, whoever (Sam I think) was going to do some work on doing a template testing pipeline. I'd love to be involved, I have a current task in my day job (data engineer) to flesh out our testing how-to / best practices for Spark jobs and I think I'll be doing something very similar for the next

Re: Spark Testing Library Discussion

Urgh hangouts did something frustrating, updated link https://hangouts.google.com/hangouts/_/ha6kusycp5fvzei2trhay4uhhqe On Mon, Apr 24, 2017 at 12:13 AM, Holden Karau wrote: > The (tentative) link for those interested is https://hangouts.google. >

Re: Spark Testing Library Discussion

The (tentative) link for those interested is https://hangouts.google.com/hangouts/_/oyjvcnffejcjhi6qazf3lysypue . On Mon, Apr 24, 2017 at 12:02 AM, Holden Karau wrote: > So 14 people have said they are available on Tuesday the 25th at 1PM > pacific so we will do this

Re: Spark Testing Library Discussion

So 14 people have said they are available on Tuesday the 25th at 1PM pacific so we will do this meeting then ( https://doodle.com/poll/69y6yab4pyf7u8bn ). Since hangouts tends to work ok on the Linux distro I'm running my default is to host this as a "hangouts-on-air" unless there are alternative

Spark Testing Library Discussion

Hi Spark Users (+ Some Spark Testing Devs on BCC), Awhile back on one of the many threads about testing in Spark there was some interest in having a chat about the state of Spark testing and what people want/need. So if you are interested in joining an online (with maybe an IRL component if