Maybe diverging a bit from the original question - but would it maybe make sense for those of us that all care about testing to try and do a hangout at some point so that we can exchange ideas?
On Thu, Dec 8, 2016 at 4:15 PM, Miguel Morales <therevolti...@gmail.com> wrote: > I would be interested in contributing. Ive created my own library for > this as well. In my blog post I talk about testing with Spark in RSpec > style: > https://medium.com/@therevoltingx/test-driven-development-w-apache-spark- > 746082b44941 > > Sent from my iPhone > > On Dec 8, 2016, at 4:09 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > > There are also libraries designed to simplify testing Spark in the various > platforms, spark-testing-base > <http://github.com/holdenk/spark-testing-base> for Scala/Java/Python (& > video https://www.youtube.com/watch?v=f69gSGSLGrY), sscheck > <https://github.com/juanrh/sscheck> (scala focused property based), > pyspark.test (python focused with py.test instead of unittest2) (& blog > post from nextdoor https://engblog.nextdoor.com/unit-testing- > apache-spark-with-py-test-3b8970dc013b#.jw3bdcej9 ) > > Good luck on your Spark Adventures :) > > P.S. > > If anyone is interested in helping improve spark testing libraries I'm > always looking for more people to be involved with spark-testing-base > because I'm lazy :p > > On Thu, Dec 8, 2016 at 2:05 PM, Lars Albertsson <la...@mapflat.com> wrote: > >> I wrote some advice in a previous post on the list: >> http://markmail.org/message/bbs5acrnksjxsrrs >> >> It does not mention python, but the strategy advice is the same. Just >> replace JUnit/Scalatest with pytest, unittest, or your favourite >> python test framework. >> >> >> I recently held a presentation on the subject. There is a video >> recording at https://vimeo.com/192429554 and slides at >> http://www.slideshare.net/lallea/test-strategies-for-data- >> processing-pipelines-67244458 >> >> You can find more material on test strategies at >> http://www.mapflat.com/lands/resources/reading-list/index.html >> >> >> >> >> Lars Albertsson >> Data engineering consultant >> www.mapflat.com >> https://twitter.com/lalleal >> +46 70 7687109 >> Calendar: https://goo.gl/6FBtlS, https://freebusy.io/la...@mapflat.com >> >> >> On Thu, Dec 8, 2016 at 4:14 PM, pseudo oduesp <pseudo20...@gmail.com> >> wrote: >> > somone can tell me how i can make unit test on pyspark ? >> > (book, tutorial ...) >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> > > > -- > Cell : 425-233-8271 <(425)%20233-8271> > Twitter: https://twitter.com/holdenkarau > > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau
