There are also libraries designed to simplify testing Spark in the various platforms, spark-testing-base <http://github.com/holdenk/spark-testing-base> for Scala/Java/Python (& video https://www.youtube.com/watch?v=f69gSGSLGrY), sscheck <https://github.com/juanrh/sscheck> (scala focused property based), pyspark.test (python focused with py.test instead of unittest2) (& blog post from nextdoor https://engblog.nextdoor.com/unit-testing-apache-spark-with-py-test-3b8970dc013b#.jw3bdcej9 )
Good luck on your Spark Adventures :) P.S. If anyone is interested in helping improve spark testing libraries I'm always looking for more people to be involved with spark-testing-base because I'm lazy :p On Thu, Dec 8, 2016 at 2:05 PM, Lars Albertsson <la...@mapflat.com> wrote: > I wrote some advice in a previous post on the list: > http://markmail.org/message/bbs5acrnksjxsrrs > > It does not mention python, but the strategy advice is the same. Just > replace JUnit/Scalatest with pytest, unittest, or your favourite > python test framework. > > > I recently held a presentation on the subject. There is a video > recording at https://vimeo.com/192429554 and slides at > http://www.slideshare.net/lallea/test-strategies-for- > data-processing-pipelines-67244458 > > You can find more material on test strategies at > http://www.mapflat.com/lands/resources/reading-list/index.html > > > > > Lars Albertsson > Data engineering consultant > www.mapflat.com > https://twitter.com/lalleal > +46 70 7687109 > Calendar: https://goo.gl/6FBtlS, https://freebusy.io/la...@mapflat.com > > > On Thu, Dec 8, 2016 at 4:14 PM, pseudo oduesp <pseudo20...@gmail.com> > wrote: > > somone can tell me how i can make unit test on pyspark ? > > (book, tutorial ...) > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau
