I just wrote a blog post on Unit testing Apache Spark with py.test https://engblog.nextdoor.com/unit-testing-apache-spark-with-py-test-3b8970dc013b
If you prefer using the py.test framework, then it might be useful. -vikas On Wed, Mar 2, 2016 at 10:59 AM, radoburansky <radoburan...@gmail.com> wrote: > I am sure you have googled this: > https://github.com/holdenk/spark-testing-base > > On Wed, Mar 2, 2016 at 6:54 PM, SRK [via Apache Spark User List] <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=26384&i=0>> wrote: > >> Hi, >> >> What is a good unit testing framework for Spark batch/streaming jobs? I >> have core spark, spark sql with dataframes and streaming api getting used. >> Any good framework to cover unit tests for these APIs? >> >> Thanks! >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-spark-user-list.1001560.n3.nabble.com/Unit-testing-framework-for-Spark-Jobs-tp26380.html >> To start a new topic under Apache Spark User List, email [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=26384&i=1> >> To unsubscribe from Apache Spark User List, click here. >> NAML >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > ------------------------------ > View this message in context: Re: Unit testing framework for Spark Jobs? > <http://apache-spark-user-list.1001560.n3.nabble.com/Unit-testing-framework-for-Spark-Jobs-tp26380p26384.html> > Sent from the Apache Spark User List mailing list archive > <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >
