Re: unit testing for spark code

Hi! Let me draw your attention to Holden's* spark-testing-base* project. The documentation is at https://github.com/holdenk/spark-testing-base/wiki. As I usually write test for spark internal features I haven't needed to test so high level. But I am interested about your experiences. Best

Re: unit testing for spark code

I've found pytest works well if you're using PySpark. Though if you have a lot of tests, running them all can be pretty slow. On Mon, Mar 22, 2021 at 6:32 AM Amit Sharma wrote: > Hi, can we write unit tests for spark code. Is there any specific > framework? > > > Thanks > Amit >

Re: unit testing for spark code

coding in Scala or Python? Are you using any IDE (IntelliJ, PyCharm) view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property

unit testing for spark code

Hi, can we write unit tests for spark code. Is there any specific framework? Thanks Amit

Re: unit testing in spark

41 >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On Dec 8, 2016, at 4:09 PM, Holden Karau <hol...@pigscanfly.ca> >>>>>> wrote: >>>>>> >>>>>> There are also librari

Re: unit testing in spark

(sorry sent an empty reply by accident) Unit testing is one of the easiest ways to isolate problems in an an internal class, things you can get wrong. But: time spent writing unit tests is time *not* spent writing integration tests. Which biases me towards the integration. What I do find is

Re: unit testing in spark

ingx/test-driven-development-w-apache-spark-746082b44941 >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On Dec 8, 2016, at 4:09 PM, Holden Karau <hol...@pigscanfly.ca> wrote: >>>>>>> >>&

Re: unit testing in spark

;> >>>>> There are also libraries designed to simplify testing Spark in the >>>>> various platforms, spark-testing-base >>>>> <http://github.com/holdenk/spark-testing-base> for Scala/Java/Python >>>>> (& video https://www.

Re: unit testing in spark

in the >>>> various platforms, spark-testing-base >>>> <http://github.com/holdenk/spark-testing-base> for Scala/Java/Python >>>> (& video https://www.youtube.com/watch?v=f69gSGSLGrY), sscheck >>>> <https://github.com/juanrh/sscheck>

Re: unit testing in spark

Hi all, I would also would like to participate on that. Greetings, Juan On Fri, Dec 9, 2016 at 6:03 AM, Michael Stratton < michael.strat...@komodohealth.com> wrote: > That sounds great, please include me so I can get involved. > > On Fri, Dec 9, 2016 at 7:39 AM, Marco Mistroni

Re: unit testing in spark

latforms, spark-testing-base >> <http://github.com/holdenk/spark-testing-base> for Scala/Java/Python (& >> video https://www.youtube.com/watch?v=f69gSGSLGrY), sscheck >> <https://github.com/juanrh/sscheck> (scala focused property based), >> pyspark.test (python focused wi

Re: unit testing in spark

(scala focused property based), > pyspark.test (python focused with py.test instead of unittest2) (& blog > post from nextdoor https://engblog.nextdoor.com/unit-testing- > apache-spark-with-py-test-3b8970dc013b#.jw3bdcej9 ) > > Good luck on your Spark Adventures :) > > P.S.

Re: unit testing in spark

atch?v=f69gSGSLGrY), sscheck (scala focused >>> property based), pyspark.test (python focused with py.test instead of >>> unittest2) (& blog post from nextdoor >>> https://engblog.nextdoor.com/unit-testing-apache-spark-with-py-test-3b8970dc013b#.jw3bdcej9 >>>

Re: unit testing in spark

e.com/watch?v=f69gSGSLGrY), sscheck > <https://github.com/juanrh/sscheck> (scala focused property based), > pyspark.test (python focused with py.test instead of unittest2) (& blog > post from nextdoor https://engblog.nextdoor.com/unit-testing- > apache-spark-with-py-test-3b89

Re: unit testing in spark

pyspark.test (python focused with py.test instead of unittest2) (& > blog post from nextdoor > https://engblog.nextdoor.com/unit-testing-apache-spark-with-py-test-3b8970dc013b#.jw3bdcej9 > ) > > Good luck on your Spark Adventures :) > > P.S. > > If anyone is inter

Re: unit testing in spark

focused property based), pyspark.test (python focused with py.test instead of unittest2) (& blog post from nextdoor https://engblog.nextdoor.com/unit-testing-apache-spark-with-py-test-3b8970dc013b#.jw3bdcej9 ) Good luck on your Spark Adventures :) P.S. If anyone is interested in helping i

Re: unit testing in spark

I wrote some advice in a previous post on the list: http://markmail.org/message/bbs5acrnksjxsrrs It does not mention python, but the strategy advice is the same. Just replace JUnit/Scalatest with pytest, unittest, or your favourite python test framework. I recently held a presentation on the

Re: unit testing in spark

Hi Pseudo, Just use unittest https://docs.python.org/2/library/unittest.html . > On 8 Dec 2016, at 19:14, pseudo oduesp wrote: > > somone can tell me how i can make unit test on pyspark ? > (book, tutorial ...)

unit testing in spark

somone can tell me how i can make unit test on pyspark ? (book, tutorial ...)

Re: Scala: Perform Unit Testing in spark

I placed the *tests* jars in the *lib* folder, Now its working. On Wed, Apr 6, 2016 at 7:34 PM, Lars Albertsson wrote: > Hi, > > I wrote a longish mail on Spark testing strategy last month, which you > may find useful: >

Re: Scala: Perform Unit Testing in spark

Hi, I wrote a longish mail on Spark testing strategy last month, which you may find useful: http://mail-archives.apache.org/mod_mbox/spark-user/201603.mbox/browser Let me know if you have follow up questions or want assistance. Regards, Lars Albertsson Data engineering consultant

Re: Scala: Perform Unit Testing in spark

I think you should specify dependencies in this way: *"org.apache.spark" % "spark-core_2.10" % "1.6.0"* % "tests" Please refer to http://www.scalatest.org/user_guide/using_scalatest_with_sbt On Fri, Apr 1, 2016 at 3:33 PM, Shishir Anshuman wrote: > When I added

Re: Scala: Perform Unit Testing in spark

When I added *"org.apache.spark" % "spark-core_2.10" % "1.6.0", *it should include spark-core_2.10-1.6.1-tests.jar. Why do I need to use the jar file explicitly? And how do I use the jars for compiling with *sbt* and running the tests on spark? On Sat, Apr 2, 2016 at 3:46 AM, Ted Yu

Re: Scala: Perform Unit Testing in spark

You need to include the following jars: jar tvf ./core/target/spark-core_2.10-1.6.1-tests.jar | grep SparkFunSuite 1787 Thu Mar 03 09:06:14 PST 2016 org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class 1780 Thu Mar 03 09:06:14 PST 2016

Re: Scala: Perform Unit Testing in spark

You can also look at spark-testing-base which works in both Scalatest and Junit and see if that works for your use case. On Friday, April 1, 2016, Ted Yu wrote: > Assuming your code is written in Scala, I would suggest using ScalaTest. > > Please take a look at the

Re: Scala: Perform Unit Testing in spark

Assuming your code is written in Scala, I would suggest using ScalaTest. Please take a look at the XXSuite.scala files under mllib/ On Fri, Apr 1, 2016 at 1:31 PM, Shishir Anshuman wrote: > Hello, > > I have a code written in scala using Mllib. I want to perform unit

Scala: Perform Unit Testing in spark

Hello, I have a code written in scala using Mllib. I want to perform unit testing it. I cant decide between Junit 4 and ScalaTest. I am new to Spark. Please guide me how to proceed with the testing. Thank you.

Unit testing and Spark Streaming

Hi, I’ve started my first experiments with Spark Streaming and started with setting up an environment using ScalaTest to do unit testing. Poked around on this mailing list and googled the topic. One of the things I wanted to be able to do is to use Scala Sequences as data source in the tests

Re: Unit testing and Spark Streaming

On Fri, Dec 12, 2014 at 2:17 PM, Eric Loots eric.lo...@gmail.com wrote: How can the log level in test mode be reduced (or extended when needed) ? Hello Eric, The following might be helpful for reducing the log messages during unit testing: http://stackoverflow.com/a/2736/236007 -- Emre

Re: Unit testing and Spark Streaming

https://github.com/jayunit100/SparkStreamingCassandraDemo On this note, I've built a framework which is mostly pure so that functional unit tests can be run composing mock data for Twitter statuses, with just regular junit... That might be relevant also. I think at some point we should come

Re: guidance on simple unit testing with Spark

the input while calling my program from the unit test and get the output. Thanks for your help. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/guidance-on-simple-unit-testing-with-Spark-tp7604.html Sent from the Apache Spark User List mailing list

Re: guidance on simple unit testing with Spark

for your help. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/guidance-on-simple-unit-testing-with-Spark-tp7604.html Sent from the Apache Spark User List mailing list archive at Nabble.com.