Re: unit testing for spark code

Hi! Let me draw your attention to Holden's* spark-testing-base* project. The documentation is at https://github.com/holdenk/spark-testing-base/wiki. As I usually write test for spark internal features I haven't needed to test so high level. But I am interested about your experiences. Best

Re: unit testing for spark code

I've found pytest works well if you're using PySpark. Though if you have a lot of tests, running them all can be pretty slow. On Mon, Mar 22, 2021 at 6:32 AM Amit Sharma wrote: > Hi, can we write unit tests for spark code. Is there any specific > framework? > > > Thanks > Amit >

Re: unit testing for spark code

coding in Scala or Python? Are you using any IDE (IntelliJ, PyCharm) view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property

unit testing for spark code

Hi, can we write unit tests for spark code. Is there any specific framework? Thanks Amit