Hi Holden Thanks Holden for pointing me the package. Indeed StreamingSuiteBase trait hides a lot, especially regarding clock manipulation. Did you encounter problems with concurrent tests execution from SBT (SPARK-2243)? I had to disable parallel execution and configure SBT to use separate JVM for tests execution (fork).
BTW. I added samples for SparkSQL as well. I would expect base trait for testing purposes in spark distribution. ManualClock should be exposed as well. And some documentation how to configure SBT to avoid problems with multiple spark contexts. I'm going to create improvement proposal on Spark issue tracker about it. On 1 March 2015 at 18:49, Holden Karau <[email protected]> wrote: > > There is also the Spark Testing Base package which is on spark-packages.org > and hides the ugly bits (it's based on the existing streaming test code but I > cleaned it up a bit to try and limit the number of internals it was touching). > > > On Sunday, March 1, 2015, Marcin Kuthan <[email protected]> wrote: >> >> I have started using Spark and Spark Streaming and I'm wondering how do you >> test your applications? Especially Spark Streaming application with window >> based transformations. >> >> After some digging I found ManualClock class to take full control over >> stream processing. Unfortunately the class is not available outside >> spark.streaming package. Are you going to expose the class for other >> developers as well? Now I have to use my custom wrapper under >> spark.streaming package. >> >> My Spark and Spark Streaming unit tests strategies are documented here: >> http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/ >> >> Your feedback is more than appreciated. >> >> Marcin >> > > > -- > Cell : 425-233-8271 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
