I just pushed some code that does this for spark-testing-base ( https://github.com/holdenk/spark-testing-base ) (its in master) and will publish an updated artifact with it for tonight.
On Fri, Aug 14, 2015 at 3:35 PM, Tathagata Das <[email protected]> wrote: > A hacky workaround is to create a customer InputDStream that creates the > right RDDs based on a function. The TestInputDStream > <https://github.com/apache/spark/blob/master/streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala#L61> > does something similar for Spark Streaming unit tests. > > TD > > On Fri, Aug 14, 2015 at 1:04 PM, Asim Jalis <[email protected]> wrote: > >> I want to test some Spark Streaming code that is using >> reduceByKeyAndWindow. If I do not enable checkpointing, I get the error: >> >> java.lang.IllegalArgumentException: requirement failed: The checkpoint >>> directory has not been set. Please set it by StreamingContext.checkpoint(). >> >> >> But if I enable checkpointing I get >> >> queueStream doesn't support checkpointing >> >> >> Is there a workaround for this? >> >> My goal is to test that the windowing logic in my code is correct. Is >> there a way to disable these strict checks or a different dstream I can use >> that I can populate programmatically and then use for testing? >> >> Thanks. >> >> Asim >> >> > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau
