A hacky workaround is to create a customer InputDStream that creates the right RDDs based on a function. The TestInputDStream <https://github.com/apache/spark/blob/master/streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala#L61> does something similar for Spark Streaming unit tests.
TD On Fri, Aug 14, 2015 at 1:04 PM, Asim Jalis <[email protected]> wrote: > I want to test some Spark Streaming code that is using > reduceByKeyAndWindow. If I do not enable checkpointing, I get the error: > > java.lang.IllegalArgumentException: requirement failed: The checkpoint >> directory has not been set. Please set it by StreamingContext.checkpoint(). > > > But if I enable checkpointing I get > > queueStream doesn't support checkpointing > > > Is there a workaround for this? > > My goal is to test that the windowing logic in my code is correct. Is > there a way to disable these strict checks or a different dstream I can use > that I can populate programmatically and then use for testing? > > Thanks. > > Asim > >
