I feel the real fix here is to remove the exception from QueueInputDStream class by reverting the fix of https://issues.apache.org/jira/browse/SPARK-8630
I can write another class that is identical to the QueueInputDStream class except it does not throw the exception. But this feels like a convoluted solution. Throwing exceptions to forbid behavior in code is risky because it can easily break legitimate uses of a class. Is there a way to reopen https://issues.apache.org/jira/browse/SPARK-8630. I have added a comment to it, but I am not sure if that will have that effect. Thanks. Asim On Fri, Aug 14, 2015 at 4:03 PM, Holden Karau <[email protected]> wrote: > I just pushed some code that does this for spark-testing-base ( > https://github.com/holdenk/spark-testing-base ) (its in master) and will > publish an updated artifact with it for tonight. > > On Fri, Aug 14, 2015 at 3:35 PM, Tathagata Das <[email protected]> > wrote: > >> A hacky workaround is to create a customer InputDStream that creates the >> right RDDs based on a function. The TestInputDStream >> <https://github.com/apache/spark/blob/master/streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala#L61> >> does something similar for Spark Streaming unit tests. >> >> TD >> >> On Fri, Aug 14, 2015 at 1:04 PM, Asim Jalis <[email protected]> wrote: >> >>> I want to test some Spark Streaming code that is using >>> reduceByKeyAndWindow. If I do not enable checkpointing, I get the error: >>> >>> java.lang.IllegalArgumentException: requirement failed: The checkpoint >>>> directory has not been set. Please set it by StreamingContext.checkpoint(). >>> >>> >>> But if I enable checkpointing I get >>> >>> queueStream doesn't support checkpointing >>> >>> >>> Is there a workaround for this? >>> >>> My goal is to test that the windowing logic in my code is correct. Is >>> there a way to disable these strict checks or a different dstream I can use >>> that I can populate programmatically and then use for testing? >>> >>> Thanks. >>> >>> Asim >>> >>> >> > > > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau > Linked In: https://www.linkedin.com/in/holdenkarau >
