Re: QueueStream Does Not Support Checkpointing

Another fix might be to remove the exception that is thrown when windowing and other stateful operations are used without checkpointing. On Fri, Aug 14, 2015 at 5:43 PM, Asim Jalis wrote: > I feel the real fix here is to remove the exception from QueueInputDStream > class by reverting the fix of

Re: QueueStream Does Not Support Checkpointing

I feel the real fix here is to remove the exception from QueueInputDStream class by reverting the fix of https://issues.apache.org/jira/browse/SPARK-8630 I can write another class that is identical to the QueueInputDStream class except it does not throw the exception. But this feels like a convolu

Re: QueueStream Does Not Support Checkpointing

I just pushed some code that does this for spark-testing-base ( https://github.com/holdenk/spark-testing-base ) (its in master) and will publish an updated artifact with it for tonight. On Fri, Aug 14, 2015 at 3:35 PM, Tathagata Das wrote: > A hacky workaround is to create a customer InputDStre

Re: QueueStream Does Not Support Checkpointing

A hacky workaround is to create a customer InputDStream that creates the right RDDs based on a function. The TestInputDStream does something similar for Spark Streaming unit tes

QueueStream Does Not Support Checkpointing

I want to test some Spark Streaming code that is using reduceByKeyAndWindow. If I do not enable checkpointing, I get the error: java.lang.IllegalArgumentException: requirement failed: The checkpoint > directory has not been set. Please set it by StreamingContext.checkpoint(). But if I enable che