Re: Spark Streaming: how to StreamingContext.queueStream

[Akhil Das]

You can do something like this:


val rddQueue = scala.collection.mutable.Queue(rdd1,rdd2,rdd3)

val qDstream = ssc.queueStream(rddQueue)




Thanks
Best Regards

On Sat, Oct 24, 2015 at 4:43 AM, Anfernee Xu  wrote:

> Hi,
>
> Here's my situation, I have some kind of offline dataset, but I want to
> form a virtual data stream feeding to Spark Streaming, my code looks like
> this
>
>
>// sort offline data by time
>  1)  JavaRDD sortedByTime = offlineDataRDD.sortBy( );
>
>// compute a list of JavaRDD,  each element JavaRDD is hosting the data
> in the same time
>// bucket.
>   2) List virtualStreamRdd = ?
>
> Queue queue = Queues.newLinkedBlockingQueue();
> queue.addAll(virtualStreamRdd);
>
> /*
>  * Create DStream from the queue
>  */
>
> 3) final JavaDStream rowDStream =
> streamingContext.queueStream(queue);
>
>
> Currently I'm stucking in 2), any suggestion is appreciated.
>
> Thanks
>
> --
> --Anfernee
>

Spark Streaming: how to StreamingContext.queueStream

[Anfernee Xu]

Hi,

Here's my situation, I have some kind of offline dataset, but I want to
form a virtual data stream feeding to Spark Streaming, my code looks like
this


   // sort offline data by time
 1)  JavaRDD sortedByTime = offlineDataRDD.sortBy( );

   // compute a list of JavaRDD,  each element JavaRDD is hosting the data
in the same time
   // bucket.
  2) List virtualStreamRdd = ?

Queue queue = Queues.newLinkedBlockingQueue();
queue.addAll(virtualStreamRdd);

/*
 * Create DStream from the queue
 */

3) final JavaDStream rowDStream =
streamingContext.queueStream(queue);


Currently I'm stucking in 2), any suggestion is appreciated.

Thanks

-- 
--Anfernee