Hi all,I wonder if there is a way to create some child streaming while using
spark streaming?For example, I create a netcat main stream, read data from a
socket, then create 3 different child streams on the main stream,in stream1, we
do fun1 on the input data then print result to screen;in
You can create as many functional derivates of your original stream by
using transformations. That's exactly the model that Spark Streaming offers.
In your example, that would become something like:
val stream = ssc.socketTextStream("localhost", )
val stream1 = stream.map(fun1)
val stream2 =
if you want to use true Spark Streaming (not the same as Hadoop
Streaming/Piping, as Mayur pointed out), you can use the DStream.union()
method as described in the following docs:
http://spark.apache.org/docs/0.9.1/streaming-custom-receivers.html
File as a stream?
I think you are confusing Spark Streaming with buffer reader. Spark
streaming is meant to process batches of data (files, packets, messages) as
they come in, infact utilizing time of packet reception as a way to create
windows etc.
In your case you are better off reading the
