Spark with Spark Streaming

2014-06-07 Thread b0c1 
Hi!

There are any way to use spark with spark streaming together to create real
time architecture?
How can I merge the spark and spark streaming result at realtime (and drop
streaming result if spark result generated)?

Thanks



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-with-Spark-Streaming-tp7164.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

unit test

2014-06-06 Thread b0c1 
Hi!

I have two question:
1.
I want to test my application. My app will write the result to elasticsearch
(stage 1) with saveAsHadoopFile. How can I write test case for it? Need to
pull up a MiniDFSCluster? Or there are other way?

My application flow plan: 
Kafka => Spark Streaming (enrich) -> Elasticsearch => Spark (map/reduce) ->
HBase

2.
Can Spark read data from elasticsearch? What is the prefered way for this?

b0c1



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/unit-test-tp7155.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.