Hi,
Maybe this helps you. For the speed layer I think something like complex
event processing as it is - to some extent - supported by Spark Streaming
can make sense. You process the events as they come in. You store them
afterwards. The Spark Streaming web page gives a nice example: trend
You may be interested in https://github.com/OryxProject/oryx which is
at heart exactly lambda architecture on Spark Streaming. With ML
pipelines on top. The architecture diagram and a peek at the code may
give you a good example of how this could be implemented. I choose to
view the batch layer as
Can you be a bit more specific about what you mean by lambda architecture?
On Thu, Aug 14, 2014 at 2:27 PM, salemi alireza.sal...@udo.edu wrote:
Hi,
How would you implement the batch layer of lamda architecture with
spark/spark streaming?
Thanks,
Ali
--
View this message in context:
below is what is what I understand under lambda architecture. The batch layer
provides the historical data and the speed layer provides the real-time
view!
All data entering the system is dispatched to both the batch layer and the
speed layer for processing.
The batch layer has two functions:
How would you implement the batch layer of lamda architecture with
spark/spark streaming?
I assume you’re familiar with resources such as
https://speakerdeck.com/mhausenblas/lambda-architecture-with-apache-spark and
are after more detailed advices?
Cheers,
Michael
--
help
you.
Thanks
Jerry
-Original Message-
From: salemi [mailto:alireza.sal...@udo.edu]
Sent: Friday, August 15, 2014 11:25 AM
To: u...@spark.incubator.apache.org
Subject: Re: spark streaming - lamda architecture
below is what is what I understand under lambda architecture. The batch
