Hi, Maybe this helps you. For the "speed" layer I think something like complex event processing as it is - to some extent - supported by Spark Streaming can make sense. You process the events as they come in. You store them afterwards. The Spark Streaming web page gives a nice example: trend analysis. You detect from a lot of incoming events the current trend with Spark Streaming and compare them to historical trends (stored in Spark, which I would compare here with a serving layer). You can use this information to see if customers are more happy (or more angry) with your new products than usual.
I hope it helps. Your use case sounds very technical, it would be interesting if you could share the business use case. Best regards, Jörn On Fri, Aug 15, 2014 at 5:25 AM, salemi <alireza.sal...@udo.edu> wrote: > below is what is what I understand under lambda architecture. The batch > layer > provides the historical data and the speed layer provides the real-time > view! > > All data entering the system is dispatched to both the batch layer and the > speed layer for processing. > The batch layer has two functions: > (i) managing the master dataset (an immutable, append-only set of raw > data), > and > (ii) to pre-compute the batch views. > > The speed layer compensates for the high latency of updates to the serving > layer and deals with recent data only. > > The serving layer indexes the batch views so that they can be queried in > low-latency, ad-hoc way. > > Any incoming query can be answered by merging results from batch views and > real-time views. > > In my system I have events coming in from Kafka sources and currently we > need to process 10,000 messages per second and write them out to hdfs and > make them available to be queried by a serving layer. > > What would be your suggestion to architecturally solve this issue? How many > solution with which would approx. be needed for the proposed architecture. > > Thanks, > Ali > > > Tathagata Das wrote > > Can you be a bit more specific about what you mean by lambda > architecture? > > > > > > On Thu, Aug 14, 2014 at 2:27 PM, salemi < > > > alireza.salemi@ > > > > wrote: > > > >> Hi, > >> > >> How would you implement the batch layer of lamda architecture with > >> spark/spark streaming? > >> > >> Thanks, > >> Ali > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html > >> Sent from the Apache Spark User List mailing list archive at Nabble.com. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: > > > user-unsubscribe@.apache > > >> For additional commands, e-mail: > > > user-help@.apache > > >> > >> > > > Tathagata Das wrote > > Can you be a bit more specific about what you mean by lambda > architecture? > > > > > > On Thu, Aug 14, 2014 at 2:27 PM, salemi < > > > alireza.salemi@ > > > > wrote: > > > >> Hi, > >> > >> How would you implement the batch layer of lamda architecture with > >> spark/spark streaming? > >> > >> Thanks, > >> Ali > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html > >> Sent from the Apache Spark User List mailing list archive at Nabble.com. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: > > > user-unsubscribe@.apache > > >> For additional commands, e-mail: > > > user-help@.apache > > >> > >> > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142p12163.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
