Hi Ali, Maybe you can take a look at twitter's Summingbird project (https://github.com/twitter/summingbird), which is currently one of the few open source choices of lambda Architecture. There's a undergoing sub-project called summingbird-spark, that might be the one you wanted, might this can help you.
Thanks Jerry -----Original Message----- From: salemi [mailto:alireza.sal...@udo.edu] Sent: Friday, August 15, 2014 11:25 AM To: u...@spark.incubator.apache.org Subject: Re: spark streaming - lamda architecture below is what is what I understand under lambda architecture. The batch layer provides the historical data and the speed layer provides the real-time view! All data entering the system is dispatched to both the batch layer and the speed layer for processing. The batch layer has two functions: (i) managing the master dataset (an immutable, append-only set of raw data), and (ii) to pre-compute the batch views. The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only. The serving layer indexes the batch views so that they can be queried in low-latency, ad-hoc way. Any incoming query can be answered by merging results from batch views and real-time views. In my system I have events coming in from Kafka sources and currently we need to process 10,000 messages per second and write them out to hdfs and make them available to be queried by a serving layer. What would be your suggestion to architecturally solve this issue? How many solution with which would approx. be needed for the proposed architecture. Thanks, Ali Tathagata Das wrote > Can you be a bit more specific about what you mean by lambda architecture? > > > On Thu, Aug 14, 2014 at 2:27 PM, salemi < > alireza.salemi@ > > wrote: > >> Hi, >> >> How would you implement the batch layer of lamda architecture with >> spark/spark streaming? >> >> Thanks, >> Ali >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-l >> amda-architecture-tp12142.html Sent from the Apache Spark User List >> mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: > user-unsubscribe@.apache >> For additional commands, e-mail: > user-help@.apache >> >> Tathagata Das wrote > Can you be a bit more specific about what you mean by lambda architecture? > > > On Thu, Aug 14, 2014 at 2:27 PM, salemi < > alireza.salemi@ > > wrote: > >> Hi, >> >> How would you implement the batch layer of lamda architecture with >> spark/spark streaming? >> >> Thanks, >> Ali >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-l >> amda-architecture-tp12142.html Sent from the Apache Spark User List >> mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: > user-unsubscribe@.apache >> For additional commands, e-mail: > user-help@.apache >> >> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142p12163.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
