>>> How would you implement the batch layer of lamda architecture with >>> spark/spark streaming?
I assume you’re familiar with resources such as https://speakerdeck.com/mhausenblas/lambda-architecture-with-apache-spark and are after more detailed advices? Cheers, Michael -- Michael Hausenblas Ireland, Europe http://mhausenblas.info/ On 15 Aug 2014, at 05:25, salemi <[email protected]> wrote: > below is what is what I understand under lambda architecture. The batch layer > provides the historical data and the speed layer provides the real-time > view! > > All data entering the system is dispatched to both the batch layer and the > speed layer for processing. > The batch layer has two functions: > (i) managing the master dataset (an immutable, append-only set of raw data), > and > (ii) to pre-compute the batch views. > > The speed layer compensates for the high latency of updates to the serving > layer and deals with recent data only. > > The serving layer indexes the batch views so that they can be queried in > low-latency, ad-hoc way. > > Any incoming query can be answered by merging results from batch views and > real-time views. > > In my system I have events coming in from Kafka sources and currently we > need to process 10,000 messages per second and write them out to hdfs and > make them available to be queried by a serving layer. > > What would be your suggestion to architecturally solve this issue? How many > solution with which would approx. be needed for the proposed architecture. > > Thanks, > Ali > > > Tathagata Das wrote >> Can you be a bit more specific about what you mean by lambda architecture? >> >> >> On Thu, Aug 14, 2014 at 2:27 PM, salemi < > >> alireza.salemi@ > >> > wrote: >> >>> Hi, >>> >>> How would you implement the batch layer of lamda architecture with >>> spark/spark streaming? >>> >>> Thanks, >>> Ali >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: > >> [email protected] > >>> For additional commands, e-mail: > >> [email protected] > >>> >>> > > > Tathagata Das wrote >> Can you be a bit more specific about what you mean by lambda architecture? >> >> >> On Thu, Aug 14, 2014 at 2:27 PM, salemi < > >> alireza.salemi@ > >> > wrote: >> >>> Hi, >>> >>> How would you implement the batch layer of lamda architecture with >>> spark/spark streaming? >>> >>> Thanks, >>> Ali >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: > >> [email protected] > >>> For additional commands, e-mail: > >> [email protected] > >>> >>> > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142p12163.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
