You may be interested in https://github.com/OryxProject/oryx which is at heart exactly "lambda architecture on Spark Streaming". With ML pipelines on top. The architecture diagram and a peek at the code may give you a good example of how this could be implemented. I choose to view the batch layer as just a long-period streaming job on Spark Streaming, and implement the speed layer as a short-period streaming job. Summingbird is a good example too although it uses Storm and MapReduce, and is architected specifically for simple aggregations. I am not sure it "generalizes" but you may not need anything complex.
On Thu, Aug 14, 2014 at 10:27 PM, salemi <[email protected]> wrote: > Hi, > > How would you implement the batch layer of lamda architecture with > spark/spark streaming? > > Thanks, > Ali > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
