You may be interested in https://github.com/OryxProject/oryx which is at heart exactly "lambda architecture on Spark Streaming". With ML pipelines on top. The architecture diagram and a peek at the code may give you a good example of how this could be implemented. I choose to view the batch layer as just a long-period streaming job on Spark Streaming, and implement the speed layer as a short-period streaming job. Summingbird is a good example too although it uses Storm and MapReduce, and is architected specifically for simple aggregations. I am not sure it "generalizes" but you may not need anything complex.
On Thu, Aug 14, 2014 at 10:27 PM, salemi <alireza.sal...@udo.edu> wrote: > Hi, > > How would you implement the batch layer of lamda architecture with > spark/spark streaming? > > Thanks, > Ali > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
