Another subject that was not that important in spark, but it could be crucial for 24/7 spark streaming, is reconstruction of lost nodes. By that, I do not mean lost data reconstruction by self healing, but bringing up new ec2 instances once they die for whatever reasons. Is this also supported in spark ec2?
On Fri, Feb 28, 2014 at 2:24 AM, Tathagata Das <tathagata.das1...@gmail.com>wrote: > Yes, the default spark EC2 cluster runs the standalone deploy mode. Since > Spark 0.9, the standalone deploy mode allows you to launch the driver app > within the cluster itself and automatically restart it if it fails. You can > read about launching your app inside the cluster > here<http://spark.incubator.apache.org/docs/latest/spark-standalone.html#connecting-an-application-to-the-cluster>. > Using this you can launch your streaming app as well. > > TD > > > On Thu, Feb 27, 2014 at 5:35 PM, Aureliano Buendia > <buendia...@gmail.com>wrote: > >> How about spark stream app itself? Does the ec2 script also provide means >> for daemonizing and monitoring spark streaming apps which are supposed to >> run 24/7? If not, any suggestions for how to do this? >> >> >> On Thu, Feb 27, 2014 at 8:23 PM, Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >>> Zookeeper is automatically set up in the cluster as Spark uses >>> Zookeeper. However, you have to setup your own input source like Kafka or >>> Flume. >>> >>> TD >>> >>> >>> On Thu, Feb 27, 2014 at 10:32 AM, Aureliano Buendia < >>> buendia...@gmail.com> wrote: >>> >>>> >>>> >>>> >>>> On Thu, Feb 27, 2014 at 6:17 PM, Tathagata Das < >>>> tathagata.das1...@gmail.com> wrote: >>>> >>>>> Yes! Spark streaming programs are just like any spark program and so >>>>> any ec2 cluster setup using the spark-ec2 scripts can be used to run spark >>>>> streaming programs as well. >>>>> >>>> >>>> Great. Does it come with any input source support as well? (Eg kafka >>>> requires setting up zookeeper). >>>> >>>> >>>>> >>>>> >>>>> >>>>> On Thu, Feb 27, 2014 at 10:11 AM, Aureliano Buendia < >>>>> buendia...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Does the ec2 support for spark 0.9 also include spark streaming? If >>>>>> not, is there an equivalent? >>>>>> >>>>>> >>>>> >>>> >>> >> >
