for development/testing i think its fine to run them side by side as you suggested, using spark standalone. just be realistic about what size data you can load with limited RAM.
On Fri, Jun 20, 2014 at 3:43 PM, Mayur Rustagi <mayur.rust...@gmail.com> wrote: > The ideal way to do that is to use a cluster manager like Yarn & mesos. > You can control how much resources to give to which node etc. > You should be able to run both together in standalone mode however you may > experience varying latency & performance in the cluster as both MR & spark > demand resources from same machines etc. > > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi <https://twitter.com/mayur_rustagi> > > > > On Fri, Jun 20, 2014 at 3:41 PM, Sameer Tilak <ssti...@live.com> wrote: > >> Dear Spark users, >> >> I have a small 4 node Hadoop cluster. Each node is a VM -- 4 virtual >> cores, 8GB memory and 500GB disk. I am currently running Hadoop on it. I >> would like to run Spark (in standalone mode) along side Hadoop on the same >> nodes. Given the configuration of my nodes, will that work? Does anyone has >> any experience in terms of stability and performance of running Spark and >> Hadoop on somewhat resource-constrained nodes. I was looking at the Spark >> documentation and there is a way to configure memory and cores for the and >> worker nodes and memory for the master node: SPARK_WORKER_CORES, >> SPARK_WORKER_MEMORY, SPARK_DAEMON_MEMORY. Any recommendations on how to >> share resource between HAdoop and Spark? >> >> >> >> >
