The ideal way to do that is to use a cluster manager like Yarn & mesos. You can control how much resources to give to which node etc. You should be able to run both together in standalone mode however you may experience varying latency & performance in the cluster as both MR & spark demand resources from same machines etc.
Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi> On Fri, Jun 20, 2014 at 3:41 PM, Sameer Tilak <[email protected]> wrote: > Dear Spark users, > > I have a small 4 node Hadoop cluster. Each node is a VM -- 4 virtual > cores, 8GB memory and 500GB disk. I am currently running Hadoop on it. I > would like to run Spark (in standalone mode) along side Hadoop on the same > nodes. Given the configuration of my nodes, will that work? Does anyone has > any experience in terms of stability and performance of running Spark and > Hadoop on somewhat resource-constrained nodes. I was looking at the Spark > documentation and there is a way to configure memory and cores for the and > worker nodes and memory for the master node: SPARK_WORKER_CORES, > SPARK_WORKER_MEMORY, SPARK_DAEMON_MEMORY. Any recommendations on how to > share resource between HAdoop and Spark? > > > >
