Sujeet: Please also see:
https://spark.apache.org/docs/latest/spark-standalone.html On Sat, May 28, 2016 at 9:19 AM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Sujeet, > > if you have a single machine then it is Spark standalone mode. > > In Standalone cluster mode Spark allocates resources based on cores. By > default, an application will grab all the cores in the cluster. > > You only have one worker that lives within the driver JVM process that you > start when you start the application with spark-shell or spark-submit in > the host where the cluster manager is running. > > The Driver node runs on the same host that the cluster manager is running. > The Driver requests the Cluster Manager for resources to run tasks. The > worker is tasked to create the executor (in this case there is only one > executor) for the Driver. The Executor runs tasks for the Driver. Only one > executor can be allocated on each worker per application. In your case you > only have > > > The minimum you will need will be 2-4G of RAM and two cores. Well that is > my experience. Yes you can submit more than one spark-submit (the driver) > but they may queue up behind the running one if there is not enough > resources. > > > You pointed out that you will be running few applications in parallel on > the same host. The likelihood is that you are using a VM machine for this > purpose and the best option is to try running the first one, Check Web GUI > on 4040 to see the progress of this Job. If you start the next JVM then > assuming it is working, it will be using port 4041 and so forth. > > > In actual fact try the command "free" to see how much free memory you have. > > > HTH > > > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 28 May 2016 at 16:42, sujeet jog <sujeet....@gmail.com> wrote: > >> Hi, >> >> I have a question w.r.t production deployment mode of spark, >> >> I have 3 applications which i would like to run independently on a single >> machine, i need to run the drivers in the same machine. >> >> The amount of resources i have is also limited, like 4- 5GB RAM , 3 - 4 >> cores. >> >> For deployment in standalone mode : i believe i need >> >> 1 Driver JVM, 1 worker node ( 1 executor ) >> 1 Driver JVM, 1 worker node ( 1 executor ) >> 1 Driver JVM, 1 worker node ( 1 executor ) >> >> The issue here is i will require 6 JVM running in parallel, for which i >> do not have sufficient CPU/MEM resources, >> >> >> Hence i was looking more towards a local mode deployment mode, would like >> to know if anybody is using local mode where Driver + Executor run in a >> single JVM in production mode. >> >> Are there any inherent issues upfront using local mode for production >> base systems.?.. >> >> >
