Hi Mich, sure that workers are mentioned in slaves file. I can see them in spark master UI and even after start they are "blocked" for this application but the cpu and memory consumption is close to nothing.
Thanks Jakub On 4 July 2016 at 18:36, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Silly question. Have you added your workers to sbin/slaves file and have > you started start-slaves.sh. > > on master node when you type jps what do you see? > > The problem seems to be that workers are ignored and spark is essentially > running in Local mode > > HTH > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 4 July 2016 at 17:05, Jakub Stransky <stransky...@gmail.com> wrote: > >> Hi Mich, >> >> I have set up spark default configuration in conf directory >> spark-defaults.conf where I specify master hence no need to put it in >> command line >> spark.master spark://spark.master:7077 >> >> the same applies to driver memory which has been increased to 4GB >> and the same is for spark.executor.memory 12GB as machines have 16GB >> >> Jakub >> >> >> >> >> On 4 July 2016 at 17:44, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >>> Hi Jakub, >>> >>> In standalone mode Spark does the resource management. Which version of >>> Spark are you running? >>> >>> How do you define your SparkConf() parameters for example setMaster etc. >>> >>> From >>> >>> spark-submit --driver-class-path spark/sqljdbc4.jar --class DemoApp >>> SparkPOC.jar 10 4.3 >>> >>> I did not see any executor, memory allocation, so I assume you are >>> allocating them somewhere else? >>> >>> HTH >>> >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 4 July 2016 at 16:31, Jakub Stransky <stransky...@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I have a spark cluster consisting of 4 nodes in a standalone mode, >>>> master + 3 workers nodes with configured available memory and cpus etc. >>>> >>>> I have an spark application which is essentially a MLlib pipeline for >>>> training a classifier, in this case RandomForest but could be a >>>> DecesionTree just for the sake of simplicity. >>>> >>>> But when I submit the spark application to the cluster via spark submit >>>> it is running out of memory. Even though the executors are "taken"/created >>>> in the cluster they are esentially doing nothing ( poor cpu, nor memory >>>> utilization) while the master seems to do all the work which finally >>>> results in OOM. >>>> >>>> My submission is following: >>>> spark-submit --driver-class-path spark/sqljdbc4.jar --class DemoApp >>>> SparkPOC.jar 10 4.3 >>>> >>>> I am submitting from the master node. >>>> >>>> By default it is running in client mode which the driver process is >>>> attached to spark-shell. >>>> >>>> Do I need to set up some settings to make MLlib algos parallelized and >>>> distributed as well or all is driven by parallel factor set on dataframe >>>> with input data? >>>> >>>> Essentially it seems that all work is just done on master and the rest >>>> is idle. >>>> Any hints what to check? >>>> >>>> Thx >>>> Jakub >>>> >>>> >>>> >>>> >>> >> >> >> -- >> Jakub Stransky >> cz.linkedin.com/in/jakubstransky >> >> > -- Jakub Stransky cz.linkedin.com/in/jakubstransky
