Yea, I checked that. There is kind of POC and wanted to keep the things as simple as possible. So far tried just ML task. I am more afraid that I actually forget to set something up e.g. some properties at the level of ML lib - for paralelization or so. But wasn't able to fine anythig like that at the documentation
On 5 July 2016 at 17:22, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Jakub, > > Any reason why you are running in standalone mode, given that your are > familiar with YARN? > > In theory your settings are correct. I checked your environment tab > settings and they look correct. > > I assume you have checked this link > > http://spark.apache.org/docs/latest/spark-standalone.html > > BTW is this issue confined to ML or any other Spark application exhibits > the same behaviour in standalone mode? > > > HTH > > > > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 5 July 2016 at 11:17, Jacek Laskowski <ja...@japila.pl> wrote: > >> Hi Jakub, >> >> You're correct - spark.master spark://master.clust:7077 - proves your >> point. You're running Spark Standalone that was set in >> conf/spark-defaults.conf perhaps. >> >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark http://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski >> >> On Tue, Jul 5, 2016 at 12:04 PM, Jakub Stransky <stransky...@gmail.com> >> wrote: >> >>> Hello, >>> >>> I am convinced that we are not running in local mode: >>> >>> Runtime Information >>> >>> Name Value >>> Java Home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre >>> Java Version 1.7.0_65 (Oracle Corporation) >>> Scala Version version 2.10.5 >>> Spark Properties >>> >>> Name Value >>> spark.app.id app-20160704121044-0003 >>> spark.app.name DemoApp >>> spark.driver.extraClassPath /home/sparkuser/sqljdbc4.jar >>> spark.driver.host 10.2.0.4 >>> spark.driver.memory 4g >>> spark.driver.port 59493 >>> spark.executor.extraClassPath /usr/local/spark-1.6.1/sqljdbc4.jar >>> spark.executor.id driver >>> spark.executor.memory 12g >>> spark.externalBlockStore.folderName >>> spark-5630dd34-4267-462e-882e-b382832bb500 >>> spark.jars file:/home/sparkuser/SparkPOC.jar >>> spark.master spark://master.clust:7077 >>> spark.scheduler.mode FIFO >>> spark.submit.deployMode client >>> System Properties >>> >>> Name Value >>> SPARK_SUBMIT true >>> awt.toolkit sun.awt.X11.XToolkit >>> file.encoding UTF-8 >>> file.encoding.pkg sun.io >>> file.separator / >>> java.awt.graphicsenv sun.awt.X11GraphicsEnvironment >>> java.awt.printerjob sun.print.PSPrinterJob >>> java.class.version 51.0 >>> java.endorsed.dirs >>> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/endorsed >>> java.ext.dirs >>> >>> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/ext:/usr/java/packages/lib/ext >>> java.home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre >>> java.io.tmpdir /tmp >>> java.library.path >>> /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >>> java.runtime.name OpenJDK Runtime Environment >>> java.runtime.version 1.7.0_65-mockbuild_2014_07_16_06_06-b00 >>> java.specification.name Java Platform API Specification >>> java.specification.vendor Oracle Corporation >>> java.specification.version 1.7 >>> java.vendor Oracle Corporation >>> java.vendor.url http://java.oracle.com/ >>> java.vendor.url.bug http://bugreport.sun.com/bugreport/ >>> java.version 1.7.0_65 >>> java.vm.info mixed mode >>> java.vm.name OpenJDK 64-Bit Server VM >>> java.vm.specification.name Java Virtual Machine Specification >>> java.vm.specification.vendor Oracle Corporation >>> java.vm.specification.version 1.7 >>> java.vm.vendor Oracle Corporation >>> java.vm.version 24.65-b04 >>> line.separator >>> os.arch amd64 >>> os.name Linux >>> os.version 2.6.32-431.29.2.el6.x86_64 >>> path.separator : >>> sun.arch.data.model 64 >>> sun.boot.class.path >>> >>> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/rhino.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/classes >>> sun.boot.library.path >>> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/amd64 >>> sun.cpu.endian little >>> sun.cpu.isalist >>> sun.io.unicode.encoding UnicodeLittle >>> sun.java.command org.apache.spark.deploy.SparkSubmit --conf >>> spark.driver.extraClassPath=/home/sparkuser/sqljdbc4.jar --class --class >>> DemoApp SparkPOC.jar 10 4.3 >>> sun.java.launcher SUN_STANDARD >>> sun.jnu.encoding UTF-8 >>> sun.management.compiler HotSpot 64-Bit Tiered Compilers >>> sun.nio.ch.bugLevel >>> sun.os.patch.level unknown >>> user.country US >>> user.dir /home/sparkuser >>> user.home /home/sparkuser >>> user.language en >>> user.name sparkuser >>> user.timezone Etc/UTC >>> Classpath Entries >>> >>> Resource Source >>> /home/sparkuser/sqljdbc4.jar System Classpath >>> /usr/local/spark-1.6.1/assembly/target/scala-2.10/spark-assembly-1.6.1-hadoop2.2.0.jar >>> System Classpath >>> /usr/local/spark-1.6.1/conf/ System Classpath >>> /usr/local/spark-1.6.1/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar >>> System Classpath >>> /usr/local/spark-1.6.1/lib_managed/jars/datanucleus-core-3.2.10.jar >>> System Classpath >>> /usr/local/spark-1.6.1/lib_managed/jars/datanucleus-rdbms-3.2.9.jar >>> System Classpath >>> http://10.2.0.4:35639/jars/SparkPOC.jar Added By User >>> >>> On 4 July 2016 at 21:43, Mich Talebzadeh <mich.talebza...@gmail.com> >>> wrote: >>> >>>> well this will be apparent from the Environment tab of GUI. It will >>>> show how the job is actually running. >>>> >>>> Jacek's point is correct. I suspect this is actually running in Local >>>> mode as it looks consuming all from the master node. >>>> >>>> HTH >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> On 4 July 2016 at 20:35, Jacek Laskowski <ja...@japila.pl> wrote: >>>> >>>>> On Mon, Jul 4, 2016 at 8:36 PM, Mathieu Longtin < >>>>> math...@closetwork.org> wrote: >>>>> >>>>>> Are you using a --master argument, or equivalent config, when calling >>>>>> spark-submit? >>>>>> >>>>>> If you don't, it runs in standalone mode. >>>>>> >>>>> >>>>> s/standalone/local[*] >>>>> >>>>> Jacek >>>>> >>>> >>>> >>> >>> >>> -- >>> Jakub Stransky >>> cz.linkedin.com/in/jakubstransky >>> >>> >> > -- Jakub Stransky cz.linkedin.com/in/jakubstransky
