check the spark UI, you'll see in the application area if its working there else its running locally.
Mayur Rustagi Ph: +919632149971 h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com https://twitter.com/mayur_rustagi On Thu, Feb 20, 2014 at 2:39 PM, Mohit Singh <[email protected]> wrote: > Cool. That checked out.. > But I still strongly feel that my simple line code job is running locally > and not on cluster? > Any suggestions. > Thanks > > > On Thu, Feb 20, 2014 at 1:45 PM, Mayur Rustagi <[email protected]>wrote: > >> netstat -an |grep 4040 check ports >> In all likelyhood you are running spark shell or something in the >> background. >> >> >> Mayur Rustagi >> Ph: +919632149971 >> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >> https://twitter.com/mayur_rustagi >> >> >> >> On Thu, Feb 20, 2014 at 1:43 PM, Mohit Singh <[email protected]> wrote: >> >>> Hi, >>> I am trying to run spark on a standalone cluster mode.. >>> But I am not sure whether it is executing on cluster at all? >>> So basically, I have 8 node cluster >>> master >>> node01 >>> node02 >>> .... >>> node07 >>> Spark is installed in master @ ~/spark >>> I have copied ~/spark across all the nodes.. >>> And then I am trying to use pyspark shell >>> >>> I load a 10 GB file..present on my master disk (not hdfs etc).. >>> and try to count the number of lines: >>> I see this: >>> 14/02/20 12:54:29 INFO TaskSetManager: Finished TID 420 in 233 ms on >>> localhost (progress: 420/922) >>> 14/02/20 12:54:29 INFO DAGScheduler: Completed ResultTask(0, 420) >>> 14/02/20 12:54:29 INFO BlockManager: Found block broadcast_0 locally >>> 14/02/20 12:54:29 INFO HadoopRDD: Input split: >>> file:/home/hadoop/data/backup/data/domain/domainz0:14126415872+33554432 >>> ..... >>> >>> >>> I am starting pyspark shell as MASTER=spark://master:7070 ./pyspark >>> >>> Though when I start pyspark.. I do see this error: >>> 14/02/20 12:59:45 INFO HttpBroadcast: Broadcast server started at >>> http://10.2.1.18:53329 >>> 14/02/20 12:59:45 INFO SparkEnv: Registering MapOutputTracker >>> 14/02/20 12:59:45 INFO HttpFileServer: HTTP File server directory is >>> /tmp/spark-c0b1aebb-1995-4849-873f-eb1a534b4af2 >>> 14/02/20 12:59:45 INFO HttpServer: Starting HTTP Server >>> 14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED >>> [email protected]:4040: java.net.BindException: Address >>> already in use >>> java.net.BindException: Address already in use >>> at sun.nio.ch.Net.bind0(Native Method) >>> at sun.nio.ch.Net.bind(Net.java:444) >>> at sun.nio.ch.Net.bind(Net.java:436) >>> at >>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) >>> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) >>> at >>> org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) >>> at >>> org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) >>> at >>> org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) >>> at >>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) >>> at org.eclipse.jetty.server.Server.doStart(Server.java:286) >>> at >>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) >>> at >>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118) >>> at >>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118) >>> at >>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118) >>> at scala.util.Try$.apply(Try.scala:161) >>> at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118) >>> at >>> org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129) >>> at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57) >>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:159) >>> at >>> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47) >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) >>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) >>> at py4j.Gateway.invoke(Gateway.java:214) >>> at >>> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) >>> at >>> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) >>> at py4j.GatewayConnection.run(GatewayConnection.java:207) >>> at java.lang.Thread.run(Thread.java:744) >>> 14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED >>> org.eclipse.jetty.server.Server@37a8d4d2: java.net.BindException: >>> Address already in use >>> java.net.BindException: Address already in use >>> at sun.nio.ch.Net.bind0(Native Method) >>> at sun.nio.ch.Net.bind(Net.java:444) >>> at sun.nio.ch.Net.bind(Net.java:436) >>> at >>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) >>> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) >>> at >>> org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) >>> at >>> org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) >>> at >>> org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) >>> at >>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) >>> at org.eclipse.jetty.server.Server.doStart(Server.java:286) >>> at >>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) >>> at >>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118) >>> at >>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118) >>> at >>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118) >>> at scala.util.Try$.apply(Try.scala:161) >>> at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118) >>> at >>> org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129) >>> at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57) >>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:159) >>> at >>> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47) >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) >>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) >>> at py4j.Gateway.invoke(Gateway.java:214) >>> at >>> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) >>> at >>> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) >>> at py4j.GatewayConnection.run(GatewayConnection.java:207) >>> at java.lang.Thread.run(Thread.java:744) >>> 14/02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. >>> Trying again. >>> 14/02/20 12:59:45 INFO JettyUtils: Error was: >>> Failure(java.net.BindException: Address already in use) >>> >>> 02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. >>> Trying again. >>> 14/02/20 12:59:45 INFO JettyUtils: Error was: >>> Failure(java.net.BindException: Address already in use) >>> 14/02/20 12:59:45 INFO SparkUI: Started Spark Web UI at >>> http://master:4041 >>> 14/02/20 12:59:45 INFO AppClient$ClientActor: Connecting to master >>> spark://master:7070... >>> 14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to >>> akka.tcp://sparkMaster@master:7070: >>> akka.remote.EndpointAssociationException: Association failed with >>> [akka.tcp://sparkMaster@master:7070] >>> 14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to >>> akka.tcp://sparkMaster@master:7070: >>> akka.remote.EndpointAssociationException: Association failed with >>> [akka.tcp://sparkMaster@master:7070] >>> 14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to >>> akka.tcp://sparkMaster@master:7070: >>> akka.remote.EndpointAssociationException: Association failed with >>> [akka.tcp://sparkMaster@master:7070] >>> 14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to >>> akka.tcp://sparkMaster@master:7070: >>> akka.remote.EndpointAssociationException: Association failed with >>> [akka.tcp://sparkMaster@master:7070] >>> >>> >>> >>> And after all this thing.. pyspark shell starts.. >>> And if kill 4040 port,, then things work just fine. >>> Can someone help me with this >>> Thanks >>> >>> >>> -- >>> Mohit >>> >>> "When you want success as badly as you want the air, then you will get >>> it. There is no other secret of success." >>> -Socrates >>> >> >> > > > -- > Mohit > > "When you want success as badly as you want the air, then you will get it. > There is no other secret of success." > -Socrates >
