lynx for accessing webui on console :) Mayur Rustagi Ph: +919632149971 h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com https://twitter.com/mayur_rustagi
On Thu, Feb 20, 2014 at 2:52 PM, Mohit Singh <[email protected]> wrote: > Ok.. That issue is fixed. I didnt set the master and port on > spark-context... > Now after setting things look alright..but now it is stuck at: > 14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 918) > 14/02/20 14:47:15 INFO TaskSetManager: Finished TID 919 in 563 ms on > master (progress: 690/922) > 14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 919) > 14/02/20 14:47:15 INFO TaskSetManager: Finished TID 920 in 558 ms on > master (progress: 691/922) > 14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 920) > 14/02/20 14:47:15 INFO TaskSetManager: Finished TID 916 in 770 ms on > master (progress: 692/922) > 14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 916) > > > And I cant access the web ui because of corporate fire wall etc.. > Any more ideas. > Thanks > > > On Thu, Feb 20, 2014 at 2:41 PM, Mayur Rustagi <[email protected]>wrote: > >> check the spark UI, you'll see in the application area if its working >> there else its running locally. >> >> Mayur Rustagi >> Ph: +919632149971 >> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >> https://twitter.com/mayur_rustagi >> >> >> >> On Thu, Feb 20, 2014 at 2:39 PM, Mohit Singh <[email protected]> wrote: >> >>> Cool. That checked out.. >>> But I still strongly feel that my simple line code job is running >>> locally and not on cluster? >>> Any suggestions. >>> Thanks >>> >>> >>> On Thu, Feb 20, 2014 at 1:45 PM, Mayur Rustagi >>> <[email protected]>wrote: >>> >>>> netstat -an |grep 4040 check ports >>>> In all likelyhood you are running spark shell or something in the >>>> background. >>>> >>>> >>>> Mayur Rustagi >>>> Ph: +919632149971 >>>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >>>> https://twitter.com/mayur_rustagi >>>> >>>> >>>> >>>> On Thu, Feb 20, 2014 at 1:43 PM, Mohit Singh <[email protected]>wrote: >>>> >>>>> Hi, >>>>> I am trying to run spark on a standalone cluster mode.. >>>>> But I am not sure whether it is executing on cluster at all? >>>>> So basically, I have 8 node cluster >>>>> master >>>>> node01 >>>>> node02 >>>>> .... >>>>> node07 >>>>> Spark is installed in master @ ~/spark >>>>> I have copied ~/spark across all the nodes.. >>>>> And then I am trying to use pyspark shell >>>>> >>>>> I load a 10 GB file..present on my master disk (not hdfs etc).. >>>>> and try to count the number of lines: >>>>> I see this: >>>>> 14/02/20 12:54:29 INFO TaskSetManager: Finished TID 420 in 233 ms on >>>>> localhost (progress: 420/922) >>>>> 14/02/20 12:54:29 INFO DAGScheduler: Completed ResultTask(0, 420) >>>>> 14/02/20 12:54:29 INFO BlockManager: Found block broadcast_0 locally >>>>> 14/02/20 12:54:29 INFO HadoopRDD: Input split: >>>>> file:/home/hadoop/data/backup/data/domain/domainz0:14126415872 >>>>> +33554432 >>>>> ..... >>>>> >>>>> >>>>> I am starting pyspark shell as MASTER=spark://master:7070 ./pyspark >>>>> >>>>> Though when I start pyspark.. I do see this error: >>>>> 14/02/20 12:59:45 INFO HttpBroadcast: Broadcast server started at >>>>> http://10.2.1.18:53329 >>>>> 14/02/20 12:59:45 INFO SparkEnv: Registering MapOutputTracker >>>>> 14/02/20 12:59:45 INFO HttpFileServer: HTTP File server directory is >>>>> /tmp/spark-c0b1aebb-1995-4849-873f-eb1a534b4af2 >>>>> 14/02/20 12:59:45 INFO HttpServer: Starting HTTP Server >>>>> 14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED >>>>> [email protected]:4040: java.net.BindException: Address >>>>> already in use >>>>> java.net.BindException: Address already in use >>>>> at sun.nio.ch.Net.bind0(Native Method) >>>>> at sun.nio.ch.Net.bind(Net.java:444) >>>>> at sun.nio.ch.Net.bind(Net.java:436) >>>>> at >>>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) >>>>> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) >>>>> at >>>>> org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) >>>>> at >>>>> org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) >>>>> at >>>>> org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) >>>>> at >>>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) >>>>> at org.eclipse.jetty.server.Server.doStart(Server.java:286) >>>>> at >>>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) >>>>> at >>>>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118) >>>>> at >>>>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118) >>>>> at >>>>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118) >>>>> at scala.util.Try$.apply(Try.scala:161) >>>>> at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118) >>>>> at >>>>> org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129) >>>>> at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57) >>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:159) >>>>> at >>>>> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47) >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>> Method) >>>>> at >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>> at >>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) >>>>> at >>>>> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) >>>>> at py4j.Gateway.invoke(Gateway.java:214) >>>>> at >>>>> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) >>>>> at >>>>> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) >>>>> at py4j.GatewayConnection.run(GatewayConnection.java:207) >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> 14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED >>>>> org.eclipse.jetty.server.Server@37a8d4d2: java.net.BindException: >>>>> Address already in use >>>>> java.net.BindException: Address already in use >>>>> at sun.nio.ch.Net.bind0(Native Method) >>>>> at sun.nio.ch.Net.bind(Net.java:444) >>>>> at sun.nio.ch.Net.bind(Net.java:436) >>>>> at >>>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) >>>>> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) >>>>> at >>>>> org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) >>>>> at >>>>> org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) >>>>> at >>>>> org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) >>>>> at >>>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) >>>>> at org.eclipse.jetty.server.Server.doStart(Server.java:286) >>>>> at >>>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) >>>>> at >>>>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118) >>>>> at >>>>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118) >>>>> at >>>>> org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118) >>>>> at scala.util.Try$.apply(Try.scala:161) >>>>> at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118) >>>>> at >>>>> org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129) >>>>> at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57) >>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:159) >>>>> at >>>>> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47) >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>> Method) >>>>> at >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>> at >>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) >>>>> at >>>>> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) >>>>> at py4j.Gateway.invoke(Gateway.java:214) >>>>> at >>>>> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) >>>>> at >>>>> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) >>>>> at py4j.GatewayConnection.run(GatewayConnection.java:207) >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> 14/02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. >>>>> Trying again. >>>>> 14/02/20 12:59:45 INFO JettyUtils: Error was: >>>>> Failure(java.net.BindException: Address already in use) >>>>> >>>>> 02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. >>>>> Trying again. >>>>> 14/02/20 12:59:45 INFO JettyUtils: Error was: >>>>> Failure(java.net.BindException: Address already in use) >>>>> 14/02/20 12:59:45 INFO SparkUI: Started Spark Web UI at >>>>> http://master:4041 >>>>> 14/02/20 12:59:45 INFO AppClient$ClientActor: Connecting to master >>>>> spark://master:7070... >>>>> 14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to >>>>> akka.tcp://sparkMaster@master:7070: >>>>> akka.remote.EndpointAssociationException: Association failed with >>>>> [akka.tcp://sparkMaster@master:7070] >>>>> 14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to >>>>> akka.tcp://sparkMaster@master:7070: >>>>> akka.remote.EndpointAssociationException: Association failed with >>>>> [akka.tcp://sparkMaster@master:7070] >>>>> 14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to >>>>> akka.tcp://sparkMaster@master:7070: >>>>> akka.remote.EndpointAssociationException: Association failed with >>>>> [akka.tcp://sparkMaster@master:7070] >>>>> 14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to >>>>> akka.tcp://sparkMaster@master:7070: >>>>> akka.remote.EndpointAssociationException: Association failed with >>>>> [akka.tcp://sparkMaster@master:7070] >>>>> >>>>> >>>>> >>>>> And after all this thing.. pyspark shell starts.. >>>>> And if kill 4040 port,, then things work just fine. >>>>> Can someone help me with this >>>>> Thanks >>>>> >>>>> >>>>> -- >>>>> Mohit >>>>> >>>>> "When you want success as badly as you want the air, then you will get >>>>> it. There is no other secret of success." >>>>> -Socrates >>>>> >>>> >>>> >>> >>> >>> -- >>> Mohit >>> >>> "When you want success as badly as you want the air, then you will get >>> it. There is no other secret of success." >>> -Socrates >>> >> >> > > > -- > Mohit > > "When you want success as badly as you want the air, then you will get it. > There is no other secret of success." > -Socrates >
