I do notice that scala 2.9.2 is being included because of net.liftweb. Also, I don't know if I just missed it before or it wasn't doing this before and my latest changes get it a little farther, but I'm now seeing the following in the spark logs:
14/02/28 20:13:29 INFO actor.ActorSystemImpl: RemoteClientStarted @akka://spark@hadoop-s1.oculus.local:35212 14/02/28 20:13:29 ERROR NettyRemoteTransport(null): dropping message RegisterApplication(ApplicationDescription(Web Service Spark Instance)) for non-local recipient akka://sparkMaster@192.168.0.46:7077/user/Master at akka://sparkMaster@hadoop-s1.oculus.local:7077 local is akka://sparkMaster@hadoop-s1.oculus.local:7077 14/02/28 20:13:49 ERROR NettyRemoteTransport(null): dropping message RegisterApplication(ApplicationDescription(Web Service Spark Instance)) for non-local recipient akka://sparkMaster@192.168.0.46:7077/user/Master at akka://sparkMaster@hadoop-s1.oculus.local:7077 local is akka://sparkMaster@hadoop-s1.oculus.local:7077 14/02/28 20:14:09 ERROR NettyRemoteTransport(null): dropping message RegisterApplication(ApplicationDescription(Web Service Spark Instance)) for non-local recipient akka://sparkMaster@192.168.0.46:7077/user/Master at akka://sparkMaster@hadoop-s1.oculus.local:7077 local is akka://sparkMaster@hadoop-s1.oculus.local:7077 14/02/28 20:14:32 INFO actor.ActorSystemImpl: RemoteClientShutdown @akka://spark@hadoop-s1.oculus.local:35212 On Sat, Feb 22, 2014 at 1:58 PM, Soumya Simanta <soumya.sima...@gmail.com>wrote: > Mostly likely all your classes/jars that are required to connect to Spark > and not being loaded or the incorrect versions are being loaded when you > start to do this from inside the web container (Tomcat). > > > > > > On Sat, Feb 22, 2014 at 1:51 PM, Nathan Kronenfeld < > nkronenf...@oculusinfo.com> wrote: > >> yes, but only when I try to connect from a web service running in Tomcat. >> >> When I try to connect using a stand-alone program, using the same >> parameters, it works fine. >> >> >> On Sat, Feb 22, 2014 at 12:15 PM, Mayur Rustagi >> <mayur.rust...@gmail.com>wrote: >> >>> So Spark is running on that IP, web ui is loading on that IP showing >>> workers & when you connect to that IP with javaAPI the cluster appears to >>> be down to it? >>> >>> Mayur Rustagi >>> Ph: +919632149971 >>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >>> https://twitter.com/mayur_rustagi >>> >>> >>> >>> On Fri, Feb 21, 2014 at 10:22 PM, Nathan Kronenfeld < >>> nkronenf...@oculusinfo.com> wrote: >>> >>>> Netstat gives exactly the expected IP address (not a 127...., but a >>>> 192...). >>>> I tried it anyway, though... exactly the same results, but with a >>>> number instead of a name. >>>> Oh, and I forgot to mention last time, in case it makes a difference - >>>> I'm running 0.8.1, not 0.9.0, at least for now >>>> >>>> >>>> >>>> On Sat, Feb 22, 2014 at 12:50 AM, Mayur Rustagi < >>>> mayur.rust...@gmail.com> wrote: >>>> >>>>> most likely the master is binding to a unique address and you are >>>>> connecting to some other internal address. Master can bind to random >>>>> internal address 127.0... or even your machine IP at that time. >>>>> Easiest is to check >>>>> netstat -an |grep 7077 >>>>> This will give you which IP to bind to exactly when launching spark >>>>> context. >>>>> >>>>> Mayur Rustagi >>>>> Ph: +919632149971 >>>>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >>>>> https://twitter.com/mayur_rustagi >>>>> >>>>> >>>>> >>>>> On Fri, Feb 21, 2014 at 9:36 PM, Nathan Kronenfeld < >>>>> nkronenf...@oculusinfo.com> wrote: >>>>> >>>>>> Can anyone help me here? >>>>>> >>>>>> I've got a small spark cluster running on three machines - hadoop-s1, >>>>>> hadoop-s2, and hadoop-s3 - with s1 acting master, and all three acting as >>>>>> workers. It works fine - I can connect with spark-shell, I can run >>>>>> jobs, I >>>>>> can see the web ui. >>>>>> >>>>>> The web UI says: >>>>>> Spark Master at spark://hadoop-s1.oculus.local:7077 >>>>>> URL: spark://hadoop-s1.oculus.local:7077 >>>>>> >>>>>> I've connected to it fine using both a scala and a java SparkContext. >>>>>> >>>>>> But when I try connecting from within a Tomcat service, I get the >>>>>> following messages: >>>>>> [INFO] 22 Feb 2014 00:27:38 - org.apache.spark.Logging$class - >>>>>> Connecting to master spark://hadoop-s1.oculus.local:7077... >>>>>> [INFO] 22 Feb 2014 00:27:58 - org.apache.spark.Logging$class - >>>>>> Connecting to master spark://hadoop-s1.oculus.local:7077... >>>>>> [ERROR] 22 Feb 2014 00:28:18 - org.apache.spark.Logging$class - All >>>>>> masters are unresponsive! Giving up. >>>>>> [ERROR] 22 Feb 2014 00:28:18 - org.apache.spark.Logging$class - Spark >>>>>> cluster looks dead, giving up. >>>>>> [ERROR] 22 Feb 2014 00:28:18 - org.apache.spark.Logging$class - >>>>>> Exiting due to error from cluster scheduler: Spark cluster looks down >>>>>> >>>>>> When I look on the spark server logs, there isn't even a sign of an >>>>>> attempted connection. >>>>>> >>>>>> I'm trying to use a JavaSparkContext, and I've printed out the >>>>>> parameters I pass in, and they work fine in a stand-alone program. >>>>>> >>>>>> Anyone have a clue why this fails? Or even how to find out why this >>>>>> fals? >>>>>> >>>>>> >>>>>> -- >>>>>> Nathan Kronenfeld >>>>>> Senior Visualization Developer >>>>>> Oculus Info Inc >>>>>> 2 Berkeley Street, Suite 600, >>>>>> Toronto, Ontario M5A 4J5 >>>>>> Phone: +1-416-203-3003 x 238 >>>>>> Email: nkronenf...@oculusinfo.com >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Nathan Kronenfeld >>>> Senior Visualization Developer >>>> Oculus Info Inc >>>> 2 Berkeley Street, Suite 600, >>>> Toronto, Ontario M5A 4J5 >>>> Phone: +1-416-203-3003 x 238 >>>> Email: nkronenf...@oculusinfo.com >>>> >>> >>> >> >> >> -- >> Nathan Kronenfeld >> Senior Visualization Developer >> Oculus Info Inc >> 2 Berkeley Street, Suite 600, >> Toronto, Ontario M5A 4J5 >> Phone: +1-416-203-3003 x 238 >> Email: nkronenf...@oculusinfo.com >> > > -- Nathan Kronenfeld Senior Visualization Developer Oculus Info Inc 2 Berkeley Street, Suite 600, Toronto, Ontario M5A 4J5 Phone: +1-416-203-3003 x 238 Email: nkronenf...@oculusinfo.com