Hi,
I tried clearing maven and ivy cache and I am a bit confused at this point
in time.
1) Running the example from the spark directory and running using
bin/run-example. It works fine as well as it prints the word counts.
2) Trying to run the same code as a seperate job.
*) Using the latest 1.0.0-SNAPSHOT it doesn't work and throws exception.
*) Using 0.9.1 doesn't throws any exception but doesn't print any word
counts.
Thanks,
Shrikar
On Thu, May 22, 2014 at 9:19 PM, Soumya Simanta <soumya.sima...@gmail.com>wrote:
> Try cleaning your maven (.m2) and ivy cache.
>
>
>
> On May 23, 2014, at 12:03 AM, Shrikar archak <shrika...@gmail.com> wrote:
>
> Yes I did a sbt publish-local. Ok I will try with Spark 0.9.1.
>
> Thanks,
> Shrikar
>
>
> On Thu, May 22, 2014 at 8:53 PM, Tathagata Das <
> tathagata.das1...@gmail.com> wrote:
>
>> How are you getting Spark with 1.0.0-SNAPSHOT through maven? Did you
>> publish Spark locally which allowed you to use it as a dependency?
>>
>> This is a weird indeed. SBT should take care of all the dependencies of
>> spark.
>>
>> In any case, you can try the last released Spark 0.9.1 and see if the
>> problem persists.
>>
>>
>> On Thu, May 22, 2014 at 3:59 PM, Shrikar archak <shrika...@gmail.com>wrote:
>>
>>> I am running as sbt run. I am running it locally .
>>>
>>> Thanks,
>>> Shrikar
>>>
>>>
>>> On Thu, May 22, 2014 at 3:53 PM, Tathagata Das <
>>> tathagata.das1...@gmail.com> wrote:
>>>
>>>> How are you launching the application? sbt run ? spark-submit? local
>>>> mode or Spark standalone cluster? Are you packaging all your code into
>>>> a jar?
>>>> Looks to me that you seem to have spark classes in your execution
>>>> environment but missing some of Spark's dependencies.
>>>>
>>>> TD
>>>>
>>>>
>>>>
>>>> On Thu, May 22, 2014 at 2:27 PM, Shrikar archak <shrika...@gmail.com>
>>>> wrote:
>>>> > Hi All,
>>>> >
>>>> > I am trying to run the network count example as a seperate standalone
>>>> job
>>>> > and running into some issues.
>>>> >
>>>> > Environment:
>>>> > 1) Mac Mavericks
>>>> > 2) Latest spark repo from Github.
>>>> >
>>>> >
>>>> > I have a structure like this
>>>> >
>>>> > Shrikars-MacBook-Pro:SimpleJob shrikar$ find .
>>>> > .
>>>> > ./simple.sbt
>>>> > ./src
>>>> > ./src/main
>>>> > ./src/main/scala
>>>> > ./src/main/scala/NetworkWordCount.scala
>>>> > ./src/main/scala/SimpleApp.scala.bk
>>>> >
>>>> >
>>>> > simple.sbt
>>>> > name := "Simple Project"
>>>> >
>>>> > version := "1.0"
>>>> >
>>>> > scalaVersion := "2.10.3"
>>>> >
>>>> > libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" %
>>>> > "1.0.0-SNAPSHOT",
>>>> > "org.apache.spark" %% "spark-streaming" %
>>>> > "1.0.0-SNAPSHOT")
>>>> >
>>>> > resolvers += "Akka Repository" at "http://repo.akka.io/releases/";
>>>> >
>>>> >
>>>> > I am able to run the SimpleApp which is mentioned in the doc but when
>>>> I try
>>>> > to run the NetworkWordCount app I get error like this am I missing
>>>> > something?
>>>> >
>>>> > [info] Running com.shrikar.sparkapps.NetworkWordCount
>>>> > 14/05/22 14:26:47 INFO spark.SecurityManager: Changing view acls to:
>>>> shrikar
>>>> > 14/05/22 14:26:47 INFO spark.SecurityManager: SecurityManager:
>>>> > authentication disabled; ui acls disabled; users with view
>>>> permissions:
>>>> > Set(shrikar)
>>>> > 14/05/22 14:26:48 INFO slf4j.Slf4jLogger: Slf4jLogger started
>>>> > 14/05/22 14:26:48 INFO Remoting: Starting remoting
>>>> > 14/05/22 14:26:48 INFO Remoting: Remoting started; listening on
>>>> addresses
>>>> > :[akka.tcp://spark@192.168.10.88:49963]
>>>> > 14/05/22 14:26:48 INFO Remoting: Remoting now listens on addresses:
>>>> > [akka.tcp://spark@192.168.10.88:49963]
>>>> > 14/05/22 14:26:48 INFO spark.SparkEnv: Registering MapOutputTracker
>>>> > 14/05/22 14:26:48 INFO spark.SparkEnv: Registering BlockManagerMaster
>>>> > 14/05/22 14:26:48 INFO storage.DiskBlockManager: Created local
>>>> directory at
>>>> >
>>>> /var/folders/r2/mbj08pb55n5d_9p8588xk5b00000gn/T/spark-local-20140522142648-0a14
>>>> > 14/05/22 14:26:48 INFO storage.MemoryStore: MemoryStore started with
>>>> > capacity 911.6 MB.
>>>> > 14/05/22 14:26:48 INFO network.ConnectionManager: Bound socket to
>>>> port 49964
>>>> > with id = ConnectionManagerId(192.168.10.88,49964)
>>>> > 14/05/22 14:26:48 INFO storage.BlockManagerMaster: Trying to register
>>>> > BlockManager
>>>> > 14/05/22 14:26:48 INFO storage.BlockManagerInfo: Registering block
>>>> manager
>>>> > 192.168.10.88:49964 with 911.6 MB RAM
>>>> > 14/05/22 14:26:48 INFO storage.BlockManagerMaster: Registered
>>>> BlockManager
>>>> > 14/05/22 14:26:48 INFO spark.HttpServer: Starting HTTP Server
>>>> > [error] (run-main) java.lang.NoClassDefFoundError:
>>>> > javax/servlet/http/HttpServletResponse
>>>> > java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>>>> > at org.apache.spark.HttpServer.start(HttpServer.scala:54)
>>>> > at
>>>> >
>>>> org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
>>>> > at
>>>> >
>>>> org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
>>>> > at
>>>> >
>>>> org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
>>>> > at
>>>> >
>>>> org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
>>>> > at
>>>> >
>>>> org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
>>>> > at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
>>>> > at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
>>>> > at
>>>> >
>>>> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:549)
>>>> > at
>>>> >
>>>> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:561)
>>>> > at
>>>> >
>>>> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:91)
>>>> > at
>>>> com.shrikar.sparkapps.NetworkWordCount$.main(NetworkWordCount.scala:39)
>>>> > at com.shrikar.sparkapps.NetworkWordCount.main(NetworkWordCount.scala)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> >
>>>> >
>>>> > Thanks,
>>>> > Shrikar
>>>> >
>>>>
>>>
>>>
>>
>
