RE: Actors and sparkcontext actions
Hi Ognen, See if this helps. I was working on this : class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], hdfsPath : String) extends Actor { def act(){ if(flag1) this.process() else this.count } private def process(){ println(sc.textFile(hdfsPath).count) //do the processing } private def count(){ println(rdd.count) //do the counting } } Thanks and Regards, Suraj Sheth -Original Message- From: Ognen Duzlevski [mailto:og...@nengoiksvelzud.com] Sent: 27 February 2014 01:09 To: u...@spark.incubator.apache.org Subject: Actors and sparkcontext actions Can someone point me to a simple, short code example of creating a basic Actor that gets a context and runs an operation such as .textFile.count? I am trying to figure out how to create just a basic actor that gets a message like this: case class Msg(filename:String, ctx: SparkContext) and then something like this: class HelloActor extends Actor { import context.dispatcher def receive = { case Msg(fn,ctx) => { // get the count here! // cts.textFile(fn).count } case _ => println("huh?") } } Where I would want to do something like: val conf = new SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9") val sc = new SparkContext(conf) val system = ActorSystem("mySystem") val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1") helloActor1 ! new Msg("test.json",sc) Thanks, Ognen
Re: Actors and sparkcontext actions
Suraj, I posted to this list a link to my blog where I detail how to do a simple actor/sparkcontext thing with the added obstacle of it being within a Scalatra servlet. Thanks for the code! Ognen On 3/4/14, 3:20 AM, Suraj Satishkumar Sheth wrote: Hi Ognen, See if this helps. I was working on this : class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], hdfsPath : String) extends Actor { def act(){ if(flag1) this.process() else this.count } private def process(){ println(sc.textFile(hdfsPath).count) //do the processing } private def count(){ println(rdd.count) //do the counting } } Thanks and Regards, Suraj Sheth -Original Message- From: Ognen Duzlevski [mailto:og...@nengoiksvelzud.com] Sent: 27 February 2014 01:09 To: u...@spark.incubator.apache.org Subject: Actors and sparkcontext actions Can someone point me to a simple, short code example of creating a basic Actor that gets a context and runs an operation such as .textFile.count? I am trying to figure out how to create just a basic actor that gets a message like this: case class Msg(filename:String, ctx: SparkContext) and then something like this: class HelloActor extends Actor { import context.dispatcher def receive = { case Msg(fn,ctx) => { // get the count here! // cts.textFile(fn).count } case _ => println("huh?") } } Where I would want to do something like: val conf = new SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9") val sc = new SparkContext(conf) val system = ActorSystem("mySystem") val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1") helloActor1 ! new Msg("test.json",sc) Thanks, Ognen -- Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. -- Jamie Zawinski
Re: Actors and sparkcontext actions
Hi Ognen, Any particular reason of choosing scalatra over options like play or spray ? Is scalatra much better in serving apis or is it due to similarity with ruby's sinatra ? Did you try the other options and then pick scalatra ? Thanks. Deb On Tue, Mar 4, 2014 at 4:50 AM, Ognen Duzlevski wrote: > Suraj, I posted to this list a link to my blog where I detail how to do a > simple actor/sparkcontext thing with the added obstacle of it being within > a Scalatra servlet. > > Thanks for the code! > Ognen > > > On 3/4/14, 3:20 AM, Suraj Satishkumar Sheth wrote: > >> Hi Ognen, >> See if this helps. I was working on this : >> >> class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], >> hdfsPath : String) extends Actor { >> >>def act(){ >> if(flag1) this.process() >> else this.count >>} >> private def process(){ >> println(sc.textFile(hdfsPath).count) >> //do the processing >>} >> private def count(){ >> println(rdd.count) >> //do the counting >>} >> >> } >> >> Thanks and Regards, >> Suraj Sheth >> >> >> -Original Message- >> From: Ognen Duzlevski [mailto:og...@nengoiksvelzud.com] >> Sent: 27 February 2014 01:09 >> To: u...@spark.incubator.apache.org >> Subject: Actors and sparkcontext actions >> >> Can someone point me to a simple, short code example of creating a basic >> Actor that gets a context and runs an operation such as .textFile.count? >> I am trying to figure out how to create just a basic actor that gets a >> message like this: >> >> case class Msg(filename:String, ctx: SparkContext) >> >> and then something like this: >> >> class HelloActor extends Actor { >> import context.dispatcher >> >> def receive = { >> case Msg(fn,ctx) => { >> // get the count here! >> // cts.textFile(fn).count >> } >> case _ => println("huh?") >> } >> } >> >> Where I would want to do something like: >> >> val conf = new >> SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello"). >> setSparkHome("/Users/maketo/plainvanilla/spark-0.9") >> val sc = new SparkContext(conf) >> val system = ActorSystem("mySystem") >> >> val helloActor1 = system.actorOf( Props[ HelloActor], name = >> "helloactor1") >> helloActor1 ! new Msg("test.json",sc) >> >> Thanks, >> Ognen >> > > -- > Some people, when confronted with a problem, think "I know, I'll use > regular expressions." Now they have two problems. > -- Jamie Zawinski > >
Re: Actors and sparkcontext actions
Deb, On 3/4/14, 9:02 AM, Debasish Das wrote: Hi Ognen, Any particular reason of choosing scalatra over options like play or spray ? Is scalatra much better in serving apis or is it due to similarity with ruby's sinatra ? Did you try the other options and then pick scalatra ? Not really. I just happen to like Scalatra, it is easy with read and easy to write. If it did not work out I would have gone probably with something like Unfiltered. Ognen