Hi Ognen,
See if this helps. I was working on this :
class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], hdfsPath :
String) extends Actor {
def act(){
if(flag1) this.process()
else this.count
}
private def process(){
println(sc.textFile(hdfsPath).count)
//do the processing
}
private def count(){
println(rdd.count)
//do the counting
}
}
Thanks and Regards,
Suraj Sheth
-----Original Message-----
From: Ognen Duzlevski [mailto:[email protected]]
Sent: 27 February 2014 01:09
To: [email protected]
Subject: Actors and sparkcontext actions
Can someone point me to a simple, short code example of creating a basic Actor
that gets a context and runs an operation such as .textFile.count?
I am trying to figure out how to create just a basic actor that gets a message
like this:
case class Msg(filename:String, ctx: SparkContext)
and then something like this:
class HelloActor extends Actor {
import context.dispatcher
def receive = {
case Msg(fn,ctx) => {
// get the count here!
// cts.textFile(fn).count
}
case _ => println("huh?")
}
}
Where I would want to do something like:
val conf = new
SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
val sc = new SparkContext(conf)
val system = ActorSystem("mySystem")
val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1")
helloActor1 ! new Msg("test.json",sc)
Thanks,
Ognen
