Hello,

Is there anything special about calling functions that parse json lines from filter?

I have code that looks like this:

jsonMatches(line:String):Boolean = {
  take a line in json format
  val jline=parse(line)
  val je = jline \ "event"
  if (je != JNothing && je.values.toString == userSuppliedEventSelector) {
     val jp = jline \ "properties"
     val lineVals = for {
       p <- userProps
       val lp = jp \ p
if (lp != JNothing && lp.values.toString == userSuppliedValueSelector)
     } yield lp.toValues.toString
     if (userSuppliedVals.toSet == lineVals)
       return true
   }
   false
}

userSuppliedEventSelector and userSuppliedValueSelector are Strings passed into the function that jsonMatches() is embedded within as is the following code.

I then do something like:

val f = sc.textFile(hdfs:\\somefile)
val ev1rdd = f.filter(jsonMatches).map(something => somethingElse)

I am getting the following
14/03/13 14:55:21 ERROR OneForOneStrategy: Job aborted: Task not serializable: java.io.NotSerializableException: com.github.ognenpv.pipeline.CountActor

I suspect that the json parsing library is not serializable?

Any ideas? What do people do to achieve what I want to do?

Thanks!
Ognen

Reply via email to