Hello,
Is there anything special about calling functions that parse json lines
from filter?
I have code that looks like this:
jsonMatches(line:String):Boolean = {
take a line in json format
val jline=parse(line)
val je = jline \ "event"
if (je != JNothing && je.values.toString == userSuppliedEventSelector) {
val jp = jline \ "properties"
val lineVals = for {
p <- userProps
val lp = jp \ p
if (lp != JNothing && lp.values.toString ==
userSuppliedValueSelector)
} yield lp.toValues.toString
if (userSuppliedVals.toSet == lineVals)
return true
}
false
}
userSuppliedEventSelector and userSuppliedValueSelector are Strings
passed into the function that jsonMatches() is embedded within as is the
following code.
I then do something like:
val f = sc.textFile(hdfs:\\somefile)
val ev1rdd = f.filter(jsonMatches).map(something => somethingElse)
I am getting the following
14/03/13 14:55:21 ERROR OneForOneStrategy: Job aborted: Task not
serializable: java.io.NotSerializableException:
com.github.ognenpv.pipeline.CountActor
I suspect that the json parsing library is not serializable?
Any ideas? What do people do to achieve what I want to do?
Thanks!
Ognen