Hi all In my spark application,I load a csv file and map the datas to a Map
vairable for later uses on driver node ,then broadcast it,every thing works
fine untill the exception java.io.FileNotFoundException occurs.the console log
information shows me the broadcast unavailable,I googled this problem,says
spark will clean up the broadcast,while these's an solution,the author
mentioned about re-broadcast,I followed this way,written some exception handle
code with `try` ,`catch`.after compliling and submitting the jar,I faced
anthoner problem,It shows " task not serializable".
so here I have there options:
1,get the right way persisting broadcast
2,solve the "task not serializable" problem re-broadcast variable
3,save the data to some kind of database,although I prefer save data in memory.
here is come code snippets:
val esRdd = kafkaDStreams.flatMap(_.split("\\n"))
.map{
case esregex(datetime, time_request) =>
var ipInfo:Array[String]=Array.empty
try{
ipInfo = Utils.getIpInfo(client_ip,b.value)
}catch{
case e:java.io.FileNotFoundException =>{
val b = Utils.load(sc,ip_lib_path)
ipInfo = Utils.getIpInfo(client_ip,b.value)
}
}
