I built my Spark Streaming app on my local machine, and an initial step in log processing is filtering out rows with spam IPs. I use the following code which works locally:
// Creates a HashSet for badIPs read in from file val badIpSource = scala.io.Source.fromFile("wrongIPlist.csv") val ipLines = badIpSource.getLines() val set = new HashSet[String]() val badIpSet = set ++ ipLines badIpSource.close() def isGoodIp(ip: String): Boolean = !badIpSet.contains(ip) But when I try this using "--master yarn-cluster" I get "Exception in thread "Thread-4" java.lang.reflect.InvocationTargetException ... Caused by: java.io.FileNotFoundException: wrongIPlist.csv (No such file or directory)". The file is there (I wasn't sure which directory it was accessing so it's in both my current client directory and my HDFS home directory), so now I'm wondering if reading a file in parallel is just not allowed in general and that's why I'm getting the error. I'd like each executor to have access to this HashSet (not a huge file, about 3000 IPs) instead of having to do a more expensive JOIN. Any recommendations on a better way to handle this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Building-a-hash-table-from-a-csv-file-using-yarn-cluster-and-giving-it-to-each-executor-tp18850.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org