For example: rows.reduceByKey(reduceKeyMapFunction)
reduceKeyMapFunction(log1: Map[String, Long], log2: Map[String, Long]): Map[String,Long] = { val bcast = broadcastv.value val countFields = dbh.getCountFields val aggs: Map[String, Long] = Map() countFields.foreach { f => val valueSum = aggLog1(f) + aggLog2(f) aggs ++ Map(f -> valueSum) } aggs } I can't pass broadcast to the reduceKeyMapFunction. I create the broadcast variable (broadcastv) in the driver but I fear it will not be initialized on the workers where the reduceKeyMapFunction runs. I've tried this but when accessing broadcastv a NPE is thrown. I can't pass it to the reduceKeyMapFunction because it can only accept two params (log1, log2). Any ideas? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Access-broadcast-variable-from-within-function-passed-to-reduceByKey-tp28082.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org