The "log" instance won't be serializable, because it will have a file handle to write to. Try defining another static method outside matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper might not be serializable either, but you didn't provide it. If it holds a database connection, same problem.
You can't suppress the warning because it's actually an error. The VoidFunction can't be serialized to send it over the cluster's network. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <gangele...@gmail.com> wrote: > In this code in foreach I am getting task not serialized exception > > > @SuppressWarnings("serial") > public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd, final > JavaSparkContext jsc) throws IOException{ > log.info("Company matcher started"); > //final JavaSparkContext jsc = getSparkContext(); > matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){ > @Override > public void call(VendorRecord t) throws Exception { > if(t !=null){ > try{ > CompanyMatcherHelper.UpdateMatchedRecord(jsc,t); > } catch (Exception e) { > log.error("ERROR while running Matcher for company " + t.getCompanyId(), > e); > } > } > } > }); > > } >
