Foreach() runs in parallel across the cluster, like map, flatMap, etc. You'll only run into problems if you call collect(), which brings the entire RDD into memory in the driver program.
Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Tue, Apr 7, 2015 at 3:50 AM, Jeetendra Gangele <[email protected]> wrote: > Lets say I follow below approach and I got RddPair with huge size .. which > can not fit into one machine ... what to run foreach on this RDD? > > On 7 April 2015 at 04:25, Jeetendra Gangele <[email protected]> wrote: > >> >> >> On 7 April 2015 at 04:03, Dean Wampler <[email protected]> wrote: >> >>> >>> On Mon, Apr 6, 2015 at 6:20 PM, Jeetendra Gangele <[email protected]> >>> wrote: >>> >>>> Thanks a lot.That means Spark does not support the nested RDD? >>>> if I pass the javaSparkContext that also wont work. I mean passing >>>> SparkContext not possible since its not serializable >>>> >>>> That's right. RDD don't nest and SparkContexts aren't serializable. >>> >>> >>>> i have a requirement where I will get JavaRDD<VendorRecord> matchRdd >>>> and I need to return the postential matches for this record from Hbase. so >>>> for each field of VendorRecord I have to do following >>>> >>>> 1. query Hbase to get the list of potential record in RDD >>>> 2. run logistic regression on RDD return from steps 1 and each element >>>> of the passed matchRdd. >>>> >>>> If I understand you correctly, each VectorRecord could correspond to >>> 0-to-N records in HBase, which you need to fetch, true? >>> >> yes thats correct each Vendorrecord corresponds to 0 to N matches >> >> >>> If so, you could use the RDD flatMap method, which takes a function a >>> that accepts each record, then returns a sequence of 0-to-N new records of >>> some other type, like your HBase records. However, running an HBase query >>> for each VendorRecord could be expensive. If you can turn this into a range >>> query or something like that, it would help. I haven't used HBase much, so >>> I don't have good advice on optimizing this, if necessary. >>> >>> Alternatively, can you do some sort of join on the VendorRecord RDD and >>> an RDD of query results from HBase? >>> >> Join will give too big result RDD of query result is returning around >> 10000 for each record and i have 2 millions to process so it will be huge >> to have this. 2 m*10000 big number >> >>> >>> For #2, it sounds like you need flatMap to return records that combine >>> the input VendorRecords and fields pulled from HBase. >>> >>> Whatever you can do to make this work like "table scans" and joins will >>> probably be most efficient. >>> >>> dean >>> >>>> >>>> >>>> >>>> On 7 April 2015 at 03:33, Dean Wampler <[email protected]> wrote: >>>> >>>>> The "log" instance won't be serializable, because it will have a file >>>>> handle to write to. Try defining another static method outside >>>>> matchAndMerge that encapsulates the call to log.error. >>>>> CompanyMatcherHelper >>>>> might not be serializable either, but you didn't provide it. If it holds a >>>>> database connection, same problem. >>>>> >>>>> You can't suppress the warning because it's actually an error. The >>>>> VoidFunction can't be serialized to send it over the cluster's network. >>>>> >>>>> dean >>>>> >>>>> Dean Wampler, Ph.D. >>>>> Author: Programming Scala, 2nd Edition >>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) >>>>> Typesafe <http://typesafe.com> >>>>> @deanwampler <http://twitter.com/deanwampler> >>>>> http://polyglotprogramming.com >>>>> >>>>> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele < >>>>> [email protected]> wrote: >>>>> >>>>>> In this code in foreach I am getting task not serialized exception >>>>>> >>>>>> >>>>>> @SuppressWarnings("serial") >>>>>> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd, >>>>>> final JavaSparkContext jsc) throws IOException{ >>>>>> log.info("Company matcher started"); >>>>>> //final JavaSparkContext jsc = getSparkContext(); >>>>>> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){ >>>>>> @Override >>>>>> public void call(VendorRecord t) throws Exception { >>>>>> if(t !=null){ >>>>>> try{ >>>>>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t); >>>>>> } catch (Exception e) { >>>>>> log.error("ERROR while running Matcher for company " + >>>>>> t.getCompanyId(), e); >>>>>> } >>>>>> } >>>>>> } >>>>>> }); >>>>>> >>>>>> } >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >> >> >> >> > > > - >
