Re: Create hashmap using two RDD's

Hi Richard, Requirement is to get latest records using a key i think hash map is a good choice for this task. As of now data comes from third party and we are not sure what's the latest record is so hash map is chosen. Is there anything better than hash map please let me know. Thanks Sri Sent

Re: Create hashmap using two RDD's

Got it ..., created hashmap and saved it to file please follow below steps .. val QuoteRDD=quotefile.map(x => x.split("\\|")). filter(line => line(0).contains("1017")). map(x => ((x(5)+x(4)) , (x(5),x(4),x(1) , if (x(15) =="B") ( {if (x(25) == "") x(9) else

Re: Create hashmap using two RDD's

Do you need the HashMap for anything else besides writing out to a file? If not, there is really no need to create one at all. You could just keep everything as RDDs. On Oct 10, 2015 11:31 AM, "kali.tumm...@gmail.com" wrote: > Got it ..., created hashmap and saved it to

Re: Create hashmap using two RDD's

You should be able to achieve what you're looking for by using foldByKey to find the latest record for each key. If you're relying on the order elements within the file to determine which ones are the "latest" (rather than sorting by some field within the file itself), call zipWithIndex first to

Re: Create hashmap using two RDD's

Thanks Richard , will give a try tomorrow... Thanks Sri Sent from my iPhone > On 10 Oct 2015, at 19:15, Richard Eggert wrote: > > You should be able to achieve what you're looking for by using foldByKey to > find the latest record for each key. If you're relying on

Re: Create hashmap using two RDD's

Hi All, I changed my way of approach now I am bale to load data into MAP and get data out using get command. val QuoteRDD=quotefile.map(x => x.split("\\|")). filter(line => line(0).contains("1017")). map(x => ((x(5)+x(4)) , (x(5),x(4),x(1) , if (x(15) =="B") if

Create hashmap using two RDD's

Hi all, I am trying to create a hashmap using two rdd, but having issues key not found do I need to convert RDD to list first ? 1) rdd has key data 2) rdd has value data Key Rdd:- val quotekey=file.map(x => x.split("\\|")).filter(line => line(0).contains("1017")).map(x => x(5)+x(4))