Hi, I ran into a very similar situation quite some time back and had then encountered this : http://issues.apache.org/jira/browse/HADOOP-475 After speaking to a few Hadoop folks, they had said complete cloning was not a straightforward option for some optimization reasons. There were a few things I tried , to run this in a single MR job emitting <k,v> from mapper one more time with some tagging info ( this bumped up S&S phase by quite a lot ); run a map only successor job etc. But keeping records in memory and writing to disk after certain threshold amount worked pretty well for me ( all this on Hadoop 0.17.2 ) Anyways, they seem to have resolved it in next Hadoop release.
Amogh On 1/12/10 10:29 PM, "Stephen Watt" <sw...@us.ibm.com> wrote: The Key Value pairs coming into my Reducer are as Follows KEY(Text) VALUE(IntWritable) A 11 A 9 B 2 B 3 I want my reducer to sum the Values for each input key and then output the key with a Text Value containing the original value and the sum. KEY(Text) VALUE(Text) A 11 20 A 9 20 B 2 5 B 3 5 Here is the issue : In the reducer, I am iterating through the values for each using values.iterator() and storing the total amount in a variable. Then I am TRYING to iterate through the keys again, except this time, writing the new value (A, new Text("11 20") in the output collector to create the Value structure displayed in the example above. This fails because it appears I can only iterate through the values for each key ONCE. I know this because additional attempts to get new iterators from the context or the Iterable type thats passed into the reducer always return false on the initial hasNext(). I have to iterate through it twice because the first time I have to sum the values and the second time I need to write the write the initial (11) value and the sum(20) as I need both values as part of a calculation in the next job. Any ideas on how to do this ? Kind regards Steve Watt