Hi Harsh, You are right, I went through my combiner code again carefully, and found that there is a bug in my code. For short, I add a type for each value, and check the type before processing. The type before and after multiple combiner iteration is not the same, so...
I found the root cause of the issue, thanks for your help, Harsh. ________________________________________ From: Xin Jing [[email protected]] Sent: Friday, October 28, 2011 2:43 PM To: [email protected] Subject: RE: problem when using combiner and MultipleOutputFormat OK, I will prepare my code to show how it works. Here another question, my combiner DOES output some records, if it the reason as you said, my combiner's behavior is wrong when run one time and multiple times, the reason that reduce cannot receive input is because of the wrong data format of the combiner? I attach the job result, please take a look. ________________________________________ From: Harsh J [[email protected]] Sent: Friday, October 28, 2011 2:33 PM To: [email protected] Subject: Re: problem when using combiner and MultipleOutputFormat Xin, I guess am now confused. I was thinking that your combiner was a reused reducer, and that was probably eating away all the outputs when run (Combiners run 0…N times (yes, 0 is possible), so that may explain your low vs. high data volume difference). I'm sure the issue is with your combiner implementation, but I'd have to see some mock code to tell you more, as am unsure on what you are attempting to do in there. On Fri, Oct 28, 2011 at 11:52 AM, Xin Jing <[email protected]> wrote: > Thanks for your answer. > > I am using different combiner and reducer. As I have said in previous mail, > when the data set is small, it works fine and the result is correct. I can > tell the functionality of my job is ok, right? > > I cannot understand what do you mean by ' Do not output to files directly > from your combiner', could you give me more hints? I combiner code, I am > using output.collect() to output my result, do I misuse it? > ________________________________________ > From: Harsh J [[email protected]] > Sent: Friday, October 28, 2011 2:11 PM > To: [email protected] > Subject: Re: problem when using combiner and MultipleOutputFormat > > Xin, > > You probably just need to write a special Combiner class instead of > reusing your Reducer class for combiner purposes. In an MR job, you > need to specifically guarantee that the combiner outputs the same type > of K-V pairs as the reducer's input. Do not output to files directly > from your combiner, and that is why you'd need a different class impl. > performing the optimization. > > On Fri, Oct 28, 2011 at 10:04 AM, Xin Jing <[email protected]> wrote: >> >> Hi all, >> I am currently encountering a tough problem, my job use MultipleOutputFormat >> to output result into different folder, and I have to use a combiner to >> enhance performance. In this situation, reduce does not work, reduce cannot >> receive any data. I searched this issue and found a related >> topic, >> http://lucene.472066.n3.nabble.com/Combiner-and-MultipleOutputs-in-Mapreduce-td1640503.html >> , >> but not get clear what the solution is really. Seems it is the constraint of >> hadoop framework? >> I found a interesting phenomenon, when I limit the map input record to a >> small number (such as 10000), the reduce is ok, it can receive data and the >> result is correct. But when the input is over a million record, the reduce >> receive nothing. I guess the reason is the combiner only be called once when >> data is small while combiner be called multiple time when data is huge. >> To summary, how can I make combiner feasible while using >> MultipleOutputFormat? Any solution or suggestion is welcome. >> >> Thanks >> > > > > -- > Harsh J > > > -- Harsh J
