Hi Jothi, thanks for your answer.

I have:

Average time taken by Map tasks: 37sec

Average time taken by Shuffle: 35mins, 12sec

Average time taken by Reduce tasks: 1sec

--
./david



2009/5/29 Jothi Padmanabhan <[email protected]>:
> Hi David,
>
> If you go to JobTrackerHistory and then click on this job and then do
> Analyse This Job, you should be able to get the split up timings for the
> individual phases of the map and reduce tasks, including the average, best
> and worst times. Could you provide those numbers so that we can get a better
> idea of how the job progressed.
>
> Jothi
>
>
> On 5/28/09 10:11 PM, "David Batista" <[email protected]> wrote:
>
>> Hi everyone,
>>
>> I'm processing XML files, around 500MB each with several documents,
>> for the map() function I pass a document from the XML file, which
>> takes some time to process depending on the size - I'm applying NER to
>> texts.
>>
>> Each document has a unique identifier, so I'm using that identifier as
>> a key and the results of parsing the document in one string as the
>> output:
>>
>> so at the end of  the map function():
>> output.collect( new Text(identifier), new Text(outputString));
>>
>> usually the outputString is around 1k-5k size
>>
>> reduce():
>> public void reduce(Text key, Iterator<Text> values,
>> OutputCollector<Text, Text> output, Reporter reporter) {
>> while (values.hasNext()) {
>> Text text = values.next();
>> try {
>> output.collect(key, text);
>> } catch (IOException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>> }
>> }
>>
>> I did a test using only 1 machine with 8 cores, and only 1 XML file,
>> it took around 3 hours to process all maps and  ~12hours for the
>> reduces!
>>
>> the XML file has 139 945 documents
>>
>> I set the jobconf for 1000 maps() and 200 reduces()
>>
>> I did took a look at graphs on the web interface during the reduce
>> phase, and indeed its the copy phase that's taking much of the time,
>> the sort and reduce phase are done almost instantly.
>>
>> Why does the copy phase takes so long? I understand that the copies
>> are made using HTTP, and the data was in really small chunks 1k-5k
>> size, but even so, being everything in the same physical machine
>> should have been faster, no?
>>
>> Any suggestions on what might be causing the copies in reduce to take so 
>> long?
>> --
>> ./david
>
>

Reply via email to