Hi Jothi, thanks for your answer. I have:
Average time taken by Map tasks: 37sec Average time taken by Shuffle: 35mins, 12sec Average time taken by Reduce tasks: 1sec -- ./david 2009/5/29 Jothi Padmanabhan <[email protected]>: > Hi David, > > If you go to JobTrackerHistory and then click on this job and then do > Analyse This Job, you should be able to get the split up timings for the > individual phases of the map and reduce tasks, including the average, best > and worst times. Could you provide those numbers so that we can get a better > idea of how the job progressed. > > Jothi > > > On 5/28/09 10:11 PM, "David Batista" <[email protected]> wrote: > >> Hi everyone, >> >> I'm processing XML files, around 500MB each with several documents, >> for the map() function I pass a document from the XML file, which >> takes some time to process depending on the size - I'm applying NER to >> texts. >> >> Each document has a unique identifier, so I'm using that identifier as >> a key and the results of parsing the document in one string as the >> output: >> >> so at the end of the map function(): >> output.collect( new Text(identifier), new Text(outputString)); >> >> usually the outputString is around 1k-5k size >> >> reduce(): >> public void reduce(Text key, Iterator<Text> values, >> OutputCollector<Text, Text> output, Reporter reporter) { >> while (values.hasNext()) { >> Text text = values.next(); >> try { >> output.collect(key, text); >> } catch (IOException e) { >> // TODO Auto-generated catch block >> e.printStackTrace(); >> } >> } >> } >> >> I did a test using only 1 machine with 8 cores, and only 1 XML file, >> it took around 3 hours to process all maps and ~12hours for the >> reduces! >> >> the XML file has 139 945 documents >> >> I set the jobconf for 1000 maps() and 200 reduces() >> >> I did took a look at graphs on the web interface during the reduce >> phase, and indeed its the copy phase that's taking much of the time, >> the sort and reduce phase are done almost instantly. >> >> Why does the copy phase takes so long? I understand that the copies >> are made using HTTP, and the data was in really small chunks 1k-5k >> size, but even so, being everything in the same physical machine >> should have been faster, no? >> >> Any suggestions on what might be causing the copies in reduce to take so >> long? >> -- >> ./david > >
