Jothi , Do have any idea , how thease speeds are calculated , I mean some mathematical expressions or stuff.
On Mon, Aug 24, 2009 at 12:54 PM, Jothi Padmanabhan <[email protected]>wrote: > The transfer rate is a little misleading. The timer for this calculation > starts when the reducer itself starts and so includes the time spent by the > reducer waiting for maps to complete. So, the speed shown when shuffling > the > first few maps might be totally misleading, it does not necessarily reflect > the network speed. You should be able to see more reasonable numbers > towards > the end of shuffle. > > Jothi > > > On 8/24/09 12:19 PM, "yang song" <[email protected]> wrote: > > > Hello, everyone > > > > When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find > that > > the copy phrase will last for a long long time. From WebUI, the message > > "reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer > speed > > is just 0.01 MB/s. Does it a regular value? How can I solve it? > > > > Thank you! > > > > P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of > JT > > is 6G while the others are default settings. > >
