Thanks Sudhakara for your reply. So if number of mappers depends on the data size ,maybe the best way to do my experiments is to increase the number of reducers based on the number of estimated blocks in data file.Actually i want to know how response time is changed by changing the number of mappers and reducers. Any idea about the way of doing this kind of experiment?
Samaneh On Sun, Apr 7, 2013 at 6:29 PM, sudhakara st <[email protected]> wrote: > Hi Samaneh, > > The number of map tasks for a given job is driven by the number > of input splits in the input data. ideally in default configurations each > input split(for a block) a map task is spawned. So your 2.5G of data > contains 44 blocks, therefore you jobs taking 44 map task. At minimum, with > FileInputFormat derivatives, job will have at least one map per file and > can have multiple maps per file if they extend beyond a single block(file > size is more that block size). The *mapred.map.tasks* parameter is just a > hint to the InputFormat for the number of maps. its does not have any > effect if the number blocks in the input date more then specified value. It > not possible to specify number mapper need run for a job. But it possible > to explicitly specify number reduce can run for a job by using * > mapred.reduce.tasks* property. > > The replication factor in not related in any to number of mapper and > reducer. > > > > > > > > On Sun, Apr 7, 2013 at 7:38 PM, Samaneh Shokuhi > <[email protected]>wrote: > > > Hi All, > > I am doing some experiments by running WordCount example on hadoop. > > I have a cluster with 7 nodes .I want to run WordCount example with > > 3mappers and 3 reducers and compare the response time with another > > experiments when number of mappers and reducers increased to 6 and 12 and > > so on. > > For first experiment i set number of the mappers and reducer to 3 in > > wordCount example source code .and also set the number of replications > to 3 > > in hadoop configurations.Also the maximum number of tasks per node is > set > > to 1 . > > But when i run the sample with a big data like 2.5 G ,i can see 44 map > > tasks and 3 reduce tasks are running !! > > > > What parameters do i need to set to have like (3Mappers,3 Reducers), > > (6M,6R) and (12M,12R) and as i mentioned i have a cluster with 1 namenode > > and 6 datanodes. > > Is number of replications related to the number of mappers and reducers > ?! > > Regards, > > Samaneh > > > > > > -- > > Regards, > ..... Sudhakara.st >
