So, after shuffle at reduce side, are the spills actually stored as map
files?
Yes.
When a reducer receives map output from multiple maps worth fs.inmemorysize.mb
in size, it sorts and spills the data to disk.
If the number of map output data files spilt to disk exceeds io.sort.factor,
additional merge step(s) are performed to reduce the file count to lesser than
that number.
When all map output data is spilt to disk, the reduce tasks starts invoking the
reduce function and passes key-value pairs in sorted order by reading them off
from the above files.
Regards,
Sriguru
-Original Message-
From: Gang Luo [mailto:lgpub...@yahoo.com.cn]
Sent: Thursday, February 04, 2010 1:58 AM
To: common-user@hadoop.apache.org
Subject: Re: sort at reduce side
Thanks for reply, Sriguru.
So, after shuffle at reduce side, are the spills actually stored as map
files?
Why I ask these questions is based on some observations as following. On a
16 nodes cluster, when I do a map join, it takes 3 and a half minutes. When
I do a reduce side join on nearly the same amount of data, it take 8
minutes before map phase complete. I am sure the computation (map function)
will not cause so much difference, the extra 4 minutes time could be only
spent on sorting at map side for reduce side join. While I also notice that
the sort time at reduce side is only 30 sec (I cannot access the online
jobtracker, the 30 sec time is actually the time reduce takes from 33%
completeness to 66% completeness). The number of reduce tasks is much
fewer than that of map tasks, which means each reduce task sort more data
than each map task (I use hash partitioner and data is uniformly
distributed). The only reason I come up with for the big difference
between the sort at map side and reduce side is the different behaviors of
these two sorts.
Anybody has some ideas why the map takes so much time for reduce side join
compared to map side join, and why there is big difference between sort at
map side and reduce side?
P.S. I join a 7.5G file with a 100M file. the sort buffer at reduce is
slightly large than that at map side.
-Gang
- 原始邮件
发件人： Srigurunath Chakravarthi srig...@yahoo-inc.com
收件人： common-user@hadoop.apache.org common-user@hadoop.apache.org
发送日期： 2010/2/3 (周三) 12:50:08 上午
主 题： RE: sort at reduce side
Hi Gang,
kept in map file. If so, in order to efficiently sort the data, reducer
actually only read the index part of each spill (which is a map file) and
sort the keys, instead of reading whole records from disk and sort them.
afaik, no. Reduces always fetches map output data and not indexes (even if
the data is from the local node, where an index may be sufficient).
Regards,
Sriguru
-Original Message-
From: Gang Luo [mailto:lgpub...@yahoo.com.cn]
Sent: Wednesday, February 03, 2010 10:40 AM
To: common-user@hadoop.apache.org
Subject: sort at reduce side
Hi all,
I want to know some more details about the sorting at the reduce side.
The intermediate result generated at the map side is stored as map file
which actually consists of two sub-files, namely index file and data file.
The index file stores the keys and it could point to corresponding record
stored in the data file. What I think is that when intermediate result
(even only part of it for each mapper) is shuffled to reducer, it is still
kept in map file. If so, in order to efficiently sort the data, reducer
actually only read the index part of each spill (which is a map file) and
sort the keys, instead of reading whole records from disk and sort them.
Does reducer actually do as what I expect?
-Gang
___
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/
___
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/
