Question on MapReduce
Hi, I am a newbie on Hadoop and have a quick question on optimal compute vs. storage resources for MapReduce. If I have a multiprocessor node with 4 processors, will Hadoop schedule higher number of Map or Reduce tasks on the system than on a uni-processor system? In other words, does Hadoop detect denser systems and schedule denser tasks on multiprocessor systems? If yes, will that imply that it makes sense to attach higher capacity storage to store more number of blocks on systems with dense compute? Any insights will be very useful. Thanks, Satheesh
Re: Question on MapReduce
Thanks, Leo. What is the config of a typical data node in a Hadoop cluster - cores, storage capacity, and connectivity (SATA?).? How many tasktrackers scheduled per core in general? Is there a best practices guide somewhere? Thanks, Satheesh On Fri, May 11, 2012 at 10:48 AM, Leo Leung wrote: > Nope, you must tune the config on that specific super node to have more > M/R slots (this is for 1.0.x) > This does not mean the JobTracker will be eager to stuff that super node > with all the M/R jobs at hand. > > It still goes through the scheduler, Capacity Scheduler is most likely > what you have. (check your config) > > IMO, If the data locality is not going to be there, your cluster is going > to suffer from Network I/O. > > > -Original Message- > From: Satheesh Kumar [mailto:nks...@gmail.com] > Sent: Friday, May 11, 2012 9:51 AM > To: common-user@hadoop.apache.org > Subject: Question on MapReduce > > Hi, > > I am a newbie on Hadoop and have a quick question on optimal compute vs. > storage resources for MapReduce. > > If I have a multiprocessor node with 4 processors, will Hadoop schedule > higher number of Map or Reduce tasks on the system than on a uni-processor > system? In other words, does Hadoop detect denser systems and schedule > denser tasks on multiprocessor systems? > > If yes, will that imply that it makes sense to attach higher capacity > storage to store more number of blocks on systems with dense compute? > > Any insights will be very useful. > > Thanks, > Satheesh >
Re: Question on MapReduce
Couple of more quick questions 1. In a Hadoop node with DAS, what is the typical storage utilization? In other words, for a given total data size, how much capacity should we plan per node given that compute is not a huge bottleneck? 2. What is the expected storage throughput from DAS to the compute on the same node assuming SATA interface? Thanks, Satheesh On Fri, May 11, 2012 at 12:58 PM, Leo Leung wrote: > > This maybe dated materials. > > Cloudera and HDP folks please correct with updates :) > > > http://www.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/ > http://www.cloudera.com/blog/2010/08/hadoophbase-capacity-planning/ > > > http://hortonworks.com/blog/best-practices-for-selecting-apache-hadoop-hardware/ > > Hope this helps. > > > > -Original Message- > From: Satheesh Kumar [mailto:nks...@gmail.com] > Sent: Friday, May 11, 2012 12:48 PM > To: common-user@hadoop.apache.org > Subject: Re: Question on MapReduce > > Thanks, Leo. What is the config of a typical data node in a Hadoop cluster > - cores, storage capacity, and connectivity (SATA?).? How many > tasktrackers scheduled per core in general? > > Is there a best practices guide somewhere? > > Thanks, > Satheesh > > On Fri, May 11, 2012 at 10:48 AM, Leo Leung wrote: > > > Nope, you must tune the config on that specific super node to have > > more M/R slots (this is for 1.0.x) This does not mean the JobTracker > > will be eager to stuff that super node with all the M/R jobs at hand. > > > > It still goes through the scheduler, Capacity Scheduler is most > > likely what you have. (check your config) > > > > IMO, If the data locality is not going to be there, your cluster is > > going to suffer from Network I/O. > > > > > > -Original Message- > > From: Satheesh Kumar [mailto:nks...@gmail.com] > > Sent: Friday, May 11, 2012 9:51 AM > > To: common-user@hadoop.apache.org > > Subject: Question on MapReduce > > > > Hi, > > > > I am a newbie on Hadoop and have a quick question on optimal compute vs. > > storage resources for MapReduce. > > > > If I have a multiprocessor node with 4 processors, will Hadoop > > schedule higher number of Map or Reduce tasks on the system than on a > > uni-processor system? In other words, does Hadoop detect denser > > systems and schedule denser tasks on multiprocessor systems? > > > > If yes, will that imply that it makes sense to attach higher capacity > > storage to store more number of blocks on systems with dense compute? > > > > Any insights will be very useful. > > > > Thanks, > > Satheesh > > >
MapReduce shuffle question
Team, can someone please clarify the following question? In the map phase, the map output is written to the local disk. And in the shuffle phase, the map output partitions are transferred to reduce nodes using http. So, my question is assuming there are no spills (data set is small enough to accommodate this), will the map output be transferred directly from memory to the reduce nodes using http without a disk access to write the map output? Or, is the map output always flushed to the disk before transferred to reduce nodes? Appreciate the help. Thanks, Satheesh
Re: 答复: MapReduce shuffle question
Thank you, Liyin, On Fri, Aug 3, 2012 at 7:33 AM, 梁李印 wrote: > When a map task is done, its output is always flushed to the disk and > merged > to one file. > The benefit is that if the reducer is failed, the map need not to re-run. > > Liyin Liang > > -邮件原件----- > 发件人: Satheesh Kumar [mailto:nks...@gmail.com] > 发送时间: 2012年8月3日 21:23 > 收件人: common-user@hadoop.apache.org > 主题: MapReduce shuffle question > > Team, can someone please clarify the following question? > > In the map phase, the map output is written to the local disk. And in the > shuffle phase, the map output partitions are transferred to reduce nodes > using http. So, my question is assuming there are no spills (data set is > small enough to accommodate this), will the map output be transferred > directly from memory to the reduce nodes using http without a disk access > to write the map output? Or, is the map output always flushed to the disk > before transferred to reduce nodes? > > Appreciate the help. > > Thanks, > Satheesh > >
Re: 答复: MapReduce shuffle question
Thank you. One more follow up question: Are there any optimizations to run map and reduces on the same nodes so that data is not transported across the network? Generally, how often and what % of map output is actually transferred over the network to reduce nodes? Thanks, Satheesh On Fri, Aug 3, 2012 at 7:33 AM, 梁李印 wrote: > When a map task is done, its output is always flushed to the disk and > merged > to one file. > The benefit is that if the reducer is failed, the map need not to re-run. > > Liyin Liang > > -邮件原件----- > 发件人: Satheesh Kumar [mailto:nks...@gmail.com] > 发送时间: 2012年8月3日 21:23 > 收件人: common-user@hadoop.apache.org > 主题: MapReduce shuffle question > > Team, can someone please clarify the following question? > > In the map phase, the map output is written to the local disk. And in the > shuffle phase, the map output partitions are transferred to reduce nodes > using http. So, my question is assuming there are no spills (data set is > small enough to accommodate this), will the map output be transferred > directly from memory to the reduce nodes using http without a disk access > to write the map output? Or, is the map output always flushed to the disk > before transferred to reduce nodes? > > Appreciate the help. > > Thanks, > Satheesh > >
Re: 答复: 答复: MapReduce shuffle question
Thanks, again, Liyin. On Sat, Aug 4, 2012 at 6:59 AM, 梁李印 wrote: > The optimization you mentioned is reduce-task locality-aware. > Unfortunately, > the current scheduler doesn't consider the reduce task's data locality. So > a > reduce task can be scheduled to any node with free slots. > The following jira is discussing this problem: > https://issues.apache.org/jira/browse/MAPREDUCE-2038 > > Liyin Liang > -邮件原件- > 发件人: Satheesh Kumar [mailto:nks...@gmail.com] > 发送时间: 2012年8月4日 1:47 > 收件人: common-user@hadoop.apache.org > 主题: Re: 答复: MapReduce shuffle question > > Thank you. One more follow up question: > > Are there any optimizations to run map and reduces on the same nodes so > that data is not transported across the network? Generally, how often and > what % of map output is actually transferred over the network to reduce > nodes? > > Thanks, > Satheesh > > On Fri, Aug 3, 2012 at 7:33 AM, 梁李印 wrote: > > > When a map task is done, its output is always flushed to the disk and > > merged > > to one file. > > The benefit is that if the reducer is failed, the map need not to re-run. > > > > Liyin Liang > > > > -邮件原件- > > 发件人: Satheesh Kumar [mailto:nks...@gmail.com] > > 发送时间: 2012年8月3日 21:23 > > 收件人: common-user@hadoop.apache.org > > 主题: MapReduce shuffle question > > > > Team, can someone please clarify the following question? > > > > In the map phase, the map output is written to the local disk. And in the > > shuffle phase, the map output partitions are transferred to reduce nodes > > using http. So, my question is assuming there are no spills (data set is > > small enough to accommodate this), will the map output be transferred > > directly from memory to the reduce nodes using http without a disk access > > to write the map output? Or, is the map output always flushed to the disk > > before transferred to reduce nodes? > > > > Appreciate the help. > > > > Thanks, > > Satheesh > > > > > >
