What is your understanding of "4 hive setups" ? How many maps or reducers are needed for your query is determined by the size of the data you got. You can alter the number of mappers needed for a query by setting the max and min splits. But I am guessing that is not what you want to achieve,
I guess what you are trying to achieve is resource sharing on jobtracker. In other words you want to dedicate X number of resources to a particular kind of work (In your case all of them are hive jobs) If you want to do that way, then please look at hadoop job queues and different schedulers like capacity scheduler. Once you create different job queues, you can use the property mapred.job.queue.name in your hive-site.conf to point to a particular queue created on jobtracker. On Thu, Oct 10, 2013 at 12:19 PM, Garg, Rinku <rinku.g...@fisglobal.com>wrote: > Hi Nitin/Rajesh, > > Thanks for the reply... > > I tried with both the ways as you guys mentioned in the mail trail... > but using that way hive does not fix the number of mappers for a hive query. > > My orginal question is : > > if there are 8 mapping tasks defined in Hadoop configuration, then is it > possible to use 4 hive setups that uses 2 mappers each? > > Please reply. > > Thanks > Rinku Garg > ------------------------------ > *From:* Nitin Pawar [nitinpawar...@gmail.com] > *Sent:* Wednesday, October 09, 2013 6:15 PM > *To:* user@hive.apache.org > *Subject:* Re: Can we reduce/fix number of mappers in Hive > > We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1 -- Very > old versions . > You may want to move to hadoop 1.2.0 and hive-0.11 to have few more > things. > > If you want to limit how many maps a particular job or particular user > can occupy, take a look at hadoop's fair scheduler can capacity schedulers. > If you want change the number of mapper and reducers for a hive query, you > can change the input min and max split sizes as said by Rajesh. > > Here are few of the settings > set mapred.min.split.size=1024000; > set mapred.max.split.size=4096000; > set hive.exec.reducers.max=2048; > > may be once you tell clearly what you want achieve we will be able to > help better. > > > > On Wed, Oct 9, 2013 at 6:09 PM, Rajesh Balamohan < > rajesh.balamo...@gmail.com> wrote: > >> Did you try adjusting fileinputformat. Min and max size parameters? >> On Oct 9, 2013 5:51 PM, "Garg, Rinku" <rinku.g...@fisglobal.com> wrote: >> >>> Hi All**** >>> >>> We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. We have >>> the following query:**** >>> >>> ** ** >>> >>> Is there any option in Hive where mappers can be reduced/fixed?**** >>> >>> ** ** >>> >>> For example if there are 8 mapping tasks defined in Hadoop >>> configuration, then is it possible to use 4 hive setups that uses 2 mappers >>> each?**** >>> >>> ** ** >>> >>> Thanks & Regards,**** >>> >>> Rinku Garg >>> >>> >>> **** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> _____________ >>> The information contained in this message is proprietary and/or >>> confidential. If you are not the intended recipient, please: (i) delete the >>> message and all copies; (ii) do not disclose, distribute or use the message >>> in any manner; and (iii) notify the sender immediately. In addition, please >>> be aware that any message addressed to our domain is subject to archiving >>> and review by persons other than the intended recipient. Thank you. >>> >> > > > -- > Nitin Pawar > _____________ > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) delete the > message and all copies; (ii) do not disclose, distribute or use the message > in any manner; and (iii) notify the sender immediately. In addition, please > be aware that any message addressed to our domain is subject to archiving > and review by persons other than the intended recipient. Thank you. > -- Nitin Pawar
