Re: Set the number of mapper in hive
Thank you very much. I will check on that. On Fri, Dec 7, 2012 at 5:48 PM, Nitin Pawar wrote: > Yes > On Dec 7, 2012 3:09 PM, "Philips Kokoh Prasetyo" > wrote: > >> Hi Nitin, >> >> Thanks for the reply. >> Do you mean using fair scheduler to separate job queue? >> http://hadoop.apache.org/docs/r0.20.2/fair_scheduler.html >> >> >> Regards, >> Philips >> >> >> On Fri, Dec 7, 2012 at 5:15 PM, Nitin Pawar wrote: >> >>> ways to handle this >>> 1) create separate job queues for hive and hbase users on jt and >>> allocate resources according to your needs >>> 2) you can not actually limit how many maps can be launched as its >>> decided on run time by looking at split size. If you want less number of >>> maps to be launched then increase split size...also keep in mind you will >>> need larger memory for maps >>> On Dec 7, 2012 2:32 PM, "Philips Kokoh Prasetyo" < >>> philipsko...@gmail.com> wrote: >>> Hi everyone, My cluster also runs HBase for real time processing. Hive query (on a big table) occupies all the map tasks so that the other service cannot run properly. Does anyone know how to limit the number of running map in hive? I see mapred.reduce.tasks in the configuration properties, but I don't see the configuration for mapper. Thank you. Regards, Philips >>> >>
Re: Set the number of mapper in hive
Yes On Dec 7, 2012 3:09 PM, "Philips Kokoh Prasetyo" wrote: > Hi Nitin, > > Thanks for the reply. > Do you mean using fair scheduler to separate job queue? > http://hadoop.apache.org/docs/r0.20.2/fair_scheduler.html > > > Regards, > Philips > > > On Fri, Dec 7, 2012 at 5:15 PM, Nitin Pawar wrote: > >> ways to handle this >> 1) create separate job queues for hive and hbase users on jt and allocate >> resources according to your needs >> 2) you can not actually limit how many maps can be launched as its >> decided on run time by looking at split size. If you want less number of >> maps to be launched then increase split size...also keep in mind you will >> need larger memory for maps >> On Dec 7, 2012 2:32 PM, "Philips Kokoh Prasetyo" >> wrote: >> >>> Hi everyone, >>> >>> My cluster also runs HBase for real time processing. Hive query (on a >>> big table) occupies all the map tasks so that the other service cannot run >>> properly. >>> Does anyone know how to limit the number of running map in hive? >>> I see mapred.reduce.tasks in the configuration properties, but I don't >>> see the configuration for mapper. >>> >>> Thank you. >>> >>> Regards, >>> Philips >>> >> >
Re: Set the number of mapper in hive
Hi Nitin, Thanks for the reply. Do you mean using fair scheduler to separate job queue? http://hadoop.apache.org/docs/r0.20.2/fair_scheduler.html Regards, Philips On Fri, Dec 7, 2012 at 5:15 PM, Nitin Pawar wrote: > ways to handle this > 1) create separate job queues for hive and hbase users on jt and allocate > resources according to your needs > 2) you can not actually limit how many maps can be launched as its decided > on run time by looking at split size. If you want less number of maps to be > launched then increase split size...also keep in mind you will need larger > memory for maps > On Dec 7, 2012 2:32 PM, "Philips Kokoh Prasetyo" > wrote: > >> Hi everyone, >> >> My cluster also runs HBase for real time processing. Hive query (on a big >> table) occupies all the map tasks so that the other service cannot run >> properly. >> Does anyone know how to limit the number of running map in hive? >> I see mapred.reduce.tasks in the configuration properties, but I don't >> see the configuration for mapper. >> >> Thank you. >> >> Regards, >> Philips >> >
Re: Set the number of mapper in hive
ways to handle this 1) create separate job queues for hive and hbase users on jt and allocate resources according to your needs 2) you can not actually limit how many maps can be launched as its decided on run time by looking at split size. If you want less number of maps to be launched then increase split size...also keep in mind you will need larger memory for maps On Dec 7, 2012 2:32 PM, "Philips Kokoh Prasetyo" wrote: > Hi everyone, > > My cluster also runs HBase for real time processing. Hive query (on a big > table) occupies all the map tasks so that the other service cannot run > properly. > Does anyone know how to limit the number of running map in hive? > I see mapred.reduce.tasks in the configuration properties, but I don't see > the configuration for mapper. > > Thank you. > > Regards, > Philips >
Set the number of mapper in hive
Hi everyone, My cluster also runs HBase for real time processing. Hive query (on a big table) occupies all the map tasks so that the other service cannot run properly. Does anyone know how to limit the number of running map in hive? I see mapred.reduce.tasks in the configuration properties, but I don't see the configuration for mapper. Thank you. Regards, Philips
