Re: Throttling hive queries

2009-12-19 Thread Zheng Shao
>>> ---------- >>> *From:* Todd Lipcon [mailto:t...@cloudera.com] >>> *Sent:* Thursday, December 17, 2009 4:24 PM >>> >>> *To:* hive-user@hadoop.apache.org >>> *Subject:* Re: Throttling hive queries >>> >>> Hi Sagi, &

Re: Throttling hive queries

2009-12-18 Thread Ryan LeCompte
e Tech Lead & Architect | Work: 650-616-6575 | >> Cell: 718-930-7947 >> >> >> -- >> *From:* Todd Lipcon [mailto:t...@cloudera.com] >> *Sent:* Thursday, December 17, 2009 4:24 PM >> >> *To:* hive-user@hadoop.apache.org >>

Re: Throttling hive queries

2009-12-18 Thread Todd Lipcon
gt; *To:* hive-user@hadoop.apache.org > *Subject:* Re: Throttling hive queries > > Hi Sagi, > > Any chance you're running on a directory that has 614 small files? > > -Todd > > On Thu, Dec 17, 2009 at 2:30 PM, Sagi, Lee wrote: > >>Todd, Here is the

RE: Throttling hive queries

2009-12-18 Thread Sagi, Lee
dd Lipcon [mailto:t...@cloudera.com] Sent: Thursday, December 17, 2009 4:24 PM To: hive-user@hadoop.apache.org Subject: Re: Throttling hive queries Hi Sagi, Any chance you're running on a directory that has 614 small files? -Todd On Thu, Dec 17, 2009 at 2:30 PM, Sagi, Lee wrote:

Re: Throttling hive queries

2009-12-17 Thread Todd Lipcon
Tech Lead & Architect | Work: 650-616-6575 | > Cell: 718-930-7947 > > > -- > *From:* Todd Lipcon [mailto:t...@cloudera.com] > *Sent:* Thursday, December 17, 2009 12:18 PM > > *To:* hive-user@hadoop.apache.org > *Subject:* Re: Throttling hive queries > > Hi L

RE: Throttling hive queries

2009-12-17 Thread Sagi, Lee
pcon [mailto:t...@cloudera.com] Sent: Thursday, December 17, 2009 12:18 PM To: hive-user@hadoop.apache.org Subject: Re: Throttling hive queries Hi Lee, The MapReduce framework in general makes it hard for you assign fewer mappers than there are blocks in the input data, when using FileInputFormat. Is

Re: Throttling hive queries

2009-12-17 Thread Todd Lipcon
#x27;2009121700') OR > (PRSC.date_key >= '2009121600' AND >PRSC.date_key < '2009121700') > > > Lee Sagi | Data Warehouse Tech Lead & Architect | Work: 650-616-6575 | > Cell: 718-930-7947 > > -Original Message- > From: Edward Capri

RE: Throttling hive queries

2009-12-17 Thread Sagi, Lee
PRSC.date_key < '2009121700') Lee Sagi | Data Warehouse Tech Lead & Architect | Work: 650-616-6575 | Cell: 718-930-7947 -Original Message- From: Edward Capriolo [mailto:edlinuxg...@gmail.com] Sent: Thursday, December 17, 2009 11:03 AM To: hive-user@hadoop.apache.org

RE: Throttling hive queries

2009-12-17 Thread Sagi, Lee
0.00% 10 As you can see the "Map Total" is 657. Lee Sagi | Data Warehouse Tech Lead & Architect | Work: 650-616-6575 | Cell: 718-930-7947 -Original Message- From: Edward Capriolo [mailto:edlinuxg...@gmail.com] Sent: Thursday, December 17, 2009 11:03 AM T

Re: Throttling hive queries

2009-12-17 Thread Edward Capriolo
You should be able hive > set mapred.map.tasks=1000 hive > set mapred.reduce.tasks=5 In some cases mappers is controlled by input files (pre hadoop 20) On Thu, Dec 17, 2009 at 1:58 PM, Sagi, Lee wrote: > Is there a way to throttle hive queries? > > For example, I want to tell hive to not use m

Throttling hive queries

2009-12-17 Thread Sagi, Lee
Is there a way to throttle hive queries? For example, I want to tell hive to not use more then 1000 mappers and 5 reducers for a particular query (or session).