Hi Sagi, Any chance you're running on a directory that has 614 small files?
-Todd On Thu, Dec 17, 2009 at 2:30 PM, Sagi, Lee <ls...@shopping.com> wrote: > Todd, Here is the job info. > > > > Counter Map Reduce Total File Systems HDFS bytes read 199,115,508 0 > 199,115,508 HDFS bytes written 0 9,665,472 9,665,472 Local bytes read 0 > 321,210,205 321,210,205 Local bytes written 204,404,812 321,210,205 > 525,615,017 Job Counters Launched reduce tasks 0 0 1 Rack-local map tasks > 0 0 614 Launched map tasks 0 0 37,130 Data-local map tasks 0 0 36,516 > org.apache.hadoop.hive.ql.exec.FilterOperator$Counter PASSED 0 10,572 > 10,572 FILTERED 0 217,305 217,305 > org.apache.hadoop.hive.ql.exec.MapOperator$Counter DESERIALIZE_ERRORS 0 0 > 0 Map-Reduce Framework Reduce input groups 0 429,557 429,557 Combine > output records 0 0 0 Map input records 429,557 0 429,557 Reduce output > records 0 0 0 Map output bytes 201,425,848 0 201,425,848 Map input bytes > 199,115,508 0 199,115,508 Map output records 429,557 0 429,557 Combine > input records 0 0 0 Reduce input records 0 429,557 429,557 > > > Lee Sagi | Data Warehouse Tech Lead & Architect | Work: 650-616-6575 | > Cell: 718-930-7947 > > > ------------------------------ > *From:* Todd Lipcon [mailto:t...@cloudera.com] > *Sent:* Thursday, December 17, 2009 12:18 PM > > *To:* hive-user@hadoop.apache.org > *Subject:* Re: Throttling hive queries > > Hi Lee, > > The MapReduce framework in general makes it hard for you assign fewer > mappers than there are blocks in the input data, when using FileInputFormat. > Is your input set about 42GB with a 64M block size, or 84G with a 128M block > size? > > -Todd > > On Thu, Dec 17, 2009 at 11:32 AM, Sagi, Lee <ls...@shopping.com> wrote: > >> Here is the query that I am running, just in case someone has an idea of >> how to improve it. >> >> SELECT >> CONCAT(CONCAT('"', PRSS.DATE_KEY), '"'), >> CONCAT(CONCAT('"', PRSC.DATE_KEY), '"'), >> CONCAT(CONCAT('"', PRSS.VOTF_REQUEST_ID), '"'), >> CONCAT(CONCAT('"', PRSC.VOTF_REQUEST_ID), '"'), >> CONCAT(CONCAT('"', PRSS.PRS_REQUEST_ID), '"'), >> CONCAT(CONCAT('"', PRSC.PRS_REQUEST_ID), '"'), >> ... >> ... >> ... >> FROM >> FCT_PRSS PRSS FULL OUTER JOIN FCT_PRSC PRSC ON >> (PRSS.PRS_REQUEST_ID = PRSC.PRS_REQUEST_ID) >> WHERE (PRSS.date_key >= '2009121600' AND >> PRSS.date_key < '2009121700') OR >> (PRSC.date_key >= '2009121600' AND >> PRSC.date_key < '2009121700') >> >> >> Lee Sagi | Data Warehouse Tech Lead & Architect | Work: 650-616-6575 | >> Cell: 718-930-7947 >> >> -----Original Message----- >> From: Edward Capriolo [mailto:edlinuxg...@gmail.com] >> Sent: Thursday, December 17, 2009 11:03 AM >> To: hive-user@hadoop.apache.org >> Subject: Re: Throttling hive queries >> >> You should be able >> >> hive > set mapred.map.tasks=1000 >> hive > set mapred.reduce.tasks=5 >> >> In some cases mappers is controlled by input files (pre hadoop 20) >> >> >> On Thu, Dec 17, 2009 at 1:58 PM, Sagi, Lee <ls...@shopping.com> wrote: >> > Is there a way to throttle hive queries? >> > >> > For example, I want to tell hive to not use more then 1000 mappers and >> >> > 5 reducers for a particular query (or session). >> > >> > >
