Hi Balaji How many reduce tasks are being triggered? If more than 1, how many of them are completed and and how many are still running? Can you try increasing the number of reduce tasks and post in the behavior you are seeing.
I believe skew is already enabled for joins in your hive configuration. Regards Bejoy KS Sent from handheld, please excuse typos. -----Original Message----- From: Ranjith <ranjith.raghuna...@gmail.com> Date: Fri, 18 May 2012 22:50:47 To: user@hive.apache.org<user@hive.apache.org> Reply-To: user@hive.apache.org Cc: user@hive.apache.org<user@hive.apache.org> Subject: Re: Reduce operation stuck This could be a result of the data being skewed. When you look at the job tracker page are you noticing just a few lurking around? Thanks, Ranjith On May 18, 2012, at 1:57 PM, Balaji Rao <sbalaji...@gmail.com> wrote: > I have a simple query running on HIVE using Amazon EMR where the > reduce operation seems to be stuck. The table is small with just > 300,000+ rows (data_analysis). > I intend to work on a 30,000,000 row table once I'm able to fix this. > The query (modified to hide some information) is as follows: > > INSERT OVERWRITE TABLE data_analysis > SELECT > a.clientid, a.isi, a.os, a.bro, a.engaged, a.iframed, > hover, hower1, hower5, hower60, > Visible, a.x1, a.x2, a.x3, > NoViewMeasure, a.x4, a.x5, a.x6, > a.alerttype, a.x7, a.x9, a.x11, a.x13, a.x17, a.x21, a.x27, a.x28, > COUNT(a.auction_id_64), SUM(CASE WHEN b.auction_id_64 IS NULL THEN > 1 ELSE 0 END) > FROM > data_subset_2 a > LEFT OUTER JOIN > OrderFields b on a.auction_id_64=b.auction_id_64 > GROUP BY > a.clientid, a.isi, a.os, a.bro, a.engaged, a.iframed, > hover, hower1, hower5, hower60, > Visible, a.x1, a.x2, a.x3, > NoViewMeasure, a.x4, a.x5, a.x6, > a.alerttype, a.x7, a.x9, a.x11, a.x13, a.x17, a.x21, a.x27, a.x28; > > > > > 2012-05-18 17:49:32,440 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:50:32,813 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:51:33,176 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:52:33,538 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:53:33,906 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:54:34,270 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:55:34,644 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:56:35,017 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:57:35,387 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:58:35,754 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:59:36,125 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:00:36,503 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:01:36,873 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:02:36,990 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:03:37,355 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:04:37,719 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:05:38,081 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:06:38,432 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:07:38,777 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:08:39,135 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:09:39,489 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:10:39,852 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:11:40,221 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:12:40,591 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:13:40,962 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:14:41,336 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:15:41,700 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:16:42,070 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:17:42,440 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:18:42,801 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:19:43,156 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:20:43,531 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:21:43,899 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:22:44,255 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:23:44,619 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:24:44,978 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:25:45,347 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:26:45,720 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:27:46,088 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:28:46,446 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:29:46,814 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:30:47,183 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:31:47,546 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:32:47,918 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:33:48,282 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:34:48,408 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:35:48,766 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:36:49,118 Stage-1 map = 100%, reduce = 97% > > Thanks, > Balaji
