I have a simple query running on HIVE using Amazon EMR where the reduce operation seems to be stuck. The table is small with just 300,000+ rows (data_analysis). I intend to work on a 30,000,000 row table once I'm able to fix this. The query (modified to hide some information) is as follows:
INSERT OVERWRITE TABLE data_analysis SELECT a.clientid, a.isi, a.os, a.bro, a.engaged, a.iframed, hover, hower1, hower5, hower60, Visible, a.x1, a.x2, a.x3, NoViewMeasure, a.x4, a.x5, a.x6, a.alerttype, a.x7, a.x9, a.x11, a.x13, a.x17, a.x21, a.x27, a.x28, COUNT(a.auction_id_64), SUM(CASE WHEN b.auction_id_64 IS NULL THEN 1 ELSE 0 END) FROM data_subset_2 a LEFT OUTER JOIN OrderFields b on a.auction_id_64=b.auction_id_64 GROUP BY a.clientid, a.isi, a.os, a.bro, a.engaged, a.iframed, hover, hower1, hower5, hower60, Visible, a.x1, a.x2, a.x3, NoViewMeasure, a.x4, a.x5, a.x6, a.alerttype, a.x7, a.x9, a.x11, a.x13, a.x17, a.x21, a.x27, a.x28; 2012-05-18 17:49:32,440 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:50:32,813 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:51:33,176 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:52:33,538 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:53:33,906 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:54:34,270 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:55:34,644 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:56:35,017 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:57:35,387 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:58:35,754 Stage-1 map = 100%, reduce = 97% 2012-05-18 17:59:36,125 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:00:36,503 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:01:36,873 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:02:36,990 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:03:37,355 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:04:37,719 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:05:38,081 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:06:38,432 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:07:38,777 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:08:39,135 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:09:39,489 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:10:39,852 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:11:40,221 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:12:40,591 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:13:40,962 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:14:41,336 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:15:41,700 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:16:42,070 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:17:42,440 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:18:42,801 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:19:43,156 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:20:43,531 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:21:43,899 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:22:44,255 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:23:44,619 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:24:44,978 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:25:45,347 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:26:45,720 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:27:46,088 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:28:46,446 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:29:46,814 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:30:47,183 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:31:47,546 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:32:47,918 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:33:48,282 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:34:48,408 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:35:48,766 Stage-1 map = 100%, reduce = 97% 2012-05-18 18:36:49,118 Stage-1 map = 100%, reduce = 97% Thanks, Balaji
