Hi Hive Users, I'm using Cloudera distribution and Hive's 13th version on my cluster.
I came across a problem where job is not making any progress after writing log line - "*Number of reduce tasks is set to 0 since there's no reduce operator*" Below is the log for the same, could you help me what kind of issue is this because this is not a code issue as if i re-run the same job it completes successfully. Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/jars/hive-common-0.13.1-cdh5.2.1.jar!/hive-log4j.properties Total jobs = 5 Launching Job 1 out of 5 Launching Job 2 out of 5 Number of reduce tasks not specified. Defaulting to jobconf value of: 10 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: Number of reduce tasks not specified. Defaulting to jobconf value of: 10 set mapreduce.job.reduces=<number> In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1431159077692_1399, Tracking URL = http://xyz.com:8088/proxy/application_1431159077692_1399/ Starting Job = job_1431159077692_1398, Tracking URL = http://xyz.com:8088/proxy/application_1431159077692_1398/ Kill Command = /opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/hadoop/bin/hadoop job -kill job_1431159077692_1399 Kill Command = /opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/hadoop/bin/hadoop job -kill job_1431159077692_1398 Hadoop job information for Stage-12: number of mappers: 5; number of reducers: 10 Hadoop job information for Stage-1: number of mappers: 5; number of reducers: 10 2015-05-12 19:59:12,298 Stage-1 map = 0%, reduce = 0% 2015-05-12 19:59:12,298 Stage-12 map = 0%, reduce = 0% 2015-05-12 19:59:20,832 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 2.5 sec 2015-05-12 19:59:20,832 Stage-12 map = 80%, reduce = 0%, Cumulative CPU 8.63 sec 2015-05-12 19:59:21,905 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 7.06 sec 2015-05-12 19:59:22,968 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 9.34 sec 2015-05-12 19:59:24,031 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.46 sec 2015-05-12 19:59:26,265 Stage-12 map = 100%, reduce = 0%, Cumulative CPU 10.92 sec 2015-05-12 19:59:32,665 Stage-12 map = 100%, reduce = 30%, Cumulative CPU 24.51 sec 2015-05-12 19:59:33,726 Stage-12 map = 100%, reduce = 100%, Cumulative CPU 57.61 sec 2015-05-12 19:59:35,021 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 20.99 sec MapReduce Total cumulative CPU time: 57 seconds 610 msec Ended Job = job_1431159077692_1399 2015-05-12 19:59:36,084 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 39.24 sec 2015-05-12 19:59:37,146 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 42.37 sec 2015-05-12 19:59:38,203 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 45.97 sec MapReduce Total cumulative CPU time: 45 seconds 970 msec Ended Job = job_1431159077692_1398 2015-05-12 19:59:45,180 WARN [main] conf.Configuration (Configuration.java:loadProperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring. 2015-05-12 19:59:45,193 WARN [main] conf.Configuration (Configuration.java:loadProperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-05-12 19:59:45,196 WARN [main] conf.Configuration (Configuration.java:loadProperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring. 2015-05-12 19:59:45,201 WARN [main] conf.Configuration (Configuration.java:loadProperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring. 2015-05-12 19:59:45,210 WARN [main] conf.Configuration (Configuration.java:loadProperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring. 2015-05-12 19:59:45,258 WARN [main] conf.Configuration (Configuration.java:loadProperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-05-12 19:59:45,792 WARN [main] conf.HiveConf (HiveConf.java:initialize(1491)) - DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore. Execution log at: /tmp/srv-hdp-mkt-d/srv-hdp-mkt-d_20150512195858_1b598453-78a8-4867-9402-d972e3c067f2.log 2015-05-12 07:59:46 Starting to launch local task to process map join; maximum memory = 257949696 2015-05-12 07:59:47 Dump the side-table into file: file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/HashTable-Stage-4/MapJoin-mapfile10--.hashtable 2015-05-12 07:59:47 Uploaded 1 File to: file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/HashTable-Stage-4/MapJoin-mapfile10--.hashtable (475 bytes) 2015-05-12 07:59:47 Dump the side-table into file: file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/HashTable-Stage-4/MapJoin-mapfile01--.hashtable 2015-05-12 07:59:47 Uploaded 1 File to: file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/HashTable-Stage-4/MapJoin-mapfile01--.hashtable (388 bytes) 2015-05-12 07:59:47 End of local task; Time Taken: 1.209 sec. Execution completed successfully MapredLocal task succeeded Launching Job 3 out of 5 Number of reduce tasks is set to 0 since there's no reduce operator