BELUGA BEHR created HIVE-17004:
----------------------------------
Summary: Calculating Number Of Reducers Looks At All Files
Key: HIVE-17004
URL: https://issues.apache.org/jira/browse/HIVE-17004
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 2.1.1
Reporter: BELUGA BEHR
When calculating the number of Mappers and Reducers, the two algorithms are
looking at different data sets. The number of Mappers are calculated based on
the number of splits and the number of Reducers are based on the number of
files within the HDFS directory. What you see is that if I add files to a
sub-directory of the HDFS directory, the number of splits remains the same
since I did not tell Hive to search recursively, and the number of Reducers
increases. Please improve this so that Reducers are looking at the same files
that are considered for splits and not at files within sub-directories (unless
configured to do so).
{code}
CREATE EXTERNAL TABLE Complaints (
a string,
b string,
c string,
d string,
e string,
f string,
g string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/user/admin/complaints';
{code}
{code}
[root@host ~]# sudo -u hdfs hdfs dfs -ls -R /user/admin/complaints
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.1.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.2.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.3.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.4.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.5.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.csv
{code}
{code}
INFO : Compiling
command(queryId=hive_20170502142020_dfcf77ef-56b7-4544-ab90-6e9726ea86ae):
select a, count(1) from complaints group by a limit 10
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:a,
type:string, comment:null), FieldSchema(name:_c1, type:bigint, comment:null)],
properties:null)
INFO : Completed compiling
command(queryId=hive_20170502142020_dfcf77ef-56b7-4544-ab90-6e9726ea86ae); Time
taken: 0.077 seconds
INFO : Executing
command(queryId=hive_20170502142020_dfcf77ef-56b7-4544-ab90-6e9726ea86ae):
select a, count(1) from complaints group by a limit 10
INFO : Query ID = hive_20170502142020_dfcf77ef-56b7-4544-ab90-6e9726ea86ae
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks not specified. Estimated from input data size: 11
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:2
INFO : Submitting tokens for job: job_1493729203063_0003
INFO : The url to track the job:
http://host:8088/proxy/application_1493729203063_0003/
INFO : Starting Job = job_1493729203063_0003, Tracking URL =
http://host:8088/proxy/application_1493729203063_0003/
INFO : Kill Command =
/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hadoop/bin/hadoop job
-kill job_1493729203063_0003
INFO : Hadoop job information for Stage-1: number of mappers: 2; number of
reducers: 11
INFO : 2017-05-02 14:20:14,206 Stage-1 map = 0%, reduce = 0%
INFO : 2017-05-02 14:20:22,520 Stage-1 map = 100%, reduce = 0%, Cumulative
CPU 4.48 sec
INFO : 2017-05-02 14:20:34,029 Stage-1 map = 100%, reduce = 27%, Cumulative
CPU 15.72 sec
INFO : 2017-05-02 14:20:35,069 Stage-1 map = 100%, reduce = 55%, Cumulative
CPU 21.94 sec
INFO : 2017-05-02 14:20:36,110 Stage-1 map = 100%, reduce = 64%, Cumulative
CPU 23.97 sec
INFO : 2017-05-02 14:20:39,233 Stage-1 map = 100%, reduce = 73%, Cumulative
CPU 25.26 sec
INFO : 2017-05-02 14:20:43,392 Stage-1 map = 100%, reduce = 100%, Cumulative
CPU 30.9 sec
INFO : MapReduce Total cumulative CPU time: 30 seconds 900 msec
INFO : Ended Job = job_1493729203063_0003
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: Map: 2 Reduce: 11 Cumulative CPU: 30.9 sec HDFS
Read: 735691149 HDFS Write: 153 SUCCESS
INFO : Total MapReduce CPU Time Spent: 30 seconds 900 msec
INFO : Completed executing
command(queryId=hive_20170502142020_dfcf77ef-56b7-4544-ab90-6e9726ea86ae); Time
taken: 36.035 seconds
INFO : OK
{code}
{code}
[root@host ~]# sudo -u hdfs hdfs dfs -ls -R /user/admin/complaints
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.1.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.2.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.3.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.4.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.5.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:12
/user/admin/complaints/Consumer_Complaints.csv
drwxr-xr-x - admin admin 0 2017-05-02 14:16 /user/admin/complaints/t
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:16
/user/admin/complaints/t/Consumer_Complaints.1.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:16
/user/admin/complaints/t/Consumer_Complaints.2.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:16
/user/admin/complaints/t/Consumer_Complaints.3.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:16
/user/admin/complaints/t/Consumer_Complaints.4.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:16
/user/admin/complaints/t/Consumer_Complaints.5.csv
-rwxr-xr-x 2 admin admin 122607137 2017-05-02 14:16
/user/admin/complaints/t/Consumer_Complaints.csv
{code}
{code}
INFO : Compiling
command(queryId=hive_20170502142929_66a476e5-0591-4abe-92b7-bd3e4973466e):
select a, count(1) from complaints group by a limit 10
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:a,
type:string, comment:null), FieldSchema(name:_c1, type:bigint, comment:null)],
properties:null)
INFO : Completed compiling
command(queryId=hive_20170502142929_66a476e5-0591-4abe-92b7-bd3e4973466e); Time
taken: 0.073 seconds
INFO : Executing
command(queryId=hive_20170502142929_66a476e5-0591-4abe-92b7-bd3e4973466e):
select a, count(1) from complaints group by a limit 10
INFO : Query ID = hive_20170502142929_66a476e5-0591-4abe-92b7-bd3e4973466e
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks not specified. Estimated from input data size: 22
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:2
INFO : Submitting tokens for job: job_1493729203063_0004
INFO : The url to track the job:
http://host:8088/proxy/application_1493729203063_0004/
INFO : Starting Job = job_1493729203063_0004, Tracking URL =
http://host:8088/proxy/application_1493729203063_0004/
INFO : Kill Command =
/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hadoop/bin/hadoop job
-kill job_1493729203063_0004
INFO : Hadoop job information for Stage-1: number of mappers: 2; number of
reducers: 22
INFO : 2017-05-02 14:29:27,464 Stage-1 map = 0%, reduce = 0%
INFO : 2017-05-02 14:29:36,829 Stage-1 map = 100%, reduce = 0%, Cumulative
CPU 10.2 sec
INFO : 2017-05-02 14:29:47,287 Stage-1 map = 100%, reduce = 14%, Cumulative
CPU 15.36 sec
INFO : 2017-05-02 14:29:49,381 Stage-1 map = 100%, reduce = 27%, Cumulative
CPU 20.76 sec
INFO : 2017-05-02 14:29:50,433 Stage-1 map = 100%, reduce = 32%, Cumulative
CPU 22.69 sec
INFO : 2017-05-02 14:29:56,743 Stage-1 map = 100%, reduce = 45%, Cumulative
CPU 27.73 sec
INFO : 2017-05-02 14:30:00,916 Stage-1 map = 100%, reduce = 64%, Cumulative
CPU 34.95 sec
INFO : 2017-05-02 14:30:06,142 Stage-1 map = 100%, reduce = 77%, Cumulative
CPU 41.49 sec
INFO : 2017-05-02 14:30:10,297 Stage-1 map = 100%, reduce = 82%, Cumulative
CPU 42.92 sec
INFO : 2017-05-02 14:30:11,334 Stage-1 map = 100%, reduce = 86%, Cumulative
CPU 45.24 sec
INFO : 2017-05-02 14:30:12,365 Stage-1 map = 100%, reduce = 100%, Cumulative
CPU 50.33 sec
INFO : MapReduce Total cumulative CPU time: 50 seconds 330 msec
INFO : Ended Job = job_1493729203063_0004
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: Map: 2 Reduce: 22 Cumulative CPU: 50.33 sec HDFS
Read: 735731640 HDFS Write: 153 SUCCESS
INFO : Total MapReduce CPU Time Spent: 50 seconds 330 msec
INFO : Completed executing
command(queryId=hive_20170502142929_66a476e5-0591-4abe-92b7-bd3e4973466e); Time
taken: 51.841 seconds
INFO : OK
{code}
https://github.com/apache/hive/blob/bc510f63de9d6baab3a5ad8a4bf4eed9c6fde8b1/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2959
Number of splits (Mappers) stay the same between the two runs, number of
Reducers increases.
*INFO : number of splits:2*
# Number of reduce tasks not specified. Estimated from input data size: 11
# Number of reduce tasks not specified. Estimated from input data size: 22
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
