Adam Szita created HIVE-20330:
---------------------------------
Summary: HCatLoader cannot handle multiple InputJobInfo objects
for a job with multiple inputs
Key: HIVE-20330
URL: https://issues.apache.org/jira/browse/HIVE-20330
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Adam Szita
Assignee: Adam Szita
While running performance tests on Pig (0.12 and 0.17) we've observed a huge
performance drop in a workload that has multiple inputs from HCatLoader.
The reason is that for a particular MR job with multiple Hive tables as input,
Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance but only
one table's information (InputJobInfo instance) gets tracked in the JobConf.
(This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
Any such call overwrites preexisting values, and thus only the last table's
information will be considered when Pig calls {{getStatistics}} to calculate
and estimate required reducer count.
In cases when there are 2 input tables, 256GB and 1MB in size respectively, Pig
will query the size information from HCat for both of them, but it will either
see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the execution
plan's DAG.
It should of course see 256.00097GB in total and use 257 reducers by default
accordingly.
In unlucky cases this will be 2MB and 1 reducer will have to struggle with
256GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
