I got a pretty straight forward multi-table join that constantly time out on 300 secs limit without any other error. The last several lines in the log are as below, any hint what went wrong? From the log, it looks out failing on loading "hashtable file from tmp file".
19 12:36:37,332 INFO [main] org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing Self RS[5] 2015-05-19 12:36:37,332 INFO [main] org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Using tag = -1 2015-05-19 12:36:37,334 INFO [main] org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Operator 5 RS initialized 2015-05-19 12:36:37,334 INFO [main] org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initialization Done 5 RS 2015-05-19 12:36:37,334 INFO [main] org.apache.hadoop.hive.ql.exec.GroupByOperator: Initialization Done 4 GBY 2015-05-19 12:36:37,334 INFO [main] org.apache.hadoop.hive.ql.exec.SelectOperator: Initialization Done 3 SEL 2015-05-19 12:36:37,336 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ObjectCache: Ignoring retrieval request: __HASH_MAP_MAPJOIN_42_container 2015-05-19 12:36:37,336 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ObjectCache: Ignoring retrieval request: __HASH_MAP_MAPJOIN_42_serde 2015-05-19 12:36:37,336 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: Try to retrieve from cache 2015-05-19 12:36:37,336 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: Did not find tables in cache 2015-05-19 12:36:37,337 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: Initialization Done 2 MAPJOIN 2015-05-19 12:36:37,337 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ObjectCache: Ignoring retrieval request: __HASH_MAP_MAPJOIN_43_container 2015-05-19 12:36:37,337 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ObjectCache: Ignoring retrieval request: __HASH_MAP_MAPJOIN_43_serde 2015-05-19 12:36:37,337 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: Try to retrieve from cache 2015-05-19 12:36:37,337 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: Did not find tables in cache 2015-05-19 12:36:37,337 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: Initialization Done 1 MAPJOIN 2015-05-19 12:36:37,337 INFO [main] org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Initialization Done 7 HASHTABLEDUMMY 2015-05-19 12:36:37,342 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG method=LoadHashtable from=org.apache.hadoop.hive.ql.exec.MapJoinOperator> 2015-05-19 12:36:37,342 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: ******* Load from HashTable for input file: hdfs://nameservice1/tmp/hive/bthomson/7cfb6499-04a0-4d96-a0fc-5001e3cdb413/hive_2015-05-19_12-24-11_656_259620018843960962-1/-mr-10005/000002_0 2015-05-19 12:36:37,343 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load back 1 hashtable file from tmp file uri:file:/data/2/hadoop/yarn/local/usercache/bthomson/appcache/application_1430337284339_2029/container_1430337284339_2029_01_000003/Stage-5.tar.gz/MapJoin-mapfile150--.hashtable 2015-05-19 12:36:56,925 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: This is not bucket map join, so cache 2015-05-19 12:36:56,925 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ObjectCache: Ignoring cache key: __HASH_MAP_MAPJOIN_43_container 2015-05-19 12:36:56,925 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ObjectCache: Ignoring cache key: __HASH_MAP_MAPJOIN_43_serde 2015-05-19 12:36:56,925 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: </PERFLOG method=LoadHashtable start=1432053397342 end=1432053416925 duration=19583 from=org.apache.hadoop.hive.ql.exec.MapJoinOperator> 2015-05-19 12:36:56,926 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG method=LoadHashtable from=org.apache.hadoop.hive.ql.exec.MapJoinOperator> 2015-05-19 12:36:56,926 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: ******* Load from HashTable for input file: hdfs://nameservice1/tmp/hive/bthomson/7cfb6499-04a0-4d96-a0fc-5001e3cdb413/hive_2015-05-19_12-24-11_656_259620018843960962-1/-mr-10005/000002_0 2015-05-19 12:36:56,926 INFO [main] org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load back 1 hashtable file from tmp file uri:file:/data/2/hadoop/yarn/local/usercache/bthomson/appcache/application_1430337284339_2029/container_1430337284339_2029_01_000003/Stage-5.tar.gz/MapJoin-mapfile141--.hashtable
