Radim Kubacki created MAPREDUCE-6312: ----------------------------------------
Summary: Hive fails due to stale proxy in ClientServiceDelegate Key: MAPREDUCE-6312 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6312 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.5.0 Reporter: Radim Kubacki ClientServiceDelegate initializes its realProxy field to AMProxy for a new or running job. Later when the job finishes it will not update this proxy to query history server and AM will not return valid data for this job. We found this while investigating https://issues.cloudera.org/browse/DISTRO-631 that describes Hive failure because it uses loop like {code} progress(JobClient jc, RunningJob rj) { ... while (!rj.isComplete() || (extraRounds > 0)) { try { Thread.sleep(1000); } catch (InterruptedException e) { } RunningJob newRj = jc.getJob(rj.getID()); if (newRj == null) { // under exceptional load, hadoop may not be able to look up status // of finished jobs (because it has purged them from memory). From // hive's perspective - it's equivalent to the job having failed. // So raise a meaningful exception throw new IOException("Could not find status of job:" + rj.getID()); } else { rj = newRj; } } {code} In this snippet JobClient.getJob will try to create RunningJob instance referring to job file in /user/$USER/.staging even when job is finished and the file is moved to /user/history/done (or /user/history/done_intermediate). Note that Hive queries can succeed if there is a timing where HDFS performs actual file delete with a delay. We can try to write a patch if there is an agreement that this should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)