We're stepping up our Hive integration, and after confronting the
HiveServer thread safety issue, I implemented a simple co-located hive
connection pool. Since we use our own java-only network communication
code, the service does not use Thrift; instead it's just a lightweight
network service that manages a pool of HiveConnections.
Of course, after implementing this I found that it was still not
thread-safe. I'm not very familiar with the hive code, but I did find a
quick fix to be surprisingly easy. In
org.apache.hadoop.hive.ql.exec.Utilities there is a static field
instance of mapredWork. I changed it to a ThreadLocal instance and
suddenly I have a thread-safe connection pool.
Now, I do understand that this is just a quick fix. The call stack from
HiveStatement is synchronous, but the ThreadLocal solution would need to
be revisited if any async processing is introduced. Also, I can't help
wondering, why is a Utilities class holding state, let alone the state
of the entire execution plan? Finally, I doubt such a simple solution
solves hive-80, so I imagine other threading issues are in play when
Thrift is involved. In the meantime however, I'm hoping the fix can be
made, patch attached.
Cliff
### Eclipse Workspace Patch 1.0
#P hive
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java (revision 808923)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java (working copy)
@@ -70,7 +70,7 @@
*/
public static enum ReduceField { KEY, VALUE, ALIAS };
- private static volatile mapredWork gWork = null;
+ private static ThreadLocal<mapredWork> gWork = new ThreadLocal<mapredWork>();
static final private Log LOG = LogFactory.getLog(Utilities.class.getName());
public static void clearMapRedWork (Configuration job) {
@@ -94,17 +94,14 @@
public static mapredWork getMapRedWork (Configuration job) {
try {
- if(gWork == null) {
- synchronized (Utilities.class) {
- if(gWork != null)
- return (gWork);
- InputStream in = new FileInputStream("HIVE_PLAN");
- mapredWork ret = deserializeMapRedWork(in, job);
- gWork = ret;
- }
- gWork.initialize();
+ mapredWork wrk = gWork.get();
+ if(wrk == null) {
+ InputStream in = new FileInputStream("HIVE_PLAN");
+ wrk = deserializeMapRedWork(in, job);
+ wrk.initialize();
+ gWork.set(wrk);
}
- return (gWork);
+ return (wrk);
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException (e);
@@ -154,10 +151,8 @@
String uriWithLink = planPath.toUri().toString() + "#HIVE_PLAN";
DistributedCache.addCacheFile(new URI(uriWithLink), job);
// Cache the object in this process too so lookups don't hit the file system
- synchronized (Utilities.class) {
- gWork = w;
- gWork.initialize();
- }
+ gWork.set(w);
+ w.initialize();
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException (e);