We're stepping up our Hive integration, and after confronting the HiveServer thread safety issue, I implemented a simple co-located hive connection pool. Since we use our own java-only network communication code, the service does not use Thrift; instead it's just a lightweight network service that manages a pool of HiveConnections.

Of course, after implementing this I found that it was still not thread-safe. I'm not very familiar with the hive code, but I did find a quick fix to be surprisingly easy. In org.apache.hadoop.hive.ql.exec.Utilities there is a static field instance of mapredWork. I changed it to a ThreadLocal instance and suddenly I have a thread-safe connection pool.

Now, I do understand that this is just a quick fix. The call stack from HiveStatement is synchronous, but the ThreadLocal solution would need to be revisited if any async processing is introduced. Also, I can't help wondering, why is a Utilities class holding state, let alone the state of the entire execution plan? Finally, I doubt such a simple solution solves hive-80, so I imagine other threading issues are in play when Thrift is involved. In the meantime however, I'm hoping the fix can be made, patch attached.

Cliff
### Eclipse Workspace Patch 1.0
#P hive
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java	(revision 808923)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java	(working copy)
@@ -70,7 +70,7 @@
    */
 
   public static enum ReduceField { KEY, VALUE, ALIAS };
-  private static volatile mapredWork gWork = null;
+  private static ThreadLocal<mapredWork> gWork = new ThreadLocal<mapredWork>();
   static final private Log LOG = LogFactory.getLog(Utilities.class.getName());
 
   public static void clearMapRedWork (Configuration job) {
@@ -94,17 +94,14 @@
 
   public static mapredWork getMapRedWork (Configuration job) {
     try {
-      if(gWork == null) {
-        synchronized (Utilities.class) {
-          if(gWork != null)
-            return (gWork);
-          InputStream in = new FileInputStream("HIVE_PLAN");
-          mapredWork ret = deserializeMapRedWork(in, job);
-          gWork = ret;
-        }
-        gWork.initialize();
+      mapredWork wrk = gWork.get();
+      if(wrk == null) {
+        InputStream in = new FileInputStream("HIVE_PLAN");
+        wrk = deserializeMapRedWork(in, job);
+        wrk.initialize();
+        gWork.set(wrk);
       }
-      return (gWork);
+      return (wrk);
     } catch (Exception e) {
       e.printStackTrace();
       throw new RuntimeException (e);
@@ -154,10 +151,8 @@
       String uriWithLink = planPath.toUri().toString() + "#HIVE_PLAN";
       DistributedCache.addCacheFile(new URI(uriWithLink), job);
       // Cache the object in this process too so lookups don't hit the file system
-      synchronized (Utilities.class) {
-        gWork = w;
-        gWork.initialize();
-      }
+      gWork.set(w);
+      w.initialize();
     } catch (Exception e) {
       e.printStackTrace();
       throw new RuntimeException (e);

Reply via email to