Ashish,

Which issue should I post it to? Hive-80?

-Cliff

On 08/28/2009 06:07 PM, Ashish Thusoo wrote:
Hi Cliff,

Can you post the patch to the JIRA?

Ashish

-----Original Message-----
From: Cliff Resnick [mailto:[email protected]]
Sent: Friday, August 28, 2009 8:53 AM
To: [email protected]
Subject: quick fix for thread-safe connection pool

We're stepping up our Hive integration, and after confronting the HiveServer 
thread safety issue, I implemented a simple co-located hive connection pool. 
Since we use our own java-only network communication code, the service does not 
use Thrift; instead it's just a lightweight network service that manages a pool 
of HiveConnections.

Of course, after implementing this I found that it was still not thread-safe. 
I'm not very familiar with the hive code, but I did find a quick fix to be 
surprisingly easy. In org.apache.hadoop.hive.ql.exec.Utilities there is a 
static field instance of mapredWork. I changed it to a ThreadLocal instance and 
suddenly I have a thread-safe connection pool.

Now, I do understand that this is just a quick fix. The call stack from 
HiveStatement is synchronous, but the ThreadLocal solution would need to be 
revisited if any async processing is introduced. Also, I can't help wondering, 
why is a Utilities class holding state, let alone the state of the entire 
execution plan? Finally, I doubt such a simple solution solves hive-80, so I 
imagine other threading issues are in play when Thrift is involved. In the 
meantime however, I'm hoping the fix can be made, patch attached.

Cliff

Reply via email to