Yes, just post it to that issue for now. We may move it to a new JIRA if needed.

Ashish 

-----Original Message-----
From: Cliff Resnick [mailto:[email protected]] 
Sent: Friday, August 28, 2009 3:15 PM
To: [email protected]
Subject: Re: quick fix for thread-safe connection pool

Ashish,

Which issue should I post it to? Hive-80?

-Cliff

On 08/28/2009 06:07 PM, Ashish Thusoo wrote:
> Hi Cliff,
>
> Can you post the patch to the JIRA?
>
> Ashish
>
> -----Original Message-----
> From: Cliff Resnick [mailto:[email protected]]
> Sent: Friday, August 28, 2009 8:53 AM
> To: [email protected]
> Subject: quick fix for thread-safe connection pool
>
> We're stepping up our Hive integration, and after confronting the HiveServer 
> thread safety issue, I implemented a simple co-located hive connection pool. 
> Since we use our own java-only network communication code, the service does 
> not use Thrift; instead it's just a lightweight network service that manages 
> a pool of HiveConnections.
>
> Of course, after implementing this I found that it was still not thread-safe. 
> I'm not very familiar with the hive code, but I did find a quick fix to be 
> surprisingly easy. In org.apache.hadoop.hive.ql.exec.Utilities there is a 
> static field instance of mapredWork. I changed it to a ThreadLocal instance 
> and suddenly I have a thread-safe connection pool.
>
> Now, I do understand that this is just a quick fix. The call stack from 
> HiveStatement is synchronous, but the ThreadLocal solution would need to be 
> revisited if any async processing is introduced. Also, I can't help 
> wondering, why is a Utilities class holding state, let alone the state of the 
> entire execution plan? Finally, I doubt such a simple solution solves 
> hive-80, so I imagine other threading issues are in play when Thrift is 
> involved. In the meantime however, I'm hoping the fix can be made, patch 
> attached.
>
> Cliff
>    

Reply via email to