Re: Re: managing HConnection
Newrelic shows 50K RPM each request to servlet == 1-3 put/get to HBase. I have mixed workload. Is it strange :) ? 2015-02-16 10:37 GMT+03:00 David chen c77...@163.com: 5 rpm? I am curious how the result is concluded?
Re: Re: Re: managing HConnection
I don't understand you. There is a single instance of servlet per application. Servlet.init method called once. Here you can instantiate HConnection and solve ANY concurrency problems. HConnection is tread-safe. Just don't close it and reuse. Then just use HConnection to get HTable. What problem you are trying to solve? 2015-02-15 9:23 GMT+03:00 David chen c77...@163.com: If sharing one HConnection for the whole application, when concurrency to access servlets increases to a threshold, whether or not it will influence the application performance? But if increasing the sharing HConnection number, the problem will be relieved?
Re: Re: Re: managing HConnection
That's true, we can share the connection, but the problem is, HConnection implements Closeable, and hence its close method is public. If you share the same connection in your api, you always have to check if the current connection is closed before you return it to the client, check previous emails to see how I was trying to achieve that. Now, I am thinking of creating a wrapper around HConnection that overrides HConnection close method and make it empty, just to make sure it's never closed On Sun, Feb 15, 2015 at 8:26 PM, Serega Sheypak serega.shey...@gmail.com wrote: I don't understand you. There is a single instance of servlet per application. Servlet.init method called once. Here you can instantiate HConnection and solve ANY concurrency problems. HConnection is tread-safe. Just don't close it and reuse. Then just use HConnection to get HTable. What problem you are trying to solve? 2015-02-15 9:23 GMT+03:00 David chen c77...@163.com: If sharing one HConnection for the whole application, when concurrency to access servlets increases to a threshold, whether or not it will influence the application performance? But if increasing the sharing HConnection number, the problem will be relieved?
Re:Re: Re: Re: managing HConnection
Sorry for the unclear represent. My problem is that whether or not a sharing Honnection can bear too many query requests?
Re: Re: managing HConnection
Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
To be honest guys I am still confused, especially that that HConnection implements Closeable and hence everyone has the right to close the connection. I wrote this code to manage connections but I am not sure about its correctness. private static class HConnectionProvider { private static HConnection hConnection; private static final Lock LOCK = new ReentrantLock(); static { hConnection = createNewConnection(); Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() { @Override public void run() { if(hConnection!=null !hConnection.isClosed()){ try { hConnection.close(); } catch (IOException e) { e.printStackTrace(); } } } })); } public static HConnection connection(){ if(!hConnection.isClosed()){ return hConnection; } boolean acquired = false; try{ acquired = LOCK.tryLock(5,TimeUnit.SECONDS); if(hConnection.isClosed()){ hConnection = createNewConnection(); } return hConnection; } catch (InterruptedException e) { throw new RuntimeException(e); }finally{ if(acquired){ LOCK.unlock(); } } } private static HConnection createNewConnection(){ try { HConnection connection = HConnectionManager.createConnection(config); return connection; } catch (IOException e) { throw new RuntimeException(e); } } } On Fri, Feb 13, 2015 at 8:57 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
What are you trying to achieve? 2015-02-13 12:36 GMT+03:00 Sleiman Jneidi jneidi.slei...@gmail.com: To be honest guys I am still confused, especially that that HConnection implements Closeable and hence everyone has the right to close the connection. I wrote this code to manage connections but I am not sure about its correctness. private static class HConnectionProvider { private static HConnection hConnection; private static final Lock LOCK = new ReentrantLock(); static { hConnection = createNewConnection(); Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() { @Override public void run() { if(hConnection!=null !hConnection.isClosed()){ try { hConnection.close(); } catch (IOException e) { e.printStackTrace(); } } } })); } public static HConnection connection(){ if(!hConnection.isClosed()){ return hConnection; } boolean acquired = false; try{ acquired = LOCK.tryLock(5,TimeUnit.SECONDS); if(hConnection.isClosed()){ hConnection = createNewConnection(); } return hConnection; } catch (InterruptedException e) { throw new RuntimeException(e); }finally{ if(acquired){ LOCK.unlock(); } } } private static HConnection createNewConnection(){ try { HConnection connection = HConnectionManager.createConnection(config); return connection; } catch (IOException e) { throw new RuntimeException(e); } } } On Fri, Feb 13, 2015 at 8:57 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
a single HConnection On Fri, Feb 13, 2015 at 11:12 AM, Serega Sheypak serega.shey...@gmail.com wrote: What are you trying to achieve? 2015-02-13 12:36 GMT+03:00 Sleiman Jneidi jneidi.slei...@gmail.com: To be honest guys I am still confused, especially that that HConnection implements Closeable and hence everyone has the right to close the connection. I wrote this code to manage connections but I am not sure about its correctness. private static class HConnectionProvider { private static HConnection hConnection; private static final Lock LOCK = new ReentrantLock(); static { hConnection = createNewConnection(); Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() { @Override public void run() { if(hConnection!=null !hConnection.isClosed()){ try { hConnection.close(); } catch (IOException e) { e.printStackTrace(); } } } })); } public static HConnection connection(){ if(!hConnection.isClosed()){ return hConnection; } boolean acquired = false; try{ acquired = LOCK.tryLock(5,TimeUnit.SECONDS); if(hConnection.isClosed()){ hConnection = createNewConnection(); } return hConnection; } catch (InterruptedException e) { throw new RuntimeException(e); }finally{ if(acquired){ LOCK.unlock(); } } } private static HConnection createNewConnection(){ try { HConnection connection = HConnectionManager.createConnection(config); return connection; } catch (IOException e) { throw new RuntimeException(e); } } } On Fri, Feb 13, 2015 at 8:57 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
What's the problem to call HConnectionManager.getConnection in Servlet.init method and pass it to your class responsible for HBase interaction? 2015-02-13 14:49 GMT+03:00 Sleiman Jneidi jneidi.slei...@gmail.com: a single HConnection On Fri, Feb 13, 2015 at 11:12 AM, Serega Sheypak serega.shey...@gmail.com wrote: What are you trying to achieve? 2015-02-13 12:36 GMT+03:00 Sleiman Jneidi jneidi.slei...@gmail.com: To be honest guys I am still confused, especially that that HConnection implements Closeable and hence everyone has the right to close the connection. I wrote this code to manage connections but I am not sure about its correctness. private static class HConnectionProvider { private static HConnection hConnection; private static final Lock LOCK = new ReentrantLock(); static { hConnection = createNewConnection(); Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() { @Override public void run() { if(hConnection!=null !hConnection.isClosed()){ try { hConnection.close(); } catch (IOException e) { e.printStackTrace(); } } } })); } public static HConnection connection(){ if(!hConnection.isClosed()){ return hConnection; } boolean acquired = false; try{ acquired = LOCK.tryLock(5,TimeUnit.SECONDS); if(hConnection.isClosed()){ hConnection = createNewConnection(); } return hConnection; } catch (InterruptedException e) { throw new RuntimeException(e); }finally{ if(acquired){ LOCK.unlock(); } } } private static HConnection createNewConnection(){ try { HConnection connection = HConnectionManager.createConnection(config); return connection; } catch (IOException e) { throw new RuntimeException(e); } } } On Fri, Feb 13, 2015 at 8:57 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks