RE: managing HConnection
Hi, Thank you Serega for the helpful reply and thanks Jneidi for asking this. I have similar confusion. So Serega, when does your application finally close the HConnection? Or the connection is NEVER closed as long as your application is running? Is it OK to NOT close the HConnection and the application exit directly? My application is a long-running service, accept user request and do CRUD to hbase. So I would like to use your model here. But, is it reasonable to keep that HConnection open a very long time, for example months? Is there any potential problem I need to take care? Also as David Chen asked, if all threads share same HConnection, it may has limitation to support high throughput, so a pool of Connections maybe better? Thanks, Ming -Original Message- From: Serega Sheypak [mailto:serega.shey...@gmail.com] Sent: Wednesday, February 04, 2015 1:02 AM To: user Subject: Re: managing HConnection Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: managing HConnection
Hi, I'm closing it in servlet.destroy. I didn't see any problems here for months. I'm using servlet lifecycle to deal with hconnection. вторник, 17 февраля 2015 г. пользователь Liu, Ming (HPIT-GADSC) написал: Hi, Thank you Serega for the helpful reply and thanks Jneidi for asking this. I have similar confusion. So Serega, when does your application finally close the HConnection? Or the connection is NEVER closed as long as your application is running? Is it OK to NOT close the HConnection and the application exit directly? My application is a long-running service, accept user request and do CRUD to hbase. So I would like to use your model here. But, is it reasonable to keep that HConnection open a very long time, for example months? Is there any potential problem I need to take care? Also as David Chen asked, if all threads share same HConnection, it may has limitation to support high throughput, so a pool of Connections maybe better? Thanks, Ming -Original Message- From: Serega Sheypak [mailto:serega.shey...@gmail.com javascript:;] Sent: Wednesday, February 04, 2015 1:02 AM To: user Subject: Re: managing HConnection Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com javascript:;: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com javascript:; wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
Newrelic shows 50K RPM each request to servlet == 1-3 put/get to HBase. I have mixed workload. Is it strange :) ? 2015-02-16 10:37 GMT+03:00 David chen c77...@163.com: 5 rpm? I am curious how the result is concluded?
Re:Re: Re: managing HConnection
In your scene, per servlet used a HConnecton object, so the response for 5 rpm is so easy. If these servlets should share the only one HConnecton object, whether or not the response latency for 5 rpm would increase?
Re: Re: Re: managing HConnection
I don't understand you. There is a single instance of servlet per application. Servlet.init method called once. Here you can instantiate HConnection and solve ANY concurrency problems. HConnection is tread-safe. Just don't close it and reuse. Then just use HConnection to get HTable. What problem you are trying to solve? 2015-02-15 9:23 GMT+03:00 David chen c77...@163.com: If sharing one HConnection for the whole application, when concurrency to access servlets increases to a threshold, whether or not it will influence the application performance? But if increasing the sharing HConnection number, the problem will be relieved?
Re:Re: Re: managing HConnection
If sharing one HConnection for the whole application, when concurrency to access servlets increases to a threshold, whether or not it will influence the application performance? But if increasing the sharing HConnection number, the problem will be relieved?
Re: Re: Re: managing HConnection
That's true, we can share the connection, but the problem is, HConnection implements Closeable, and hence its close method is public. If you share the same connection in your api, you always have to check if the current connection is closed before you return it to the client, check previous emails to see how I was trying to achieve that. Now, I am thinking of creating a wrapper around HConnection that overrides HConnection close method and make it empty, just to make sure it's never closed On Sun, Feb 15, 2015 at 8:26 PM, Serega Sheypak serega.shey...@gmail.com wrote: I don't understand you. There is a single instance of servlet per application. Servlet.init method called once. Here you can instantiate HConnection and solve ANY concurrency problems. HConnection is tread-safe. Just don't close it and reuse. Then just use HConnection to get HTable. What problem you are trying to solve? 2015-02-15 9:23 GMT+03:00 David chen c77...@163.com: If sharing one HConnection for the whole application, when concurrency to access servlets increases to a threshold, whether or not it will influence the application performance? But if increasing the sharing HConnection number, the problem will be relieved?
Re:Re: Re: Re: managing HConnection
Sorry for the unclear represent. My problem is that whether or not a sharing Honnection can bear too many query requests?
Re: managing HConnection
It can. 5 rpm, no problem. понедельник, 16 февраля 2015 г. пользователь David chen написал: Sorry for the unclear represent. My problem is that whether or not a sharing Honnection can bear too many query requests?
Re: Re: managing HConnection
Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
To be honest guys I am still confused, especially that that HConnection implements Closeable and hence everyone has the right to close the connection. I wrote this code to manage connections but I am not sure about its correctness. private static class HConnectionProvider { private static HConnection hConnection; private static final Lock LOCK = new ReentrantLock(); static { hConnection = createNewConnection(); Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() { @Override public void run() { if(hConnection!=null !hConnection.isClosed()){ try { hConnection.close(); } catch (IOException e) { e.printStackTrace(); } } } })); } public static HConnection connection(){ if(!hConnection.isClosed()){ return hConnection; } boolean acquired = false; try{ acquired = LOCK.tryLock(5,TimeUnit.SECONDS); if(hConnection.isClosed()){ hConnection = createNewConnection(); } return hConnection; } catch (InterruptedException e) { throw new RuntimeException(e); }finally{ if(acquired){ LOCK.unlock(); } } } private static HConnection createNewConnection(){ try { HConnection connection = HConnectionManager.createConnection(config); return connection; } catch (IOException e) { throw new RuntimeException(e); } } } On Fri, Feb 13, 2015 at 8:57 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
What are you trying to achieve? 2015-02-13 12:36 GMT+03:00 Sleiman Jneidi jneidi.slei...@gmail.com: To be honest guys I am still confused, especially that that HConnection implements Closeable and hence everyone has the right to close the connection. I wrote this code to manage connections but I am not sure about its correctness. private static class HConnectionProvider { private static HConnection hConnection; private static final Lock LOCK = new ReentrantLock(); static { hConnection = createNewConnection(); Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() { @Override public void run() { if(hConnection!=null !hConnection.isClosed()){ try { hConnection.close(); } catch (IOException e) { e.printStackTrace(); } } } })); } public static HConnection connection(){ if(!hConnection.isClosed()){ return hConnection; } boolean acquired = false; try{ acquired = LOCK.tryLock(5,TimeUnit.SECONDS); if(hConnection.isClosed()){ hConnection = createNewConnection(); } return hConnection; } catch (InterruptedException e) { throw new RuntimeException(e); }finally{ if(acquired){ LOCK.unlock(); } } } private static HConnection createNewConnection(){ try { HConnection connection = HConnectionManager.createConnection(config); return connection; } catch (IOException e) { throw new RuntimeException(e); } } } On Fri, Feb 13, 2015 at 8:57 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
a single HConnection On Fri, Feb 13, 2015 at 11:12 AM, Serega Sheypak serega.shey...@gmail.com wrote: What are you trying to achieve? 2015-02-13 12:36 GMT+03:00 Sleiman Jneidi jneidi.slei...@gmail.com: To be honest guys I am still confused, especially that that HConnection implements Closeable and hence everyone has the right to close the connection. I wrote this code to manage connections but I am not sure about its correctness. private static class HConnectionProvider { private static HConnection hConnection; private static final Lock LOCK = new ReentrantLock(); static { hConnection = createNewConnection(); Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() { @Override public void run() { if(hConnection!=null !hConnection.isClosed()){ try { hConnection.close(); } catch (IOException e) { e.printStackTrace(); } } } })); } public static HConnection connection(){ if(!hConnection.isClosed()){ return hConnection; } boolean acquired = false; try{ acquired = LOCK.tryLock(5,TimeUnit.SECONDS); if(hConnection.isClosed()){ hConnection = createNewConnection(); } return hConnection; } catch (InterruptedException e) { throw new RuntimeException(e); }finally{ if(acquired){ LOCK.unlock(); } } } private static HConnection createNewConnection(){ try { HConnection connection = HConnectionManager.createConnection(config); return connection; } catch (IOException e) { throw new RuntimeException(e); } } } On Fri, Feb 13, 2015 at 8:57 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: Re: managing HConnection
What's the problem to call HConnectionManager.getConnection in Servlet.init method and pass it to your class responsible for HBase interaction? 2015-02-13 14:49 GMT+03:00 Sleiman Jneidi jneidi.slei...@gmail.com: a single HConnection On Fri, Feb 13, 2015 at 11:12 AM, Serega Sheypak serega.shey...@gmail.com wrote: What are you trying to achieve? 2015-02-13 12:36 GMT+03:00 Sleiman Jneidi jneidi.slei...@gmail.com: To be honest guys I am still confused, especially that that HConnection implements Closeable and hence everyone has the right to close the connection. I wrote this code to manage connections but I am not sure about its correctness. private static class HConnectionProvider { private static HConnection hConnection; private static final Lock LOCK = new ReentrantLock(); static { hConnection = createNewConnection(); Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() { @Override public void run() { if(hConnection!=null !hConnection.isClosed()){ try { hConnection.close(); } catch (IOException e) { e.printStackTrace(); } } } })); } public static HConnection connection(){ if(!hConnection.isClosed()){ return hConnection; } boolean acquired = false; try{ acquired = LOCK.tryLock(5,TimeUnit.SECONDS); if(hConnection.isClosed()){ hConnection = createNewConnection(); } return hConnection; } catch (InterruptedException e) { throw new RuntimeException(e); }finally{ if(acquired){ LOCK.unlock(); } } } private static HConnection createNewConnection(){ try { HConnection connection = HConnectionManager.createConnection(config); return connection; } catch (IOException e) { throw new RuntimeException(e); } } } On Fri, Feb 13, 2015 at 8:57 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, really, I can share one Hconnection for the whole application. It's done by design. I have several servlets. Each servlet has 1-2 controllers working with hbase internally (put/get/e.t.c) Right now I don't see any reason to refactor code and share single HConnection for all controllers in servlets. 2015-02-13 6:56 GMT+03:00 David chen c77...@163.com: Hi Serega, I am very interesting in the reason why per application need to create 5 instead of only one HConnection instances during servlet initialization? At 2015-02-04 01:01:38, Serega Sheypak serega.shey...@gmail.com wrote: Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: managing HConnection
Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks
Re: managing HConnection
Hi, guys from group helped me a lot. I did solve pretty the same problem (CRUD web-app) 1. Use single instance of HConnection per application. 2. Instantiate it once. 3. create HTable instance for each CRUD operation and safely close it (try-catch-finally). Use the same HConnection to create any HTable for CRUD operation. 4. DO NOT close HConnection after CRUD operation I have logic controllers which get HConnection injection in HttpServlet.init method. So I have 5 HConnection instances per application created during servlet initialization 2015-02-03 18:12 GMT+03:00 Ted Yu yuzhih...@gmail.com: Please see '61.1. Cluster Connections' under http://hbase.apache.org/book.html#architecture.client Cheers On Tue, Feb 3, 2015 at 6:47 AM, sleimanjneidi jneidi.slei...@gmail.com wrote: Hi all, I am using hbase-0.98.1-cdh5.1.4 client and I am a bit confused by the documentation of HConnection. The document says the following: HConnection instances can be shared. Sharing is usually what you want because rather than each HConnection instance having to do its own discovery of regions out on the cluster, instead, all clients get to share the one cache of locations. HConnectionManager does the sharing for you if you go by it getting connections. Sharing makes cleanup of HConnections awkward. . So now I have a simple question: Can I share the same HConnection instance in my entire application? And write some magic code to know when to close or never close at all? Or I have to create an instance and close it every time I do a CRUD operation ? Many thanks