Correct. No more guessing about what might happen behind the scenes.
________________________________ From: Ted Yu <yuzhih...@gmail.com> To: lars hofhansl <la...@apache.org> Cc: "dev@hbase.apache.org" <dev@hbase.apache.org> Sent: Sunday, August 4, 2013 8:08 PM Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98 w.r.t. HConstants.HBASE_CLIENT_PAUSE in #3, if user wants a different pause, a new connection would be created explicitly in the new model, right ? Cheers On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org> wrote: > Let's do a little quiz: > > HTable t1 = new HTable(conf); > t1.close(); > > // 1. Will the next line create a new HConnection behind the scenes (along > with re-creating all the caches)? > // (If so, it will be expensive, if not, when is the first HConnection > actually released?) > HTable t2 = new HTable(conf); > > // 2. how about this one? > HTable t2 = new HTable(new Configuration(conf)); > > // 3. or now? > conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000); > HTable t3 = new HTable(conf); > > // 4. and now? > conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000); > HTable t4 = new HTable(conf); > > // 5. how many connections are opened now? > t4.close(); > > This stuff is convoluted and needlessly complicated. And this is not > because the code is bad, but because the abstraction is simply inadequate. > A client wants to connect to a cluster and then do some action on that > cluster (via HTable as a convenience). > If the cluster connection is implicit it leads to all of the above > considerations. > > (#1: Yes, #2: no, #3: yes, #4: no, #5: I don't really know, id'd have run > it to see) > > -- Lars > > > > ------------------------------ > *From:* Ted Yu <yuzhih...@gmail.com> > *To:* lars hofhansl <la...@apache.org> > *Cc:* "dev@hbase.apache.org" <dev@hbase.apache.org> > *Sent:* Sunday, August 4, 2013 7:39 PM > > *Subject:* Re: Heads up, HTablePool will be deprecated in 0.94, > 0.95/0.96, and removed in 0.98 > > In the Connections "managing" HTables case, don't we need to figure out > when an HConnection should be released ? > > On Sun, Aug 4, 2013 at 7:23 PM, lars hofhansl <la...@apache.org> wrote: > > Just look at HConnectionKey part, and hoops we go through to detect > whether HConnections are the same or not, when to cache them, when/how to > release them. > In fact almost all HConnectionManager does is managing HConnections on > behalf of HTable, when it should be other way around. > > Typically, when things get hard to explain (check out the comments in > HConnectionManager) there is either an abstraction missing, or the > abstraction is not right. > The reverse (Connections "managing" HTables) has none of this. > > > -- Lars > > _______________________________ > From: Ted Yu <yuzhih...@gmail.com> > To: dev@hbase.apache.org; lars hofhansl <la...@apache.org> > Sent: Sunday, August 4, 2013 4:27 PM > Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, > and removed in 0.98 > > > > bq. no funny business with unique Configurations > > Mind telling us what is funny about this part ? > > > On Sat, Aug 3, 2013 at 10:41 PM, lars hofhansl <la...@apache.org> wrote: > > Correct. The HConnection is naturally shared between the HTables. > >There is no longer any need to worry about this (no funny business with > unique Configurations, in fact most of the code in HConnectionManager can > be removed in trunk). > > > >It is also correct that the code now has to hold on the created > HConnection, rather asking HConnectionManager for it. > > > >-- Lars > > > > > > > >________________________________ > > From: Nick Dimiduk <ndimi...@gmail.com> > >To: dev@hbase.apache.org > >Sent: Saturday, August 3, 2013 8:56 PM > > > >Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, > and removed in 0.98 > > > > > >On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > >> Does this mean that user code wouldn't be able to depend > >> on HConnectionManager for connection sharing ? > >> > > > >My read of the above is that the HConnection instance is shared across > >consumers, is the shared connection. Am I reading that correctly? > > > >On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >> > >> > Ah, I find the JIRA - HBASE-9117. > >> > > >> > Cheers > >> > > >> > > >> > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org> > wrote: > >> > > >> >> Yeah, I filed a separate ticket for the API removal in trunk. > >> >> > >> >> > >> >> > >> >> ________________________________ > >> >> From: Ted Yu <yuzhih...@gmail.com> > >> >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org> > >> >> Sent: Friday, August 2, 2013 10:31 PM > >> >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, > 0.95/0.96, > >> >> and removed in 0.98 > >> >> > >> >> > >> >> bq. HConnectionManager.getConnection() will be removed. > >> >> > >> >> I don't see the above change in 6580-trunk.txt > >> >> Would the above be done in next patch or in another JIRA ? > >> >> > >> >> Cheers > >> >> > >> >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> > wrote: > >> >> > >> >> > See. https://issues.apache.org/jira/browse/HBASE-6580 > >> >> > > >> >> > The new proposed API looks like this: > >> >> > > >> >> > Here's the proposed new API: > >> >> > * HConnectionManager: > >> >> > public static HConnection createConnection(Configuration conf) > >> >> > public static HConnection createConnection(Configuration conf, > >> >> > ExecutorService pool) > >> >> > > >> >> > * HConnection: > >> >> > public HTableInterface getTable(byte[] tableName) throws > >> IOException > >> >> > public HTableInterface getTable(byte[] tableName, > ExecutorService > >> >> > pool) throws IOException > >> >> > public HTableInterface getTable(String tableName) throws > >> IOException > >> >> > > >> >> > By default HConnectionImplementation will create an ExecutorService > >> when > >> >> > needed. The ExecutorService can optionally passed be passed in. > >> >> > HTableInterfaces are retrieved from the HConnection. By default the > >> >> > HConnection's ExecutorService is used, but optionally that can be > >> >> > overridden for each HTable. > >> >> > > >> >> > In 0.98/trunk: > >> >> > > >> >> > 1. HTablePool will be removed. It is not longer needed. > >> >> > 2. All constructors in HTable will be removed and changed to be > >> >> protected. > >> >> > All code use HTableInterface only. > >> >> > 3. HConnectionManager.getConnection() will be removed. > >> >> > 3. All HConnection caching (deleteConnection, etc,etc) will be > >> removed, > >> >> as > >> >> > it is no longer needed. > >> >> > > >> >> > > >> >> > The new flow of setting up a client would look like this: > >> >> > > >> >> > ----- Snip ----- > >> >> > // connection to the cluster > >> >> > HConnection conn = HConnectionManager.createConnection(conf); > >> >> > ... > >> >> > // When the cluster connection is established get an > HTableInterface > >> for > >> >> > each operation or thread. > >> >> > // HConnection.getTable(...) is lightweight. The table is really > just > >> a > >> >> > convenient place to call table method and for a temporary batch > cache. > >> >> > // It is in fact less overhead than HTablePool had when retrieving > a > >> >> > cached HTable. > >> >> > // The HTableInterface returned is not thread safe as before. > >> >> > // It's fine to get 1000's of these. > >> >> > // Don't cache the longer than the lifetime of the HConnection > >> >> > HTableInterface table = conn.getTable("MyTable"); > >> >> > ... > >> >> > // just flushes outstanding commit, no futher cleanup needed, can > be > >> >> > omitted. > >> >> > // HConnection holds no references to the returned HTable objects, > >> they > >> >> > can be GC'd as soon as they leave scope. > >> >> > table.close(); > >> >> > ... > >> >> > conn.close(); // done with the cluster, release resources > >> >> > ----- Snip ----- > >> >> > > >> >> > The HConnection will maintain and share its own ThreadPool for all > >> batch > >> >> > operations executed by the HTables. > >> >> > This can overridden per HConnection and/or per individual HTable > >> object. > >> >> > > >> >> > I will commit the new API to all branches early next week. > >> >> > > >> >> > Questions? Comments? Concerns? Praise? > >> >> > > >> >> > -- Lars > >> >> > >> > > >> > > >> > > > > >
