RE: [DBCP] Please let us start to improve the connection pool
Craig wrote: But I think we might be using the term recycle differently. By recycle, do you mean if a connection has been setting in the pool for a long time and is not allocated to an application, so we can close it now? No, thats pool administration. IMO its an implementation detail that should be of no concern of the user, its existence should only be exposed indirectly through settings such as test while idle. Although in practice it helps if you know whats going on. Or, by recycle, do you mean if a connection has been allocated to an application but not returned for a long time, the pool is allowed to grab it back again -- but you want to notify the application first. The grab it back behavior is already configurable (and not enabled by default) -- and it's this functionality that I object to having at all. If there was no grab it back we wouldn't have to worry about notifying anyone that it was about to happen :-). This was what actually I meant by recycling, from the POV that the whole pooling pattern is about re-cycling objects dispensed with by one Object but of use to another (as in trash re-cycling) to reduce the overhead of creating identical new objects. Its my opinion that it is beyond the remit of the pool to re-use objects which have not been positively freed by the Object which was last assigned them. My reasons are that it builds in the potential for unpredictable failures, and only benefits broken code. So I think we agree... If we're talking about the second use of recycled above, IMHO, I think adding support for recovering abandoned connections at all was a mistake. Doing anything to make it work better (knowing all the while that it cannot be made perfect) simply perpetuates the mistake. I'd much rather see this whole area of functionality deprecated, rather than continuing to mislead people into believing that its OK to depend on something that cannot ever work reliably 100% of the time. I'd agree with that too. What has now occured to me is that this philosophy also delegates the responsibility to the connection consumer for ensuring that refrences, direct or indirect, to the connection are never used once the connection has been passed back to the pool. Otherwise this is another area where confusing behaviour could result from subtle bad code. The notion expressed by someone else of creating new wrappers for each request delegating to truly pooled connections, with the wrappers discarded when the connection is returned to the pool would prevent refrences to wrappers being used to access the real pooled resources. On the other hand this safety might negate too much of the performance boost pooling provides. d. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
The notion expressed by someone else of creating new wrappers for each request delegating to truly pooled connections, with the wrappers discarded when the connection is returned to the pool would prevent refrences to wrappers being used to access the real pooled resources. On the other hand this safety might negate too much of the performance boost pooling provides. Due to the discussion, i have disabled the abandoned connection feature and try to live without it, in fact, i am not happy with this, but maybe sometimes one might create an alternative pool with this features ... In debug-mode there sould be a list of connections handed out, but ... i hear it already ... this should be done by the user. However, passing an wrapped instance of the pooled connection, like my submitted patch do e.g. new ConnectionFacade(pool.getConnection()) could never ever cost so much performance like creating a new connection, nor could it slow down the application. Java should handle lightweight objects like this in lightspeed. And how often do an application request an connection compared to the number db-operations done via this connection? Using ThreadLocal or the proxy method does not help anything, and do not solve the problem. The application works on the proxy, but the pool get back the real connection, and there is no chance to notify the proxy in any way. We (the developer) do have definitly no chance to handle the abandoned connection situation without add some code to DBCP, maybe i oversee what, but then, please provide a working example where an application access the pool through the dbcp-datasource defined in an tomcat configuration. I am ready to learn something :-) Mario - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
- Original Message -
From: Mario Ivankovits [EMAIL PROTECTED]
To: Jakarta Commons Developers List [EMAIL PROTECTED]
Sent: Monday, March 24, 2003 8:57 PM
Subject: Re: [DBCP] Please let us start to improve the connection pool
1) ThreadLocal
This might not help anything, or do i oversee something?
Thread A gets a connection from the pool
Thread A loops until the abandoned connection interval is passed
The connection will be passed back to the pool.
Thread B gets a connection from the pool (the connection which was passed
back automatically by some pool mechanism)
Thread B starts a transaction
Thread A wakes up from sleep and call commit()
Thread B roll back transaction
I use thread locals this way in web applications :
MyServlet extends SomeServlet{
void service(Request request, Response reslonse)throws
ServletExeption,IOException{
Connection lazyConnctionWrapper = decorate(getConection());
ThreadLocalConnection.set(lazyConnctionWrapper);
try{
super.service(request, response);
lazyConnctionWrapper.commit()//IfOpen();
}catch( Throwable t ){
lazyConnctionWrapper.rollback();//IfOpen();
}finally{
lazyConnctionWrapper.close();//IfOpen();
ThreadLocalConnection.set(null);//not very usefull
}
}
}
class ThreadLocalConnection{
static ThreadLocal connections = new ThreadLocal();
static Connection get(){
Connection result = connections.get();
if(result == null ){
throw IllegalStateExeption(no connection for current thread');
}else {
return result;
}
}
static void set(Connection connection){
connections.set(connection);
}
ThreadLocalConnection.get() is used to get connection in application,
lazyConnctionWrapper.close();//IfOpen(); is single place in application (s
ingle servlet) I need to close connection;
The same idea I use in JMS applications (single message listener/dispatcher
per application)
Well, Thread A and B do always have their own TheadLocal and so, the
Thread
do not know, that the connection is abandoned.
2) Proxy
Your proxy looks much more interesting, but i think, it does also not
correctly handle the problem described above. The proxy do not change
the
thread, only the connection does. For sure, i do not have much experiences
with the java Proxy thinggy, but why should
if( owner.get() == Thread.currentThread() )
ever by not true, the thread is still alive.
But maybe we could catch the method passivate and set the connection
to
null, now the connection is really released.
I have to try this.
I my case, i can easy implement such method. We have only one central
place
where we ge a connection from the pool, but i suggest this might not be
correct for all sort of application out there. Such an application then
has
problems to decorate the connection instance.
At least, i do not understand why this should be more elegant, if it works
EVERY application has to read the docs, and implement this. This is like
cut and paste in an OO design, isnt it?
Mario
class MySafeConnection implements InvokationHandler{
Connection connection;
WeakReference owner;
MySafeConnection(Connection connection){
this.connection = connection;
owner = new WeakReference( Thread.currentThread() );
}
Object invoke(Object obj, Method method, Object args[] ) throw
Throwable{
if( owner.get() == Thread.currentThread() ){
method.invoke( connection, args );
}else{
throw new IllegalStateExeption();
}
}
static Connection decorate(Connection connection){
return
(Connection)Proxy.newProxyInstance(connection.getClass().getClassLoader(),
new Class[] {Connection.class}, new
MySafeConnection(connection));
}
}
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
I use thread locals this way in web applications :
MyServlet extends SomeServlet{
...cut ...
1) This helps you to access this connection on any place within your thread,
good. But if the pool gets back the connection the thread do not loose the
instance of the pooled-connection, and so, it is possible, that two threads
uses the same connection. This happens only if you use the abandoned
configuration !!!
2) Sometimes it is needet to use two connections within one request. e.g. to
write to an log, to implement some numbercycle in the database. These have
to be commited without affecting any other db transaction, and so a new
connection is requested from the pool.
We do also have a single point where we pass back the connections needet for
the main request, and all other connections like connections to write to an
db log, or the numbercycle are also capsuled very good.
I know, the case were the pool might run into the abandoned connection
situation are very rare.
I think about this feature more like an last exit if an programmer has
made an mistake, and it helps in debugging with the stacktrace.
But to use these abandoned connection configuration in real life there are
more other things todo, e.g. reset the idle time in certain situation e.g.
Resultset.next(), prev(), ...
And maybe it is one of the 90% time spent in programming for 10% of the
problems, so i have disabled this function in my configuration.
As a consequence i vote for deprecate this feature to avoid this discussion
come up in the future again.
Mario
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
I will add some stuff to [dbutils], ThreadLocal/Proxy stuff doe's not depends on pool implementation. It is better to validate connectionout of pool too, in getConnection method (validates once per transaction if needs to invoke some lazy connection method) - Original Message - From: Mario Ivankovits [EMAIL PROTECTED] To: Jakarta Commons Developers List [EMAIL PROTECTED] Sent: Tuesday, March 25, 2003 6:24 PM Subject: Re: [DBCP] Please let us start to improve the connection pool The notion expressed by someone else of creating new wrappers for each request delegating to truly pooled connections, with the wrappers discarded when the connection is returned to the pool would prevent refrences to wrappers being used to access the real pooled resources. On the other hand this safety might negate too much of the performance boost pooling provides. Due to the discussion, i have disabled the abandoned connection feature and try to live without it, in fact, i am not happy with this, but maybe sometimes one might create an alternative pool with this features ... In debug-mode there sould be a list of connections handed out, but ... i hear it already ... this should be done by the user. However, passing an wrapped instance of the pooled connection, like my submitted patch do e.g. new ConnectionFacade(pool.getConnection()) could never ever cost so much performance like creating a new connection, nor could it slow down the application. Java should handle lightweight objects like this in lightspeed. And how often do an application request an connection compared to the number db-operations done via this connection? Using ThreadLocal or the proxy method does not help anything, and do not solve the problem. The application works on the proxy, but the pool get back the real connection, and there is no chance to notify the proxy in any way. We (the developer) do have definitly no chance to handle the abandoned connection situation without add some code to DBCP, maybe i oversee what, but then, please provide a working example where an application access the pool through the dbcp-datasource defined in an tomcat configuration. I am ready to learn something :-) Mario - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
You can store connection in ThreadLocal or in request attribute. It must be trivial to implement and you do not need workarounds. Use SessionWrapper to detect legacy stuff and throw exeption, if some code tries to put connection to session. I am sure wokarounds will not help in production. I am currently working on an simmilar solution for detecting not passed back connections. Thank you for the discussion ... Mario PS.: At least i am currently not convinced, but however there are more important things in life ;-) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
It was interesting discussion :) I see we have some stuff to depricate in dbcp, but not to improve workaraunds. We do not ignore patches, and possible it will be applied, but not by me. - Original Message - From: Mario Ivankovits [EMAIL PROTECTED] To: Jakarta Commons Developers List [EMAIL PROTECTED] Sent: Tuesday, March 25, 2003 10:19 PM Subject: Re: [DBCP] Please let us start to improve the connection pool You can store connection in ThreadLocal or in request attribute. It must be trivial to implement and you do not need workarounds. Use SessionWrapper to detect legacy stuff and throw exeption, if some code tries to put connection to session. I am sure wokarounds will not help in production. I am currently working on an simmilar solution for detecting not passed back connections. Thank you for the discussion ... Mario PS.: At least i am currently not convinced, but however there are more important things in life ;-) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
On Tue, 25 Mar 2003, Juozas Baliuka wrote: Date: Tue, 25 Mar 2003 19:14:49 +0200 From: Juozas Baliuka [EMAIL PROTECTED] Reply-To: Jakarta Commons Developers List [EMAIL PROTECTED] To: Jakarta Commons Developers List [EMAIL PROTECTED] Subject: Re: [DBCP] Please let us start to improve the connection pool Is it for drivers like postgresql jdbc ? It returns thread safe connection (but I do not trust and do use this feature). IIRC, it was originally added to assist applications that always found themselves running out of connections, even at low volumes of transactions (so it was not an issue of too many simultaneous requests). Normally, that means the app is borrowing connections and not returning them -- at some point, you run out. Craig - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
There are several circumstances where two different pieces of code can be
inadvertently sharing the same connection.
Connection pool user is responsable for this. User can store connection in
ThreadLocal or
implement something like this:
class MySafeConnection implements InvokationHandler{
Connection connection;
WeakReference owner;
MySafeConnection(Connection connection){
this.connection = connection;
owner = new WeakReference( Thread.currentThread() );
}
Object invoke(Object obj, Method method, Object args[] ) throw Throwable{
if( owner.get() == Thread.currentThread() ){
method.invoke( connection, args );
}else{
throw new IllegalStateExeption();
}
}
static Connection decorate(Connection connection){
return
(Connection)Proxy.newProxyInstance(connection.getClass().getClassLoader(),
new Class[] {Connection.class}, new
MySafeConnection(connection));
}
}
It is trivial, isn't it ?
I do not think stuff like this must be included in to pool distribution, but
it can be in documentation.
- Original Message -
From: Mario Ivankovits [EMAIL PROTECTED]
To: Jakarta Commons Developers List [EMAIL PROTECTED]
Sent: Monday, March 24, 2003 6:34 PM
Subject: [DBCP] Please let us start to improve the connection pool
Hello !
There are some bug or RFE in DBCP which sould be fixed.
Some people (including me) have already posted in this list, but it seems
there is no DBCP developer anymore or he/she is very busy.
Could one of the DBCP people please comment on how and when development in
DBCP could advance.
I would like to help in improving it (i have already posted a patch for
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=17677), but if there is no
one in this list who is willing to discuss, it is hard to do it right.
Hope to hear from one of you ...
Ciao,
Mario
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: [DBCP] Please let us start to improve the connection pool
Craig, IMHO, any application that depends on the connection pool for recovering abandoned connections (whether or not it recycles them) is broken. Far better is to focus your energy on avoiding all the cases where you grab a connection from the pool but fail to return it for some reason. One simple way to do this is to encapsulate your JDBC-using code in a try/catch/finally block, something like this: I haven't used DBCP for anthing yet, though we're proposing to use it for James in place of our homespun pool. However from what I understand of this discussion there are two things going on here; Thing one, I agree with you, code failing to return connections to the pool should lead to failure, and the sooner this can be done the better for identifying such broken code. Thing two, DBCP will apparently make a value judgement about an assigned connection, and is capable of recycling it with no notification to the code which has checked it out. In my opinion thing two is wrong or incomplete as it creates a situation where potential failure is built in, difficult to reproduce and difficult identify the cause of. In the case of JDBC connection pooling may be reasonable to want to keep a connection even when it is idle, because connections can aquire state which is expensive to reproduce. Is it not, then, unresonable to allow the pool to silently and forcably recycle apparently idle but valuable connections? My solution would either be to make it possible to turn off the forcable recycling of connections, or to make the pool capable of notifying code that its connection has been recycled. Is that reasonable? d. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [DBCP] Please let us start to improve the connection pool
Craig R. McClanahan wrote: On Mon, 24 Mar 2003, Danny Angus wrote: Date: Mon, 24 Mar 2003 22:28:29 - From: Danny Angus [EMAIL PROTECTED] Reply-To: Jakarta Commons Developers List [EMAIL PROTECTED] To: Jakarta Commons Developers List [EMAIL PROTECTED] Subject: RE: [DBCP] Please let us start to improve the connection pool Craig, IMHO, any application that depends on the connection pool for recovering abandoned connections (whether or not it recycles them) is broken. Far better is to focus your energy on avoiding all the cases where you grab a connection from the pool but fail to return it for some reason. One simple way to do this is to encapsulate your JDBC-using code in a try/catch/finally block, something like this: I haven't used DBCP for anthing yet, though we're proposing to use it for James in place of our homespun pool. However from what I understand of this discussion there are two things going on here; Thing one, I agree with you, code failing to return connections to the pool should lead to failure, and the sooner this can be done the better for identifying such broken code. Thing two, DBCP will apparently make a value judgement about an assigned connection, and is capable of recycling it with no notification to the code which has checked it out. will -- can be configured to In my opinion thing two is wrong or incomplete as it creates a situation where potential failure is built in, difficult to reproduce and difficult identify the cause of. There is a lot of flexibility in how you configure DBCP -- most of it inherited from the commons-pool capabilities that lie underneath. In all cases, the use of these features is configurable. * Test On Borrow -- The pool can perform some tests on an object instance that the pool is about to return to the application that asked for it. * Test On Return -- The pool can perform the same sorts of tests, but this time when the application returns the instance to the pool. * Test While Idle -- The pool can perform checks on connections that are in the pool (not currently allocated to an application) to make sure that it is still valid. DBCP (in particular, the BasicDataSource class found there) implements flags to enable each of these behaviors individually, along with a validation query that is used to perform the test. In addition, there are tuning parameters to control things like how often the test when idle checks are performed, and how many connections are tested each time. In the case of JDBC connection pooling may be reasonable to want to keep a connection even when it is idle, because connections can aquire state which is expensive to reproduce. Is it not, then, unresonable to allow the pool to silently and forcably recycle apparently idle but valuable connections? Keeping physical JDBC connections open between uses by the app, and allowing those connections to be shared, is the whole point of DBCP :-). But I think we might be using the term recycle differently. By recycle, do you mean if a connection has been setting in the pool for a long time and is not allocated to an application, so we can close it now? If so, that behavior is already configurable -- just set the maxIdle property to zero and no idle connection will ever get harvested. There's no application to notify in this case, because the instance being harvested wasn't allocated to an application at the time. Or, by recycle, do you mean if a connection has been allocated to an application but not returned for a long time, the pool is allowed to grab it back again -- but you want to notify the application first. The grab it back behavior is already configurable (and not enabled by default) -- and it's this functionality that I object to having at all. If there was no grab it back we wouldn't have to worry about notifying anyone that it was about to happen :-). My solution would either be to make it possible to turn off the forcable recycling of connections, or to make the pool capable of notifying code that its connection has been recycled. You can already turn off the forcible recycling -- in fact, you explicitly have to turn it *on* (i.e., for BasicDataSource, you have to explicitly set the removeAbandoned property and related values to enable it). But, I contend that no one should ever do that :-). Is that reasonable? If we're talking about the second use of recycled above, IMHO, I think adding support for recovering abandoned connections at all was a mistake. Doing anything to make it work better (knowing all the while that it cannot be made perfect) simply perpetuates the mistake. I'd much rather see this whole area of functionality deprecated, rather than continuing to mislead people into believing that its OK to depend on something that cannot ever work reliably 100% of the time. But that's just my opinion. I think Craig is correct. If code is not closing connections (and thereby not returning them to the pool), it is not the responsibility
