Re: [sqlalchemy] dispose/close question
On Tuesday, April 3, 2018 at 11:41:43 AM UTC-4, Mike Bayer wrote: > > right, the dispose() will emit "close connection" commands on the > socket which will leak into the parent process. > But only if `close()` has been called -- right? > they're not ! :) that's one of the "features" of QueuePool. only > when they get checked in again does the QP have any idea who they are. > > I wrote an alternative pool some months ago that works identically to > QueuePool but does the more traditional approach of having a fixed > "slot" for every possible connection.That would be a nice pool to > use someday, but unfortunately doesn't have the ten years of > production use by thousands of applications behind it. > Makes perfect sense. I'm surprised I couldn't find a debugging version of the Queue Pool that just notes the object ids as text (this way there's no reference). Like most things SqlAlchemy, It's probably in there but I missed it. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] dispose/close question
On Mon, Apr 2, 2018 at 10:19 PM, Jonathan Vanasco wrote: > looking at the current `engine.dispose()` (which I should have done sooner), > that's pretty much what it's doing -- right? > > https://bitbucket.org/zzzeek/sqlalchemy/src/55371f4cffa730f65f1b687e9f6287d2ac189227/lib/sqlalchemy/engine/base.py?at=master&fileviewer=file-view-default#base.py-1899:1923 > > > self.pool.dispose() > self.pool = self.pool.recreate() > self.dispatch.engine_disposed(self) > > > or are you thinking of something that would not call the dispose(), this way > the main process can keep that connection? right, the dispose() will emit "close connection" commands on the socket which will leak into the parent process. > > something like... > > original_pool = dbSession.connection().engine.detach() > > >def detach(self): >"""replaces the pool""" > >old_pool = self.pool > >self.pool = self.pool.recreate() > >return old_pool > > > Are the checked-out connections stored anywhere? The > 'Pool._all_connections' attribute seems to be more like '_idle_connections' > or '_available_connections' they're not ! :) that's one of the "features" of QueuePool. only when they get checked in again does the QP have any idea who they are. I wrote an alternative pool some months ago that works identically to QueuePool but does the more traditional approach of having a fixed "slot" for every possible connection.That would be a nice pool to use someday, but unfortunately doesn't have the ten years of production use by thousands of applications behind it. > > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] dispose/close question
looking at the current `engine.dispose()` (which I should have done sooner), that's pretty much what it's doing -- right? https://bitbucket.org/zzzeek/sqlalchemy/src/55371f4cffa730f65f1b687e9f6287d2ac189227/lib/sqlalchemy/engine/base.py?at=master&fileviewer=file-view-default#base.py-1899:1923 self.pool.dispose()self.pool = self.pool.recreate() self.dispatch.engine_disposed(self) or are you thinking of something that would not call the dispose(), this way the main process can keep that connection? something like... original_pool = dbSession.connection().engine.detach() def *detach(self): """replaces the pool"""* old_pool = self.pool self.pool = self.pool.recreate() return old_pool Are the checked-out connections stored anywhere? The 'Pool._all_connections' attribute seems to be more like '_idle_connections' or '_available_connections' -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] dispose/close question
we should fix that an add a new method like engine.detach() or something like that which strictly de-associates the engine from the pool of the parent process and makes a new one. right now the way to do that is: engine.pool = engine.pool.recreate() gives it a brand new pool and just loses references to the old one. On Mon, Apr 2, 2018 at 1:53 PM, Jonathan Vanasco wrote: > > > On Monday, April 2, 2018 at 1:40:06 PM UTC-4, Mike Bayer wrote: >> >> now we can probably do better than that but the dispose() case is not >> supposed to be that common or critical, it's usually for when you just >> did a fork(), and you don't want to reference any of the filehandles >> in the parent process (but to that extent, dispose() is not ideal >> because it will step on those connections that might be *used* by the >> parentso...more bugs :) ) > > > One person's uncommon is another person's everyday ;) > > Someone was running a pyramid app with SqlAlchemy connection issues. They > were running in a gunicorn container, and connected to the database pre-fork > to pull out some configuration settings - at the very least they needed to > call `Engine.dispose()`. There may be some other things going on too, but > that's the guaranteed way to screw up database connections running in > gunicorn, uwsgi and a few other servers. > > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] dispose/close question
On Monday, April 2, 2018 at 1:40:06 PM UTC-4, Mike Bayer wrote: > > now we can probably do better than that but the dispose() case is not > supposed to be that common or critical, it's usually for when you just > did a fork(), and you don't want to reference any of the filehandles > in the parent process (but to that extent, dispose() is not ideal > because it will step on those connections that might be *used* by the > parentso...more bugs :) ) > One person's uncommon is another person's everyday ;) Someone was running a pyramid app with SqlAlchemy connection issues. They were running in a gunicorn container, and connected to the database pre-fork to pull out some configuration settings - at the very least they needed to call `Engine.dispose()`. There may be some other things going on too, but that's the guaranteed way to screw up database connections running in gunicorn, uwsgi and a few other servers. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] dispose/close question
On Mon, Apr 2, 2018 at 1:36 PM, Jonathan Vanasco wrote: > clarifying: I had run tests, and the checked-out connections are still open > after dispose(), and then garbage collected out. > now we can probably do better than that but the dispose() case is not supposed to be that common or critical, it's usually for when you just did a fork(), and you don't want to reference any of the filehandles in the parent process (but to that extent, dispose() is not ideal because it will step on those connections that might be *used* by the parentso...more bugs :) ) > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] dispose/close question
clarifying: I had run tests, and the checked-out connections are still open after dispose(), and then garbage collected out. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] dispose/close question
On Monday, April 2, 2018 at 1:21:26 PM UTC-4, Mike Bayer wrote: > > Connections that are checked out stay open, they > still refer to that dereferenced pool, and I think at the moment they > don't get closed until they are garbage collected along with that pool > object. > Thanks. That's what I thought was happening, but wanted to make sure. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] dispose/close question
On Mon, Apr 2, 2018 at 1:10 PM, Jonathan Vanasco wrote: > a question came up in another list, and I just wanted to confirm a behavior > > it looks like `engine.dispose()` doesn't explicitly `close()` any open > connections, but they inherently get closed by the method's behavior. is > that right? engine.dispose() calls dispose() on the pool which will close connections that are currently checked in. That connection pool is then discarded. Connections that are checked out stay open, they still refer to that dereferenced pool, and I think at the moment they don't get closed until they are garbage collected along with that pool object. > > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] dispose/close question
a question came up in another list, and I just wanted to confirm a behavior it looks like `engine.dispose()` doesn't explicitly `close()` any open connections, but they inherently get closed by the method's behavior. is that right? -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
