Re: [sqlalchemy] postgres/pyscopg failing to connect after a certain number of tests, stuck!
Thanks Mike! engine.dispose() fixed it. Makes sense that it was a garbage collection problem. For any others finding this, I just added a call to engine.dispose() to class level tearDown and it's all good. On Thu, Sep 8, 2016 at 10:39 AM, Iain Duncan wrote: > I've discovered too that the issue only happens when all my test suites > get called from one call to discovery. I've I run then in 8 calls (one to > each package, with each holding 1 to 5 test suites of about 1 to 10 tests > each), there's no issue. :-/ > > iain > > On Thu, Sep 8, 2016 at 10:36 AM, Iain Duncan > wrote: > >> Thanks Mike, no we aren't. I think we are just making a new engine per >> suite, not per test though, we have the engine init code in the class level >> setup function, and per test setUp is making new sessions. We do drop the >> tables per test, which I expect we are not doing correctly as I'm a >> postgres newb. >> >> Should I be adding dispose in class level teardown? >> >> thanks! >> Iain >> >> >> >> >> On Thu, Sep 8, 2016 at 10:26 AM, Mike Bayer >> wrote: >> >>> Engine per test is v inefficient, are you at least calling dispose() on >>> each one in teardown? >>> >>> >>> On Thursday, September 8, 2016, Iain Duncan >>> wrote: >>> Hi Jonathan, sure can, here it is below. I think maybe this has something to do with the fact that we are creating a new engine and dropping the schema on each pass maybe? Because it seems to *only* happen when we run about 75 tests in a row. Thanks for looking! class FunctionalTestSuite(unittest.TestCase): """ base class for functional test suite with db seeding" """ # derived class needs to set this to the package of the app under test # IE app_module = examp.app app_package = None # derived class needs to put settings in here settings = { 'sqlalchemy.url': None } def __init__(self, *args, **kwargs): super(FunctionalTestSuite, self).__init__(*args, **kwargs) @classmethod def setUpClass(cls): "class level setUp creates tables" configure_mappers() # separate session factory for our test runner cls.engine = engine_from_config(cls.settings, prefix='sqlalchemy.') cls.session_factory = sessionmaker(bind=cls.engine) @classmethod def init_db(cls): "initialize the database and create sessions, normally called from setUp" # drop and recreate tables once per test # NB: this has to be done differently for postgres because it's a constraint nazi if 'postgres' in cls.settings['sqlalchemy.url']: cls.engine.execute("drop owned by semaphore") cls.engine.execute("create schema if not exists public") else: Base.metadata.drop_all(bind=cls.engine) Base.metadata.create_all(bind=cls.engine) def init_sessions(self): "init seed and confirm sessions, normally called from setUp" # create sessions for seeding and separate for confirming self.seed_dbs = self.session_factory() self.confirm_dbs = self.session_factory() def init_app(self): "init test wsgi app, normally called from setUp" self.app = self.app_package.main({}, **self.settings) self.testapp = webtest.TestApp(self.app) def setUp(self): # for tests where we want fresh db on *every* test, we use this self.init_db() self.init_sessions() self.init_app() def tearDown(self): self.seed_dbs.close() self.confirm_dbs.close() On Wed, Sep 7, 2016 at 8:37 PM, Jonathan Vanasco wrote: > can you share the test harness? > > -- > You received this message because you are subscribed to the Google > Groups "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout. >>> -- >>> You received this message because you are subscribed to the Google >>>
Re: [sqlalchemy] postgres/pyscopg failing to connect after a certain number of tests, stuck!
I've discovered too that the issue only happens when all my test suites get called from one call to discovery. I've I run then in 8 calls (one to each package, with each holding 1 to 5 test suites of about 1 to 10 tests each), there's no issue. :-/ iain On Thu, Sep 8, 2016 at 10:36 AM, Iain Duncan wrote: > Thanks Mike, no we aren't. I think we are just making a new engine per > suite, not per test though, we have the engine init code in the class level > setup function, and per test setUp is making new sessions. We do drop the > tables per test, which I expect we are not doing correctly as I'm a > postgres newb. > > Should I be adding dispose in class level teardown? > > thanks! > Iain > > > > > On Thu, Sep 8, 2016 at 10:26 AM, Mike Bayer > wrote: > >> Engine per test is v inefficient, are you at least calling dispose() on >> each one in teardown? >> >> >> On Thursday, September 8, 2016, Iain Duncan >> wrote: >> >>> Hi Jonathan, sure can, here it is below. I think maybe this has >>> something to do with the fact that we are creating a new engine and >>> dropping the schema on each pass maybe? Because it seems to *only* happen >>> when we run about 75 tests in a row. >>> >>> Thanks for looking! >>> >>> class FunctionalTestSuite(unittest.TestCase): >>> """ >>> base class for functional test suite with db seeding" >>> """ >>> >>> # derived class needs to set this to the package of the app under test >>> # IE app_module = examp.app >>> app_package = None >>> >>> # derived class needs to put settings in here >>> settings = { >>> 'sqlalchemy.url': None >>> } >>> >>> def __init__(self, *args, **kwargs): >>> super(FunctionalTestSuite, self).__init__(*args, **kwargs) >>> >>> @classmethod >>> def setUpClass(cls): >>> "class level setUp creates tables" >>> configure_mappers() >>> # separate session factory for our test runner >>> cls.engine = engine_from_config(cls.settings, prefix='sqlalchemy.') >>> cls.session_factory = sessionmaker(bind=cls.engine) >>> >>> @classmethod >>> def init_db(cls): >>> "initialize the database and create sessions, normally called from >>> setUp" >>> # drop and recreate tables once per test >>> # NB: this has to be done differently for postgres because it's a >>> constraint nazi >>> if 'postgres' in cls.settings['sqlalchemy.url']: >>> cls.engine.execute("drop owned by semaphore") >>> cls.engine.execute("create schema if not exists public") >>> else: >>> Base.metadata.drop_all(bind=cls.engine) >>> Base.metadata.create_all(bind=cls.engine) >>> >>> def init_sessions(self): >>> "init seed and confirm sessions, normally called from setUp" >>> # create sessions for seeding and separate for confirming >>> self.seed_dbs = self.session_factory() >>> self.confirm_dbs = self.session_factory() >>> >>> def init_app(self): >>> "init test wsgi app, normally called from setUp" >>> self.app = self.app_package.main({}, **self.settings) >>> self.testapp = webtest.TestApp(self.app) >>> >>> def setUp(self): >>> # for tests where we want fresh db on *every* test, we use this >>> self.init_db() >>> self.init_sessions() >>> self.init_app() >>> >>> def tearDown(self): >>> self.seed_dbs.close() >>> self.confirm_dbs.close() >>> >>> >>> On Wed, Sep 7, 2016 at 8:37 PM, Jonathan Vanasco >>> wrote: >>> can you share the test harness? -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "sqlalchemy" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to sqlalchemy+unsubscr...@googlegroups.com. >>> To post to this group, send email to sqlalchemy@googlegroups.com. >>> Visit this group at https://groups.google.com/group/sqlalchemy. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "sqlalchemy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sqlalchemy+unsubscr...@googlegroups.com. >> To post to this group, send email to sqlalchemy@googlegroups.com. >> Visit this group at https://groups.google.com/group/sqlalchemy. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message beca
Re: [sqlalchemy] postgres/pyscopg failing to connect after a certain number of tests, stuck!
Thanks Mike, no we aren't. I think we are just making a new engine per suite, not per test though, we have the engine init code in the class level setup function, and per test setUp is making new sessions. We do drop the tables per test, which I expect we are not doing correctly as I'm a postgres newb. Should I be adding dispose in class level teardown? thanks! Iain On Thu, Sep 8, 2016 at 10:26 AM, Mike Bayer wrote: > Engine per test is v inefficient, are you at least calling dispose() on > each one in teardown? > > > On Thursday, September 8, 2016, Iain Duncan > wrote: > >> Hi Jonathan, sure can, here it is below. I think maybe this has something >> to do with the fact that we are creating a new engine and dropping the >> schema on each pass maybe? Because it seems to *only* happen when we run >> about 75 tests in a row. >> >> Thanks for looking! >> >> class FunctionalTestSuite(unittest.TestCase): >> """ >> base class for functional test suite with db seeding" >> """ >> >> # derived class needs to set this to the package of the app under test >> # IE app_module = examp.app >> app_package = None >> >> # derived class needs to put settings in here >> settings = { >> 'sqlalchemy.url': None >> } >> >> def __init__(self, *args, **kwargs): >> super(FunctionalTestSuite, self).__init__(*args, **kwargs) >> >> @classmethod >> def setUpClass(cls): >> "class level setUp creates tables" >> configure_mappers() >> # separate session factory for our test runner >> cls.engine = engine_from_config(cls.settings, prefix='sqlalchemy.') >> cls.session_factory = sessionmaker(bind=cls.engine) >> >> @classmethod >> def init_db(cls): >> "initialize the database and create sessions, normally called from >> setUp" >> # drop and recreate tables once per test >> # NB: this has to be done differently for postgres because it's a >> constraint nazi >> if 'postgres' in cls.settings['sqlalchemy.url']: >> cls.engine.execute("drop owned by semaphore") >> cls.engine.execute("create schema if not exists public") >> else: >> Base.metadata.drop_all(bind=cls.engine) >> Base.metadata.create_all(bind=cls.engine) >> >> def init_sessions(self): >> "init seed and confirm sessions, normally called from setUp" >> # create sessions for seeding and separate for confirming >> self.seed_dbs = self.session_factory() >> self.confirm_dbs = self.session_factory() >> >> def init_app(self): >> "init test wsgi app, normally called from setUp" >> self.app = self.app_package.main({}, **self.settings) >> self.testapp = webtest.TestApp(self.app) >> >> def setUp(self): >> # for tests where we want fresh db on *every* test, we use this >> self.init_db() >> self.init_sessions() >> self.init_app() >> >> def tearDown(self): >> self.seed_dbs.close() >> self.confirm_dbs.close() >> >> >> On Wed, Sep 7, 2016 at 8:37 PM, Jonathan Vanasco >> wrote: >> >>> can you share the test harness? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "sqlalchemy" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to sqlalchemy+unsubscr...@googlegroups.com. >>> To post to this group, send email to sqlalchemy@googlegroups.com. >>> Visit this group at https://groups.google.com/group/sqlalchemy. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "sqlalchemy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sqlalchemy+unsubscr...@googlegroups.com. >> To post to this group, send email to sqlalchemy@googlegroups.com. >> Visit this group at https://groups.google.com/group/sqlalchemy. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] postgres/pyscopg failing to connect after a certain number of tests, stuck!
Engine per test is v inefficient, are you at least calling dispose() on each one in teardown? On Thursday, September 8, 2016, Iain Duncan wrote: > Hi Jonathan, sure can, here it is below. I think maybe this has something > to do with the fact that we are creating a new engine and dropping the > schema on each pass maybe? Because it seems to *only* happen when we run > about 75 tests in a row. > > Thanks for looking! > > class FunctionalTestSuite(unittest.TestCase): > """ > base class for functional test suite with db seeding" > """ > > # derived class needs to set this to the package of the app under test > # IE app_module = examp.app > app_package = None > > # derived class needs to put settings in here > settings = { > 'sqlalchemy.url': None > } > > def __init__(self, *args, **kwargs): > super(FunctionalTestSuite, self).__init__(*args, **kwargs) > > @classmethod > def setUpClass(cls): > "class level setUp creates tables" > configure_mappers() > # separate session factory for our test runner > cls.engine = engine_from_config(cls.settings, prefix='sqlalchemy.') > cls.session_factory = sessionmaker(bind=cls.engine) > > @classmethod > def init_db(cls): > "initialize the database and create sessions, normally called from > setUp" > # drop and recreate tables once per test > # NB: this has to be done differently for postgres because it's a > constraint nazi > if 'postgres' in cls.settings['sqlalchemy.url']: > cls.engine.execute("drop owned by semaphore") > cls.engine.execute("create schema if not exists public") > else: > Base.metadata.drop_all(bind=cls.engine) > Base.metadata.create_all(bind=cls.engine) > > def init_sessions(self): > "init seed and confirm sessions, normally called from setUp" > # create sessions for seeding and separate for confirming > self.seed_dbs = self.session_factory() > self.confirm_dbs = self.session_factory() > > def init_app(self): > "init test wsgi app, normally called from setUp" > self.app = self.app_package.main({}, **self.settings) > self.testapp = webtest.TestApp(self.app) > > def setUp(self): > # for tests where we want fresh db on *every* test, we use this > self.init_db() > self.init_sessions() > self.init_app() > > def tearDown(self): > self.seed_dbs.close() > self.confirm_dbs.close() > > > On Wed, Sep 7, 2016 at 8:37 PM, Jonathan Vanasco > wrote: > >> can you share the test harness? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "sqlalchemy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sqlalchemy+unsubscr...@googlegroups.com >> >> . >> To post to this group, send email to sqlalchemy@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/sqlalchemy. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com > > . > To post to this group, send email to sqlalchemy@googlegroups.com > . > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.