Re: [sqlalchemy] postgres/pyscopg failing to connect after a certain number of tests, stuck!

[Iain Duncan]

Thanks Mike! engine.dispose() fixed it. Makes sense that it was a garbage
collection problem.

For any others finding this, I just added a call to engine.dispose() to
class level tearDown and it's all good.

On Thu, Sep 8, 2016 at 10:39 AM, Iain Duncan 
wrote:

> I've discovered too that the issue only happens when all my test suites
> get called from one call to discovery. I've I run then in 8 calls (one to
> each package, with each holding 1 to 5 test suites of about 1 to 10 tests
> each), there's no issue. :-/
>
> iain
>
> On Thu, Sep 8, 2016 at 10:36 AM, Iain Duncan 
> wrote:
>
>> Thanks Mike, no we aren't. I think we are just making a new engine per
>> suite, not per test though, we have the engine init code in the class level
>> setup function, and per test setUp is making new sessions. We do drop the
>> tables per test, which I expect we are not doing correctly as I'm a
>> postgres newb.
>>
>> Should I be adding dispose in class level teardown?
>>
>> thanks!
>> Iain
>>
>>
>>
>>
>> On Thu, Sep 8, 2016 at 10:26 AM, Mike Bayer 
>> wrote:
>>
>>> Engine per test is v inefficient, are you at least calling dispose() on
>>> each one in teardown?
>>>
>>>
>>> On Thursday, September 8, 2016, Iain Duncan 
>>> wrote:
>>>
 Hi Jonathan, sure can, here it is below. I think maybe this has
 something to do with the fact that we are creating a new engine and
 dropping the schema on each pass maybe? Because it seems to *only* happen
 when we run about 75 tests in a row.

 Thanks for looking!

 class FunctionalTestSuite(unittest.TestCase):
 """
 base class for functional test suite with db seeding"
 """

 # derived class needs to set this to the package of the app under test
 # IE app_module = examp.app
 app_package = None

 # derived class needs to put settings in here
 settings = {
 'sqlalchemy.url': None
 }

 def __init__(self, *args, **kwargs):
 super(FunctionalTestSuite, self).__init__(*args, **kwargs)

 @classmethod
 def setUpClass(cls):
 "class level setUp creates tables"
 configure_mappers()
 # separate session factory for our test runner
 cls.engine = engine_from_config(cls.settings, prefix='sqlalchemy.')
 cls.session_factory = sessionmaker(bind=cls.engine)

 @classmethod
 def init_db(cls):
 "initialize the database and create sessions, normally called from 
 setUp"
 # drop and recreate tables once per test
 # NB: this has to be done differently for postgres because it's a 
 constraint nazi
 if 'postgres' in cls.settings['sqlalchemy.url']:
 cls.engine.execute("drop owned by semaphore")
 cls.engine.execute("create schema if not exists public")
 else:
 Base.metadata.drop_all(bind=cls.engine)
 Base.metadata.create_all(bind=cls.engine)

 def init_sessions(self):
 "init seed and confirm sessions, normally called from setUp"
 # create sessions for seeding and separate for confirming
 self.seed_dbs = self.session_factory()
 self.confirm_dbs = self.session_factory()

 def init_app(self):
 "init test wsgi app, normally called from setUp"
 self.app = self.app_package.main({}, **self.settings)
 self.testapp = webtest.TestApp(self.app)

 def setUp(self):
 # for tests where we want fresh db on *every* test, we use this
 self.init_db()
 self.init_sessions()
 self.init_app()

 def tearDown(self):
 self.seed_dbs.close()
 self.confirm_dbs.close()


 On Wed, Sep 7, 2016 at 8:37 PM, Jonathan Vanasco 
 wrote:

> can you share the test harness?
>
> --
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>

 --
 You received this message because you are subscribed to the Google
 Groups "sqlalchemy" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to sqlalchemy+unsubscr...@googlegroups.com.
 To post to this group, send email to sqlalchemy@googlegroups.com.
 Visit this group at https://groups.google.com/group/sqlalchemy.
 For more options, 

Re: [sqlalchemy] postgres/pyscopg failing to connect after a certain number of tests, stuck!

[Iain Duncan]

I've discovered too that the issue only happens when all my test suites get
called from one call to discovery. I've I run then in 8 calls (one to each
package, with each holding 1 to 5 test suites of about 1 to 10 tests each),
there's no issue. :-/

iain

On Thu, Sep 8, 2016 at 10:36 AM, Iain Duncan 
wrote:

> Thanks Mike, no we aren't. I think we are just making a new engine per
> suite, not per test though, we have the engine init code in the class level
> setup function, and per test setUp is making new sessions. We do drop the
> tables per test, which I expect we are not doing correctly as I'm a
> postgres newb.
>
> Should I be adding dispose in class level teardown?
>
> thanks!
> Iain
>
>
>
>
> On Thu, Sep 8, 2016 at 10:26 AM, Mike Bayer 
> wrote:
>
>> Engine per test is v inefficient, are you at least calling dispose() on
>> each one in teardown?
>>
>>
>> On Thursday, September 8, 2016, Iain Duncan 
>> wrote:
>>
>>> Hi Jonathan, sure can, here it is below. I think maybe this has
>>> something to do with the fact that we are creating a new engine and
>>> dropping the schema on each pass maybe? Because it seems to *only* happen
>>> when we run about 75 tests in a row.
>>>
>>> Thanks for looking!
>>>
>>> class FunctionalTestSuite(unittest.TestCase):
>>> """
>>> base class for functional test suite with db seeding"
>>> """
>>>
>>> # derived class needs to set this to the package of the app under test
>>> # IE app_module = examp.app
>>> app_package = None
>>>
>>> # derived class needs to put settings in here
>>> settings = {
>>> 'sqlalchemy.url': None
>>> }
>>>
>>> def __init__(self, *args, **kwargs):
>>> super(FunctionalTestSuite, self).__init__(*args, **kwargs)
>>>
>>> @classmethod
>>> def setUpClass(cls):
>>> "class level setUp creates tables"
>>> configure_mappers()
>>> # separate session factory for our test runner
>>> cls.engine = engine_from_config(cls.settings, prefix='sqlalchemy.')
>>> cls.session_factory = sessionmaker(bind=cls.engine)
>>>
>>> @classmethod
>>> def init_db(cls):
>>> "initialize the database and create sessions, normally called from 
>>> setUp"
>>> # drop and recreate tables once per test
>>> # NB: this has to be done differently for postgres because it's a 
>>> constraint nazi
>>> if 'postgres' in cls.settings['sqlalchemy.url']:
>>> cls.engine.execute("drop owned by semaphore")
>>> cls.engine.execute("create schema if not exists public")
>>> else:
>>> Base.metadata.drop_all(bind=cls.engine)
>>> Base.metadata.create_all(bind=cls.engine)
>>>
>>> def init_sessions(self):
>>> "init seed and confirm sessions, normally called from setUp"
>>> # create sessions for seeding and separate for confirming
>>> self.seed_dbs = self.session_factory()
>>> self.confirm_dbs = self.session_factory()
>>>
>>> def init_app(self):
>>> "init test wsgi app, normally called from setUp"
>>> self.app = self.app_package.main({}, **self.settings)
>>> self.testapp = webtest.TestApp(self.app)
>>>
>>> def setUp(self):
>>> # for tests where we want fresh db on *every* test, we use this
>>> self.init_db()
>>> self.init_sessions()
>>> self.init_app()
>>>
>>> def tearDown(self):
>>> self.seed_dbs.close()
>>> self.confirm_dbs.close()
>>>
>>>
>>> On Wed, Sep 7, 2016 at 8:37 PM, Jonathan Vanasco 
>>> wrote:
>>>
 can you share the test harness?

 --
 You received this message because you are subscribed to the Google
 Groups "sqlalchemy" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to sqlalchemy+unsubscr...@googlegroups.com.
 To post to this group, send email to sqlalchemy@googlegroups.com.
 Visit this group at https://groups.google.com/group/sqlalchemy.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to sqlalchemy+unsubscr...@googlegroups.com.
>>> To post to this group, send email to sqlalchemy@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/sqlalchemy.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to sqlalchemy+unsubscr...@googlegroups.com.
>> To post to this group, send email to sqlalchemy@googlegroups.com.
>> Visit this group at https://groups.google.com/group/sqlalchemy.
>> 

Re: [sqlalchemy] postgres/pyscopg failing to connect after a certain number of tests, stuck!

[Iain Duncan]

Thanks Mike, no we aren't. I think we are just making a new engine per
suite, not per test though, we have the engine init code in the class level
setup function, and per test setUp is making new sessions. We do drop the
tables per test, which I expect we are not doing correctly as I'm a
postgres newb.

Should I be adding dispose in class level teardown?

thanks!
Iain




On Thu, Sep 8, 2016 at 10:26 AM, Mike Bayer 
wrote:

> Engine per test is v inefficient, are you at least calling dispose() on
> each one in teardown?
>
>
> On Thursday, September 8, 2016, Iain Duncan 
> wrote:
>
>> Hi Jonathan, sure can, here it is below. I think maybe this has something
>> to do with the fact that we are creating a new engine and dropping the
>> schema on each pass maybe? Because it seems to *only* happen when we run
>> about 75 tests in a row.
>>
>> Thanks for looking!
>>
>> class FunctionalTestSuite(unittest.TestCase):
>> """
>> base class for functional test suite with db seeding"
>> """
>>
>> # derived class needs to set this to the package of the app under test
>> # IE app_module = examp.app
>> app_package = None
>>
>> # derived class needs to put settings in here
>> settings = {
>> 'sqlalchemy.url': None
>> }
>>
>> def __init__(self, *args, **kwargs):
>> super(FunctionalTestSuite, self).__init__(*args, **kwargs)
>>
>> @classmethod
>> def setUpClass(cls):
>> "class level setUp creates tables"
>> configure_mappers()
>> # separate session factory for our test runner
>> cls.engine = engine_from_config(cls.settings, prefix='sqlalchemy.')
>> cls.session_factory = sessionmaker(bind=cls.engine)
>>
>> @classmethod
>> def init_db(cls):
>> "initialize the database and create sessions, normally called from 
>> setUp"
>> # drop and recreate tables once per test
>> # NB: this has to be done differently for postgres because it's a 
>> constraint nazi
>> if 'postgres' in cls.settings['sqlalchemy.url']:
>> cls.engine.execute("drop owned by semaphore")
>> cls.engine.execute("create schema if not exists public")
>> else:
>> Base.metadata.drop_all(bind=cls.engine)
>> Base.metadata.create_all(bind=cls.engine)
>>
>> def init_sessions(self):
>> "init seed and confirm sessions, normally called from setUp"
>> # create sessions for seeding and separate for confirming
>> self.seed_dbs = self.session_factory()
>> self.confirm_dbs = self.session_factory()
>>
>> def init_app(self):
>> "init test wsgi app, normally called from setUp"
>> self.app = self.app_package.main({}, **self.settings)
>> self.testapp = webtest.TestApp(self.app)
>>
>> def setUp(self):
>> # for tests where we want fresh db on *every* test, we use this
>> self.init_db()
>> self.init_sessions()
>> self.init_app()
>>
>> def tearDown(self):
>> self.seed_dbs.close()
>> self.confirm_dbs.close()
>>
>>
>> On Wed, Sep 7, 2016 at 8:37 PM, Jonathan Vanasco 
>> wrote:
>>
>>> can you share the test harness?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to sqlalchemy+unsubscr...@googlegroups.com.
>>> To post to this group, send email to sqlalchemy@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/sqlalchemy.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to sqlalchemy+unsubscr...@googlegroups.com.
>> To post to this group, send email to sqlalchemy@googlegroups.com.
>> Visit this group at https://groups.google.com/group/sqlalchemy.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit 

Re: [sqlalchemy] postgres/pyscopg failing to connect after a certain number of tests, stuck!

[Mike Bayer]

Engine per test is v inefficient, are you at least calling dispose() on
each one in teardown?

On Thursday, September 8, 2016, Iain Duncan 
wrote:

> Hi Jonathan, sure can, here it is below. I think maybe this has something
> to do with the fact that we are creating a new engine and dropping the
> schema on each pass maybe? Because it seems to *only* happen when we run
> about 75 tests in a row.
>
> Thanks for looking!
>
> class FunctionalTestSuite(unittest.TestCase):
> """
> base class for functional test suite with db seeding"
> """
>
> # derived class needs to set this to the package of the app under test
> # IE app_module = examp.app
> app_package = None
>
> # derived class needs to put settings in here
> settings = {
> 'sqlalchemy.url': None
> }
>
> def __init__(self, *args, **kwargs):
> super(FunctionalTestSuite, self).__init__(*args, **kwargs)
>
> @classmethod
> def setUpClass(cls):
> "class level setUp creates tables"
> configure_mappers()
> # separate session factory for our test runner
> cls.engine = engine_from_config(cls.settings, prefix='sqlalchemy.')
> cls.session_factory = sessionmaker(bind=cls.engine)
>
> @classmethod
> def init_db(cls):
> "initialize the database and create sessions, normally called from 
> setUp"
> # drop and recreate tables once per test
> # NB: this has to be done differently for postgres because it's a 
> constraint nazi
> if 'postgres' in cls.settings['sqlalchemy.url']:
> cls.engine.execute("drop owned by semaphore")
> cls.engine.execute("create schema if not exists public")
> else:
> Base.metadata.drop_all(bind=cls.engine)
> Base.metadata.create_all(bind=cls.engine)
>
> def init_sessions(self):
> "init seed and confirm sessions, normally called from setUp"
> # create sessions for seeding and separate for confirming
> self.seed_dbs = self.session_factory()
> self.confirm_dbs = self.session_factory()
>
> def init_app(self):
> "init test wsgi app, normally called from setUp"
> self.app = self.app_package.main({}, **self.settings)
> self.testapp = webtest.TestApp(self.app)
>
> def setUp(self):
> # for tests where we want fresh db on *every* test, we use this
> self.init_db()
> self.init_sessions()
> self.init_app()
>
> def tearDown(self):
> self.seed_dbs.close()
> self.confirm_dbs.close()
>
>
> On Wed, Sep 7, 2016 at 8:37 PM, Jonathan Vanasco  > wrote:
>
>> can you share the test harness?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to sqlalchemy+unsubscr...@googlegroups.com
>> 
>> .
>> To post to this group, send email to sqlalchemy@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/sqlalchemy.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sqlalchemy+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to sqlalchemy@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.