Re: [Django] #25329: _nodb_connection is open for an entire test suite run

[Django]

#25329: _nodb_connection is open for an entire test suite run
-+-
 Reporter:  mewtaylor|Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Testing framework|  Version:  master
 Severity:  Normal   |   Resolution:  fixed
 Keywords:  nodb, testing, test  | Triage Stage:  Ready for
  runner |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Claude Paroz ):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 In [changeset:"b2f6e421a332ce45cf0f1f2f6bd5962765a013f1" b2f6e42]:
 {{{
 #!CommitTicketReference repository=""
 revision="b2f6e421a332ce45cf0f1f2f6bd5962765a013f1"
 Fixed #25329 -- Prevented _nodb_connection from being left open
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.14d2945bea5c78f438fbe16490a6f53e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25329: _nodb_connection is open for an entire test suite run

[Django]

#25329: _nodb_connection is open for an entire test suite run
-+-
 Reporter:  mewtaylor|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Testing framework|  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:  nodb, testing, test  | Triage Stage:  Ready for
  runner |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by timgraham):

 * needs_better_patch:  1 => 0
 * stage:  Accepted => Ready for checkin


Comment:

 Claude, any concerns?

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.90e048b5e5dfbc3549eba64633c9f4e7%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25329: _nodb_connection is open for an entire test suite run

[Django]

#25329: _nodb_connection is open for an entire test suite run
-+-
 Reporter:  mewtaylor|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Testing framework|  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:  nodb, testing, test  | Triage Stage:  Accepted
  runner |
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by timgraham):

 * needs_better_patch:  0 => 1


Comment:

 Still a test failure on the patch.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.e9f311af8227f8cf2f4f147a67057e48%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25329: _nodb_connection is open for an entire test suite run

[Django]

#25329: _nodb_connection is open for an entire test suite run
-+-
 Reporter:  mewtaylor|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Testing framework|  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:  nodb, testing, test  | Triage Stage:  Accepted
  runner |
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by adamchainz):

 * has_patch:  0 => 1


Comment:

 Pull request: https://github.com/django/django/pull/5262

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.6d40b0f45d50f34d4f7d82003df91c8f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25329: _nodb_connection is open for an entire test suite run

[Django]

#25329: _nodb_connection is open for an entire test suite run
-+-
 Reporter:  mewtaylor|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Testing framework|  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:  nodb, testing, test  | Triage Stage:  Accepted
  runner |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by adamchainz):

 I can replicate this on 1.8.4:

 {{{
 $ mysql -uroot
 ...
 mysql> set @@global.wait_timeout=10;
 Query OK, 0 rows affected (0.00 sec)

 mysql> Bye
 $ runtests  # 30 seconds
 ...
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/django/db/backends/base/creation.py", line 503, in
 destroy_test_db
 self._destroy_test_db(test_database_name, verbosity)
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/django/db/backends/base/creation.py", line 521, in
 _destroy_test_db
 % self.connection.ops.quote_name(test_database_name))
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/django/db/backends/utils.py", line 64, in execute
 return self.cursor.execute(sql, params)
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/django/db/utils.py", line 97, in __exit__
 six.reraise(dj_exc_type, dj_exc_value, traceback)
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/django/db/backends/utils.py", line 62, in execute
 return self.cursor.execute(sql)
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/django_mysql/monkey_patches.py", line 42, in execute
 return orig_execute(self, sql, args)
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/django/db/backends/mysql/base.py", line 124, in execute
 return self.cursor.execute(query, args)
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/MySQLdb/cursors.py", line 220, in execute
 self.errorhandler(self, exc, value)
   File "/opt/yplan/venv/local/lib/python2.7/site-
 packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
 raise errorvalue
 django.db.utils.OperationalError: (2006, 'MySQL server has gone away')
 }}}

 It definitely is `_nodb_connection`, since `_destroy_test_db` only
 connects to that database:

 {{{
 def _destroy_test_db(self, test_database_name, verbosity):
 """
 Internal implementation - remove the test db tables.
 """
 # Remove the test database to clean up after
 # ourselves. Connect to the previous database (not the test
 database)
 # to do so, because it's not allowed to delete a database while
 being
 # connected to it.
 with self.connection._nodb_connection.cursor() as cursor:
 }}}

 I think `_nodb_connection` should reconnect everytime you access it,
 rather than it being a `cached_property`, which seems like a premature
 optimization to me.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.67d0b43d2a26dfeda5907f1a128f0220%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25329: _nodb_connection is open for an entire test suite run

[Django]

#25329: _nodb_connection is open for an entire test suite run
-+-
 Reporter:  mewtaylor|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Testing framework|  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:  nodb, testing, test  | Triage Stage:  Accepted
  runner |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by mewtaylor):

 Replying to [comment:1 jarshwah]:
 > How certain are you that the nodb connection is to blame?
 I'm relatively certain – I also ran a trace on the tear down method to
 verify that it was looking at the right name for the test database, and it
 was. When I investigated the source of {{{BaseDatabaseCreation}}} further
 I saw that both {{{_create_test_db()}}} and {{{_destroy_test_db()}}} use a
 nodb connection – for create, it uses the line {{{with
 self._nodb_connection.cursor() as cursor}}}, whereas for destroy, it uses
 the line {{{with self.connection._nodb_connection.cursor() as cursor}}}.

 I believe they are retrieving the same connection each time, because even
 though create uses the property on BaseDatabaseCreation itself, that
 property appears to return {{{self.connection._nodb_connection}}} (see
 
[https://github.com/django/django/blob/master/django/db/backends/base/creation.py#L24-L29
 here]). And, since {{{self.connection}}} is set on init, that was what
 made me think it was possible that the connection was being kept open for
 the entire time. Also it appears to be a
 
[https://github.com/django/django/blob/master/django/db/backends/base/base.py#L571-L586
 cached_property] on the connection object too.
 >
 > I think that if the nodb connection can be safely closed, then it should
 be closed. It can be created again to tear down the test database if it's
 needed. Is the nodb connection used after test database creation other
 than for tear down?

 I'm not 100% sure. Inside of {{{BaseDatabaseCreation}}} it doesn't appear
 to be used anywhere else. A cursory search of the repo didn't reveal it
 being used elsewhere, though.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.d702a88ef4193d696cd81b4cd0500d72%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25329: _nodb_connection is open for an entire test suite run

[Django]

#25329: _nodb_connection is open for an entire test suite run
-+-
 Reporter:  mewtaylor|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Testing framework|  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:  nodb, testing, test  | Triage Stage:  Accepted
  runner |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by jarshwah):

 * needs_better_patch:   => 0
 * needs_tests:   => 0
 * version:  1.8 => master
 * needs_docs:   => 0
 * type:  Uncategorized => Bug
 * stage:  Unreviewed => Accepted


Comment:

 I've run into this same issue when running the test suite. It usually
 happens when I'm debugging and have a set_trace waiting around for awhile.
 I assumed it was the regular database connections used throughout the
 tests that were going away though.

 How certain are you that the nodb connection is to blame?

 I think that if the nodb connection can be safely closed, then it should
 be closed. It can be created again to tear down the test database if it's
 needed. Is the nodb connection used after test database creation other
 than for tear down?

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.a86d4b53f534762eb92127be19dfa0ba%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

[Django] #25329: _nodb_connection is open for an entire test suite run

[Django]

#25329: _nodb_connection is open for an entire test suite run
--+
 Reporter:  mewtaylor |  Owner:  nobody
 Type:  Uncategorized | Status:  new
Component:  Testing   |Version:  1.8
  framework   |
 Severity:  Normal|   Keywords:  nodb, testing, test runner
 Triage Stage:  Unreviewed|  Has patch:  0
Easy pickings:  0 |  UI/UX:  0
--+
 I was running into an issue with my test suite in which I would get
 occasionally get an {{{OperationalError: (2006, 'MySQL server has gone
 away'}}} error during test database teardown – i.e. during
 {{{connection.creation.destroy_test_db(old_name, self.verbosity,
 self.keepdb)}}} in the test runner.

 I would often run into this when running the entire test suite, or tests
 against multiple apps in my project – running tests against just one app,
 or one unit test, would not cause this tear down issue. Upon
 investigation, I discovered that by setting {{{wait_timeout}}} on my MySQL
 instance high (it had been set to 180 seconds, so I tried setting it back
 to the default 28800 and running tests again), I was able to prevent the
 error from occurring on runs of the entire test suite, or multi-app tests.

 After a bit of investigation, it appeared to me that part of the issue I
 was experiencing was that the {{{_nodb_connection}}} property on the
 {{{BaseDatabaseCreation}}} object was being set on instantiation, and then
 kept alive for the duration of the tests. If my understanding is indeed
 the case, I was wondering if it might be possible/better to have this
 connection be closed after test database creation (i.e. in
 {{{create_test_db()}}}), and reopened again for tear down, rather than
 being kept open for the entire test duration.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/052.4f27ba08d75e9235f4dc64a39fcda358%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.