Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
---+-
 Reporter:  kthhrv |Owner:  nobody
 Type:  Bug|   Status:  closed
Component:  Testing framework  |  Version:  1.2
 Severity:  Normal |   Resolution:  needsinfo
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+-
Changes (by Jacob Walls):

 * status:  new => closed
 * resolution:   => needsinfo


Comment:

 It's likely that the use case from comment:14 is solved through the
 combination of setting `'TEST': {'MIGRATE': False}` (new in 3.1) and the
 fix for #23273 (merged for 4.1) to skip creating the `django_migrations`
 table.

 Setting back to `needsinfo` in case someone has details to add around the
 interaction of read-only databases with the test runner.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b108821679e6902ae9bdbd2d17dc5e35%40djangoproject.com.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
---+
 Reporter:  kthhrv |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Testing framework  |  Version:  1.2
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by Carl Marshall):

 * status:  closed => new
 * resolution:  needsinfo =>


Comment:

 I'd like to add some more to this again as I'm having this problem with
 Django 2.1.

 I have some large Oracle DBs which are read only, and my app is providing
 an API to. All my models are set to be `managed=False`, and I've set the
 databases to have:

 {{{
 'TEST': {
 'CREATE_DB': False,
 'CREATE_USER': False,
 'USER': env('DATABASE_USER_DEV'),
 'PASSWORD': env('DATABASE_PASSWORD_DEV'),
 }
 }}}

 However, running `python manage.py test` results in
 `django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create
 the django_migrations table (ORA-02000: missing ALWAYS keyword)` which
 suggests to me that we need a "read-only" database option that will stop
 all attempts to generate data on these connections.

 For reference, the app uses a local sqlite or postgress DB for the data it
 can store (and the standard Django apps) as the default database, but
 there is no scope to be able to get write access to these external
 databases, or have them duplicated for testing in any meaningful way (and
 I understand Russell Keith-Mageen concern about live data in tests).

 I'm not sure what the best solution here would be, for example:

 * Using Create DB as False doesn't always mean you can't test the database
 * Tests need to look for any table on the database that is managed, if
 none found, treat as read-only (seems like an expensive option)
 * Have a READ_ONLY test flag that would cause the standard tests to skip
 some of the write based tests

 The main output I want is to be able to run my tests (including the
 standard ones from Django and other packages) without the early
 termination of the test script because of the migrations exception.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.34ef3669a1b25a737da092b6c746b5d9%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
---+-
 Reporter:  kthhrv |Owner:  nobody
 Type:  Bug|   Status:  closed
Component:  Testing framework  |  Version:  1.2
 Severity:  Normal |   Resolution:  needsinfo
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+-
Changes (by shaib):

 * status:  new => closed
 * cc: shaib (added)
 * resolution:   => needsinfo


Comment:

 6 years after a general workaround was suggested, and almost two years
 since the last comment, I think that anybody who wants improvement on this
 should specify exactly what they need and why.

 In particular -- the combination of `--keepdb` and database routers
 disallowing migrations on the read-only database will stop any automatic
 schema changes, and using Luke's suggestions for test-cases should stop
 data changes as well.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.cb62f1d094096f72da23846dacc4587a%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
---+
 Reporter:  kthhrv |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Testing framework  |  Version:  1.2
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by gchp):

 * cc: gregchapple1@… (added)


Comment:

 As of b7aa7c4ab4372d2b7994d252c8bc87f77dd217ae you can use the `--keepdb`
 parameter to use an existing database rather than creating a new one for
 testing. This sounds more or less like what shai is describing above. It
 will run any outstanding migrations, and flush the data, but ultimately
 the database will be preserved at the end of the run.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.f4437495645624ac22cc01267b9d0da2%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
---+
 Reporter:  kthhrv |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Testing framework  |  Version:  1.2
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by shai):

 The Oracle backend supports a test parameter `CREATE_DB` that tells it not
 to create the (tablespace, in this context equivalent to database) for
 testing but rather use an existing one. It still tries to create tables,
 but I guess the `managed=False` setting would take care of that in this
 context.

 I would like to make that parameter apply to all database backends.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b33f1c32db777cb0f0289557c3fd240f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
---+
 Reporter:  kthhrv |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Testing framework  |  Version:  1.2
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by RK):

 What about if it were possible to specify in the database parameters that
 the test runner should create the test databases using a different
 database server config entirely? Eg:
 {{{
 DATABASES = {
 'default': {
 'ENGINE': 'django.db.backends.mysql',
 'NAME': 'myproject',
 'HOST': 'maindb',
  # ... plus some other settings
 },
 'readonly': {
 'ENGINE': 'django.db.backends.mysql',
 'NAME': 'myproject',
 'HOST': 'readonlydb',
 'TEST_DATABASE': 'default'
 # ... plus some other settings
 }
 }
 }}}
 This would trigger the test runner to create another test database on the
 "default" server, for use with anything during the test which tries to
 access the "readonly" database.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.81d97fd9e442a21b4ced6b8bf67e8971%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
---+
 Reporter:  kthhrv |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Testing framework  |  Version:  1.2
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by oinopion):

 * cc: tomek@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.5c26f7dc51a6fa36e037197fdc4199ae%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
+---
  Reporter:  kthhrv | Owner:  nobody
Status:  new| Milestone:
 Component:  Testing framework  |   Version:  1.2   
Resolution: |  Keywords:
 Stage:  Accepted   | Has_patch:  0 
Needs_docs:  0  |   Needs_tests:  0 
Needs_better_patch:  0  |  
+---
Comment (by russellm):

 #14339 was a ticket requesting the alternate use case I described.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
+---
  Reporter:  kthhrv | Owner:  nobody
Status:  new| Milestone:
 Component:  Testing framework  |   Version:  1.2   
Resolution: |  Keywords:
 Stage:  Accepted   | Has_patch:  0 
Needs_docs:  0  |   Needs_tests:  0 
Needs_better_patch:  0  |  
+---
Changes (by russellm):

  * stage:  Unreviewed => Accepted

Comment:

 This sort of thing makes my teeth itch, because it's *really* bad testing
 practice -- tests should be repeatable, and anything with live data, by
 definition, isn't guaranteed repeatable.

 A more interesting, but related use case is when you don't have permission
 to create or destroy databases, but you do have access to a database that
 you can use for testing purposes. Since solving this use case essentially
 implies solving the use case for this ticket, I'll mark this accepted.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
+---
  Reporter:  kthhrv | Owner:  nobody
Status:  new| Milestone:
 Component:  Testing framework  |   Version:  1.2   
Resolution: |  Keywords:
 Stage:  Unreviewed | Has_patch:  0 
Needs_docs:  0  |   Needs_tests:  0 
Needs_better_patch:  0  |  
+---
Comment (by jonathan...@gmail.com):

 brilliant formatting on my above statement... sorry about that

 Code change above should read:

 {{{
   def setup_databases(self, **kwargs):
 from django.db import connections
 old_names = []
 mirrors = []
 for alias in connections:
 connection = connections[alias]
 +   # If the alias name is contained within this setting,
 +   # just skip round the loop.
 +   # This causes all tests to occur against the aliased
 +   # connection, and not create a test db.
 +   try:
 +   if alias in settings.RUN_TESTS_ON_LIVE_DB:
 +   continue
 +   except:
 +   pass

 # If the database is a test mirror, redirect it's connection
 # instead of creating a test database.
 if connection.settings_dict['TEST_MIRROR']:
 mirrors.append((alias, connection))
 mirror_alias = connection.settings_dict['TEST_MIRROR']
 connections._connections[alias] =
 connections[mirror_alias]
 else:

 old_names.append((connection,
 connection.settings_dict['NAME']))
 connection.creation.create_test_db(self.verbosity,
 autoclobber=not self.interactive)
 return old_names, mirrors

 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
+---
  Reporter:  kthhrv | Owner:  nobody
Status:  new| Milestone:
 Component:  Testing framework  |   Version:  1.2   
Resolution: |  Keywords:
 Stage:  Unreviewed | Has_patch:  0 
Needs_docs:  0  |   Needs_tests:  0 
Needs_better_patch:  0  |  
+---
Comment (by jonathan...@gmail.com):

 A simple hack that seems to work for this is to modify the action of
 setup_databases in django/test/simple.py

  def setup_databases(self, **kwargs):
 from django.db import connections
 old_names = []
 mirrors = []
 for alias in connections:
 connection = connections[alias]
 +   # If the alias name is contained within this setting,
 +   # just skip round the loop.
 +   # This causes all tests to occur against the aliased
 +   # connection, and not create a test db.
 +   try:
 +   if alias in settings.RUN_TESTS_ON_LIVE_DB:
 +   continue
 +   except:
 +   pass

 # If the database is a test mirror, redirect it's connection
 # instead of creating a test database.
 if connection.settings_dict['TEST_MIRROR']:
 mirrors.append((alias, connection))
 mirror_alias = connection.settings_dict['TEST_MIRROR']
 connections._connections[alias] =
 connections[mirror_alias]
 else:

 old_names.append((connection,
 connection.settings_dict['NAME']))
 connection.creation.create_test_db(self.verbosity,
 autoclobber=not self.interactive)
 return old_names, mirrors

 and then place the names of the connections inside the variable in your
 settings.py:

 RUN_TESTS_ON_LIVE_DB = ['dbname',]

 This seems to force the tests to run against the live database, write
 operations will simply fail due to permissions and one no longer needs
 complex alternative db setups.

 Hack..

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
+---
  Reporter:  kthhrv | Owner:  nobody
Status:  new| Milestone:
 Component:  Testing framework  |   Version:  1.2   
Resolution: |  Keywords:
 Stage:  Unreviewed | Has_patch:  0 
Needs_docs:  0  |   Needs_tests:  0 
Needs_better_patch:  0  |  
+---
Changes (by lukeplant):

  * needs_better_patch:  => 0
  * needs_tests:  => 0
  * needs_docs:  => 0

Comment:

 What kind of workaround do you envisage?

 If you are looking for read-write tests, the normal work-around for a
 read-only database is to create a separate, writeable, blank database and
 use that for your tests. You will need a different settings file, or a
 settings file that has some way of knowing to use a different DB when you
 are in test mode.

 If you cannot create a new database, and cannot create any tables in your
 existing one, you would have to run on the live data. Assuming this
 database is being used for something, you are presumably not going to want
 to do any writes to the DB at all, which limits you to read-only tests on
 your actual live data. Is that what you are wanting?  It is already
 possible - just use the !TestCase and test runners provided by unittest
 rather than using the Django subclasses and manage.py, and your tests will
 connect to the normal database. You will miss out on some goodness from
 Django's !TestCase, but not much, because most of it is to do with writing
 to databases.  You can also instantiate your own Client for testing views,
 so you don't miss out on that.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

[Django] #14296: 'manage.py test' failing for apps that access read-only databases

[Django]

#14296: 'manage.py test' failing for apps that access read-only databases
---+
 Reporter:  kthhrv |   Owner:  nobody
   Status:  new|   Milestone:
Component:  Testing framework  | Version:  1.2   
 Keywords: |   Stage:  Unreviewed
Has_patch:  0  |  
---+
 http://groups.google.com/group/django-
 
users/browse_thread/thread/466ff5d2ab57c5a1/8131fd4a7d7a7cb6?lnk=gst&q=testing#8131fd4a7d7a7cb6

 searching 'Testing framework' component i can't find a ticket for this
 issue so here it is.

 I have an app that accesses two databases one of which is read-only, when
 i run 'manage.py test ' i get an insufficient privileges error while
 its trying to create/destroy the tables on the read-only database. This
 causes the test run to fail then and there.

 all the models in the read-only app are marked managed=False.

 we can't always have RW access to legacy databases.

 is there a workaround for this issue? i'd love to start running tests on
 this app.

 thx
 Keith

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.