Re: How to view the generated SQL for test models?

[Brutus Schraiber]

Hi Michal,


# The `ProgrammingError` Exception

Am 18.01.2016 um 20:23 schrieb Michal Petrucha:
> Could you perhaps post the full traceback of the ProgrammingError?

The original exception was (full traceback at the end of the mail):

```
django.db.utils.ProgrammingError: (1064, "You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version for 
the
right syntax to use near 'None, None) NOT NULL)' at line 1")
```

Since the error message is kinda meh without SQL to look at, I wanted to 
know how to see the SQL Django generates.


> It smells a bit like your test models don't get imported early enough to
> be fully initialized before database creation happens. And this might be
> caused precisely by the fact that your models are not inside models.py.

Having no SQL to look at, made the debugging slightly tedious, but I found 
the error in the `core.PlaytimeMixinTestModel` — could narrow it down a bit 
thanks to verbose mode (see the full traceback below):

A `Decimal Field` was missing two arguments that where necessary 
(`max_digits` and `decimal_places`). This slipped by unnoticed, when using 
the SQLite backend local but raises an Error on the MySQL DB on the staging 
servers.

After fixing it, the test run fine on MySQL too.


# `test.py`

> As for your original question, unless you put all your models into
> models.py of an app inside INSTALLED_APPS, or at least import them from
> within such a models.py module, they won't be considered at all when
> generating migrations, or when dumping out the creation SQL.
>
> When you run a management command, ``django.setup()`` is called, which
> populates the model registry by importing the models module of each
> installed app, not much else. Particularly not app.tests. So there's no
> way for the ``sql`` management command to be aware of your test models.
>
> The tests module is only imported when running tests by the test runner
> (unless you import it from somewhere else in your app, but that's not a
> very usual thing to do). That happens after ``django.setup()`` has
> finished, at which point models should already be properly initialized,
> which, obviously, is not the case with models in tests.py.
>
> In theory, it should work – the test runner tries to import all test
> modlues before setting up the test database, so in case everything is
> correct, all deferred model setup actions should still finish before
> creating the database. However, if there is some kind of error, they might
> not, and that means it will be harder to debug those situations.

Thanks for the detailed info. Makes some things clearer.

I wasn't aware why it worked — but it usually does. It's common style for 
the team I work with to put models that are used only for testing of an in 
a `test.py` file. BTW: even when the actual test are spread to a test 
package.

I couldn't find any documentation about it though.


> Really, do yourself a favor, create a separate app for your tests, with
> its own models, and its own settings. That is the most painless way to
> have additional testing models.
>
> You can take inspiration from the following section of the docs:
> 
https://docs.djangoproject.com/en/1.9/topics/testing/advanced/#using-the-django-test-runner-to-test-reusable-applications

It is quite a bit overhead — even in it's simplest form — but sounds like a 
clean way to handle such cases. I will consider it in the future.

Thanks again for your answer
-brutus



# Full Traceback

```
$ django-admin test app -v 3
Creating test database for alias 'default' ('yogalessontv_test')...
Got an error creating the test database: (1007, "Can't create database 
'yogalessontv_test'; database exists")
Type 'yes' if you would like to try deleting the test database 
'yogalessontv_test', or 'no' to cancel: yes
Destroying old test database 'default'...
Operations to perform:
Synchronize unmigrated apps: streambox, profiles, core, suit, contact, 
allauth, messages, staticfiles, debug_toolbar
Apply all migrations: admin, auth, carousel, sites, pages, socialaccount, 
blog, contenttypes, account, sessions, videos
Synchronizing apps without migrations:
Running pre-migrate handlers for application core
Running pre-migrate handlers for application pages
Running pre-migrate handlers for application blog
Running pre-migrate handlers for application videos
Running pre-migrate handlers for application carousel
Running pre-migrate handlers for application suit
Running pre-migrate handlers for application admin
Running pre-migrate handlers for application auth
Running pre-migrate handlers for application contenttypes
Running pre-migrate handlers for application sessions
Running pre-migrate handlers for application sites
Running pre-migrate handlers for application allauth
Running pre-migrate handlers for application account
Running pre-migrate handlers for application socialaccount
Running pre-migrate handlers for application debug_toolbar
Creating tables...
Creating 

Re: How to view the generated SQL for test models?

['Brutus' via Django users]

Hi Michal,


# The `ProgrammingError` Exception

Am 18.01.2016 um 20:23 schrieb Michal Petrucha:

Could you perhaps post the full traceback of the ProgrammingError?


The original exception was (full traceback at the end of the mail):

```
django.db.utils.ProgrammingError: (1064, "You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version for the
right syntax to use near 'None, None) NOT NULL)' at line 1")
```

Since the error message is kinda meh without SQL to look at, I wanted to know 
how to see the SQL Django generates.




It smells a bit like your test models don't get imported early enough to
be fully initialized before database creation happens. And this might be
caused precisely by the fact that your models are not inside models.py.


Having no SQL to look at, made the debugging slightly tedious, but I found the 
error in the `core.PlaytimeMixinTestModel` — could narrow it down a bit thanks 
to verbose mode (see the full traceback below):


A `Decimal Field` was missing two arguments that where necessary (`max_digits` 
and `decimal_places`). This slipped by unnoticed, when using the SQLite 
backend local but raises an Error on the MySQL DB on the staging servers.


After fixing it, the test run fine on MySQL too.


# `test.py`


As for your original question, unless you put all your models into
models.py of an app inside INSTALLED_APPS, or at least import them from
within such a models.py module, they won't be considered at all when
generating migrations, or when dumping out the creation SQL.

When you run a management command, ``django.setup()`` is called, which
populates the model registry by importing the models module of each
installed app, not much else. Particularly not app.tests. So there's no
way for the ``sql`` management command to be aware of your test models.

The tests module is only imported when running tests by the test runner
(unless you import it from somewhere else in your app, but that's not a
very usual thing to do). That happens after ``django.setup()`` has
finished, at which point models should already be properly initialized,
which, obviously, is not the case with models in tests.py.

In theory, it should work – the test runner tries to import all test
modlues before setting up the test database, so in case everything is
correct, all deferred model setup actions should still finish before
creating the database. However, if there is some kind of error, they might
not, and that means it will be harder to debug those situations.


Thanks for the detailed info. Makes some things clearer.

I wasn't aware why it worked — but it usually does. It's common style for the 
team I work with to put models that are used only for testing of an in a 
`test.py` file. BTW: even when the actual test are spread to a test package.


I couldn't find any documentation about it though.



Really, do yourself a favor, create a separate app for your tests, with
its own models, and its own settings. That is the most painless way to
have additional testing models.

You can take inspiration from the following section of the docs:
https://docs.djangoproject.com/en/1.9/topics/testing/advanced/#using-the-django-test-runner-to-test-reusable-applications


It is quite a bit overhead — even in it's simplest form — but sounds like a 
clean way to handle such cases. I will consider it in the future.


Thanks again for your answer
-brutus



# Full Traceback

```
$ django-admin test app -v 3
Creating test database for alias 'default' ('yogalessontv_test')...
Got an error creating the test database: (1007, "Can't create database 
'yogalessontv_test'; database exists")
Type 'yes' if you would like to try deleting the test database 
'yogalessontv_test', or 'no' to cancel: yes

Destroying old test database 'default'...
Operations to perform:
  Synchronize unmigrated apps: streambox, profiles, core, suit, contact, 
allauth, messages, staticfiles, debug_toolbar
  Apply all migrations: admin, auth, carousel, sites, pages, socialaccount, 
blog, contenttypes, account, sessions, videos

Synchronizing apps without migrations:
Running pre-migrate handlers for application core
Running pre-migrate handlers for application pages
Running pre-migrate handlers for application blog
Running pre-migrate handlers for application videos
Running pre-migrate handlers for application carousel
Running pre-migrate handlers for application suit
Running pre-migrate handlers for application admin
Running pre-migrate handlers for application auth
Running pre-migrate handlers for application contenttypes
Running pre-migrate handlers for application sessions
Running pre-migrate handlers for application sites
Running pre-migrate handlers for application allauth
Running pre-migrate handlers for application account
Running pre-migrate handlers for application socialaccount
Running pre-migrate handlers for application debug_toolbar
  Creating tables...
Creating table core_trimstringsmixintestmodel
   

Re: How to view the generated SQL for test models?

[Michal Petrucha]

On Mon, Jan 18, 2016 at 04:25:59AM -0800, Brutus Schraiber wrote:
> Am Samstag, 16. Januar 2016 22:23:06 UTC+1 schrieb Vijay Khemlani:
> >
> > At least to me it doesn't make a lot of sense to define new models in your 
> > tests, but I also don't know the particular problem you are solving.
> >
> 
> I have an app, that defines a couple of *abstract models* and *mixins* in 
> it's `models.py`, that are only used in other apps.
> 
> To test those abstract models and mixins I created concrete models (from 
> `django.db.models.Model`), that use these abstract models and mixins, in 
> `test.py`.
> 
> I created them in `test.py`, so that the test models don't get created in 
> the DB outside of test running. I thought this the way to handle such 
> things in Django?
> 
> At least for me this generally works fine. In a special case I get an 
> `django.db.utils.ProgrammingError` from one of my test models though. To 
> debug this, I wanted to take a look at the SQL Django generated for it.

Hi Brutus,

Could you perhaps post the full traceback of the ProgrammingError? It
smells a bit like your test models don't get imported early enough to
be fully initialized before database creation happens. And this might
be caused precisely by the fact that your models are not inside
models.py.

As for your original question, unless you put all your models into
models.py of an app inside INSTALLED_APPS, or at least import them
from within such a models.py module, they won't be considered at all
when generating migrations, or when dumping out the creation SQL.

When you run a management command, ``django.setup()`` is called, which
populates the model registry by importing the models module of each
installed app, not much else. Particularly not app.tests. So there's
no way for the ``sql`` management command to be aware of your test
models.

The tests module is only imported when running tests by the test
runner (unless you import it from somewhere else in your app, but
that's not a very usual thing to do). That happens after
``django.setup()`` has finished, at which point models should already
be properly initialized, which, obviously, is not the case with models
in tests.py.

In theory, it should work – the test runner tries to import all test
modlues before setting up the test database, so in case everything is
correct, all deferred model setup actions should still finish before
creating the database. However, if there is some kind of error, they
might not, and that means it will be harder to debug those situations.

Really, do yourself a favor, create a separate app for your tests,
with its own models, and its own settings. That is the most painless
way to have additional testing models.

You can take inspiration from the following section of the docs:
https://docs.djangoproject.com/en/1.9/topics/testing/advanced/#using-the-django-test-runner-to-test-reusable-applications

Regards,

Michal

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20160118192319.GZ20308%40konk.org.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: Digital signature

Re: How to view the generated SQL for test models?

[Brutus Schraiber]

Am Samstag, 16. Januar 2016 22:23:06 UTC+1 schrieb Vijay Khemlani:
>
> At least to me it doesn't make a lot of sense to define new models in your 
> tests, but I also don't know the particular problem you are solving.
>

I have an app, that defines a couple of *abstract models* and *mixins* in 
it's `models.py`, that are only used in other apps.

To test those abstract models and mixins I created concrete models (from 
`django.db.models.Model`), that use these abstract models and mixins, in 
`test.py`.

I created them in `test.py`, so that the test models don't get created in 
the DB outside of test running. I thought this the way to handle such 
things in Django?

At least for me this generally works fine. In a special case I get an 
`django.db.utils.ProgrammingError` from one of my test models though. To 
debug this, I wanted to take a look at the SQL Django generated for it.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/16d20af4-064a-46b7-98ae-b5a9e8105e15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to view the generated SQL for test models?

[Brutus Schraiber]

Am Samstag, 16. Januar 2016 22:23:06 UTC+1 schrieb Vijay Khemlani:
>
> At least to me it doesn't make a lot of sense to define new models in your 
> tests, but I also don't know the particular problem you are solving.
>

I have an app, that defines a couple of *abstract models* and *mixins* in 
it's `models.py`, that are only used in other apps.

To test those abstract models and mixins I created concrete models (from 
`django.db.models.Model`), that use these abstract models and mixins, in 
`test.py`.

I created them in `test.py`, so that the test models don't get created in 
the DB outside of test running. I thought this the way to handle such 
things in Django?

At least for me this generally works fine. In a special case I get an 
`django.db.utils.ProgrammingError` from one of my test models though. To 
debug this, I wanted to take a look at the SQL Django generated for it.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/384d01d0-711e-4310-bc1a-a887343e1990%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to view the generated SQL for test models?

[Michal Petrucha]

On Mon, Jan 11, 2016 at 07:06:22AM -0800, Brutus Schraiber wrote:
> Hi all…
> 
> I have some models in a `test.py` file. How can I view the SQL that will be 
> generated for them?
> 
> Something like `django-admin sql` or similar but for models defined in 
> `test.py` not in `models.py`.

Django always expects models in the models module. If you need a
separate set of models just for tests, you'll need to set up a
separate test app with any additional test models.

Good luck,
Michal

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20160116231915.GX20308%40konk.org.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: Digital signature

Re: How to view the generated SQL for test models?

[Vijay Khemlani]

At least to me it doesn't make a lot of sense to define new models in your
tests, but I also don't know the particular problem you are solving.

On Sat, Jan 16, 2016 at 1:27 PM, Brutus Schraiber  wrote:

> Wrong place to ask such things?
>
> This doesn't seem too uncommon or complicated, or am I missing the point?
>
> Am Montag, 11. Januar 2016 16:06:22 UTC+1 schrieb Brutus Schraiber:
>>
>> Hi all…
>>
>> I have some models in a `test.py` file. How can I view the SQL that will
>> be generated for them?
>>
>> Something like `django-admin sql` or similar but for models defined in
>> `test.py` not in `models.py`.
>>
>> Regards,
>> -brutus
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/d74d74d4-770b-464a-bcf0-55b90424a724%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CALn3ei13bKPpM0yDzYKCvmuDS_gm-9O_AH_J0nDiKQqm2TARXw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to view the generated SQL for test models?

[Brutus Schraiber]

Wrong place to ask such things?

This doesn't seem too uncommon or complicated, or am I missing the point?

Am Montag, 11. Januar 2016 16:06:22 UTC+1 schrieb Brutus Schraiber:
>
> Hi all…
>
> I have some models in a `test.py` file. How can I view the SQL that will 
> be generated for them?
>
> Something like `django-admin sql` or similar but for models defined in 
> `test.py` not in `models.py`.
>
> Regards,
> -brutus
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/d74d74d4-770b-464a-bcf0-55b90424a724%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.