Re: A lot of Problems with Migrating (conceptual)

[Jim Illback]

Excellent - thanks much.

On Aug 18, 2017, at 4:02 PM, knbk 
> wrote:

There are various reasons why you should never execute queries at startup. One 
you've seen: it makes it impossible to migrate a new database. Other reasons 
include that the choices are cached and are only renewed when the server 
restarts, and that when running tests, the query is executed before the test 
database is set up and will run on the main database.

The solution in this case is to use a ModelChoiceField[1], i.e.:

category = forms.ModelChoiceField(queryset=Category.objects.all(), 
required=False)

This allows to pass in a queryset which is evaluated whenever the field is used 
to make sure the values are up-to-date. Since querysets are lazy, this won't 
execute any queries on startup.

[1] https://docs.djangoproject.com/en/1.11/ref/forms/fields/#modelchoicefield

On Saturday, August 19, 2017 at 12:14:48 AM UTC+2, subaru_87 wrote:
Here’s a “problem” with migrations that I’ve had. It only occurs on a first 
time installation (like a first-time upload to Heroku, for example).

In forms.py, I have a field dependent upon a column in another table (Category):
category = forms.TypedChoiceField(required=False, 
choices=[(category.pk, 
category.name) for category in Category.objects.all()])

Running makemigrations or migrate if there’s no pre-existing database with a 
table named “Category” gives an error saying as much. However, I’m expecting 
migrate to create the tables required. Therefore, it’s a catch-22.

Of course, the “easy” answer is to comment out the above code, add in:
 category = forms.CharField(max_length=12)
And run the initial migrations. Then, turn around and comment out that same 
code, uncomment the real code shown above, and run migrations again. It works, 
but it seems a bit awkward.

Has anyone else had this issue? Or am I doing something wrong?

Thanks much,
Jim Illback


On Aug 18, 2017, at 1:21 AM, James Bennett  
wrote:

On Thu, Aug 17, 2017 at 1:03 PM, Antonis Christofides 
 wrote:

Second, just to make things clear, the word "migration" has two meanings. The 
original meaning of migration is to switch to another software system (e.g. 
migrate from MySQL to PostgreSQL, or migrate a repository from subversion to 
git). In Django, the term "migration" means something else: to update the 
database to the new schema. In my opinion this terminology is wrong and 
confusing (apparently it comes from Ruby on Rails, but I'm not certain), and a 
better one would have been "dbupdate" or something, but since it's migration 
we'll keep it and you'll have to understand which meaning is meant in each case.

An important thing worth knowing is that Django's migration framework does 
*not* only track the database schema. It also tracks the state of the Python 
classes which define the models, including options which do not affect the 
underlying DB schema, and making such changes to a model will create the need 
for a migration.

This is absolutely necessary, though, because the migration framework generates 
a frozen snapshot of the state at each migration, so that you can access the 
ORM to do things like data updates via the RunPython migration operator. If you 
were to change, say, the default ordering of a model without also generating a 
migration to record that change (even though it has no effect on the database 
schema), you could run into trouble if you later removed a field referenced by 
the old ordering, since one of the intermediate ORM snapshots would attempt to 
order queries by a now-nonexistent column.

--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users...@googlegroups.com.
To post to this group, send email to djang...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAL13Cg_0p1pMtMR2y2eGry1UhQaSRpswntcnMWtHNdPgV1Ph9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
django-users+unsubscr...@googlegroups.com.
To post to this group, send email to 
django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 

Re: A lot of Problems with Migrating (conceptual)

[knbk]

There are various reasons why you should never execute queries at startup. 
One you've seen: it makes it impossible to migrate a new database. Other 
reasons include that the choices are cached and are only renewed when the 
server restarts, and that when running tests, the query is executed before 
the test database is set up and will run on the main database. 

The solution in this case is to use a ModelChoiceField[1], i.e.:

category = forms.ModelChoiceField(queryset=Category.objects.all(), 
required=False)

This allows to pass in a queryset which is evaluated whenever the field is 
used to make sure the values are up-to-date. Since querysets are lazy, this 
won't execute any queries on startup. 

[1] 
https://docs.djangoproject.com/en/1.11/ref/forms/fields/#modelchoicefield

On Saturday, August 19, 2017 at 12:14:48 AM UTC+2, subaru_87 wrote:
>
> Here’s a “problem” with migrations that I’ve had. It only occurs on a 
> first time installation (like a first-time upload to Heroku, for example).
>
> In forms.py, I have a field dependent upon a column in another table 
> (Category):
> category = forms.TypedChoiceField(required=False, choices=[(
> category.pk, category.name) for category in Category.objects.all()])
>
> Running makemigrations or migrate if there’s no pre-existing database with 
> a table named “Category” gives an error saying as much. However, I’m 
> expecting migrate to create the tables required. Therefore, it’s a 
> catch-22. 
>
> Of course, the “easy” answer is to comment out the above code, add in:
>  category = forms.CharField(max_length=12) 
> And run the initial migrations. Then, turn around and comment out that 
> same code, uncomment the real code shown above, and run migrations again. 
> It works, but it seems a bit awkward.
>
> Has anyone else had this issue? Or am I doing something wrong?
>
> Thanks much,
> Jim Illback
>
>
> On Aug 18, 2017, at 1:21 AM, James Bennett  > wrote:
>
> On Thu, Aug 17, 2017 at 1:03 PM, Antonis Christofides <
> ant...@djangodeployment.com > wrote:
>
>> Second, just to make things clear, the word "migration" has two meanings. 
>> The original meaning of migration is to switch to another software system 
>> (e.g. migrate from MySQL to PostgreSQL, or migrate a repository from 
>> subversion to git). In Django, the term "migration" means something else: 
>> to update the database to the new schema. In my opinion this terminology is 
>> wrong and confusing (apparently it comes from Ruby on Rails, but I'm not 
>> certain), and a better one would have been "dbupdate" or something, but 
>> since it's migration we'll keep it and you'll have to understand which 
>> meaning is meant in each case.
>>
>
> An important thing worth knowing is that Django's migration framework does 
> *not* only track the database schema. It also tracks the state of the 
> Python classes which define the models, including options which do not 
> affect the underlying DB schema, and making such changes to a model will 
> create the need for a migration.
>
> This is absolutely necessary, though, because the migration framework 
> generates a frozen snapshot of the state at each migration, so that you can 
> access the ORM to do things like data updates via the RunPython migration 
> operator. If you were to change, say, the default ordering of a model 
> without also generating a migration to record that change (even though it 
> has no effect on the database schema), you could run into trouble if you 
> later removed a field referenced by the old ordering, since one of the 
> intermediate ORM snapshots would attempt to order queries by a 
> now-nonexistent column.
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users...@googlegroups.com .
> To post to this group, send email to djang...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CAL13Cg_0p1pMtMR2y2eGry1UhQaSRpswntcnMWtHNdPgV1Ph9w%40mail.gmail.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/d878688d-00a3-452d-8cb8-956dcf41a52e%40googlegroups.com.
For more options, visit 

Re: A lot of Problems with Migrating (conceptual)

[Jim Illback]

Here’s a “problem” with migrations that I’ve had. It only occurs on a first 
time installation (like a first-time upload to Heroku, for example).

In forms.py, I have a field dependent upon a column in another table (Category):
category = forms.TypedChoiceField(required=False, choices=[(category.pk, 
category.name) for category in Category.objects.all()])

Running makemigrations or migrate if there’s no pre-existing database with a 
table named “Category” gives an error saying as much. However, I’m expecting 
migrate to create the tables required. Therefore, it’s a catch-22.

Of course, the “easy” answer is to comment out the above code, add in:
 category = forms.CharField(max_length=12)
And run the initial migrations. Then, turn around and comment out that same 
code, uncomment the real code shown above, and run migrations again. It works, 
but it seems a bit awkward.

Has anyone else had this issue? Or am I doing something wrong?

Thanks much,
Jim Illback


On Aug 18, 2017, at 1:21 AM, James Bennett 
> wrote:

On Thu, Aug 17, 2017 at 1:03 PM, Antonis Christofides 
> wrote:

Second, just to make things clear, the word "migration" has two meanings. The 
original meaning of migration is to switch to another software system (e.g. 
migrate from MySQL to PostgreSQL, or migrate a repository from subversion to 
git). In Django, the term "migration" means something else: to update the 
database to the new schema. In my opinion this terminology is wrong and 
confusing (apparently it comes from Ruby on Rails, but I'm not certain), and a 
better one would have been "dbupdate" or something, but since it's migration 
we'll keep it and you'll have to understand which meaning is meant in each case.

An important thing worth knowing is that Django's migration framework does 
*not* only track the database schema. It also tracks the state of the Python 
classes which define the models, including options which do not affect the 
underlying DB schema, and making such changes to a model will create the need 
for a migration.

This is absolutely necessary, though, because the migration framework generates 
a frozen snapshot of the state at each migration, so that you can access the 
ORM to do things like data updates via the RunPython migration operator. If you 
were to change, say, the default ordering of a model without also generating a 
migration to record that change (even though it has no effect on the database 
schema), you could run into trouble if you later removed a field referenced by 
the old ordering, since one of the intermediate ORM snapshots would attempt to 
order queries by a now-nonexistent column.

--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
django-users+unsubscr...@googlegroups.com.
To post to this group, send email to 
django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAL13Cg_0p1pMtMR2y2eGry1UhQaSRpswntcnMWtHNdPgV1Ph9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/A0879DE1-7EED-434F-8D71-3FFE1FB2755D%40hotmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Best way to structure a django project with multiple levels of sub apps?

[James Schneider]

Once again, I accidently sent this email. Not a fan of the sensitivity of
the screen on my new phone. Continued below.

I think if you better define an app vs. a model, you'll find that you need
only a few apps.

IMHO, an app should represent a logical segment of your project, broad
enough that a majority of the business logic is self-contained and relies
little on other apps. Apps should be very high level.


Forgot to mention one exception. I always have a 'common' app that contains
all of the models (or views, etc.) used by multiple other applications. In
a majority of cases, these are abstract models I use for all of my concrete
models in other apps that include fields for auditing such as created_by
and created_on. Other examples would include abstract models for addresses,
and master views that include logic such as requiring a user be logged in
used as mixins for other apps.


In your case, you would likely have apps for 'accounting' and
'engineering', along with other distinct units within the business. The
business processes and logic for engineering have little or no overlap with
that of accounting. Same with HR, IT, and so on. That's literally as
detailed as your apps should be, in which case you'll probably end up with
a dozen or so at worst. If it were me, I'd have even less than that with
even larger groupings, maybe something along the lines of 'sales' and
'operations' as the only two apps.

One of the key ideas to help understand here is that a Python file
containing models does not have to be called models.py. In fact, I usually
don't have a models.py at all. Instead, I create a Python module within the
app called 'models', which is nothing more than a directory named 'models'
that contains __init__.py and all of the Python files that contains my
models. Each of those files contains models that are further separated
logically. The structure looks like this (some structure omitted for
brevity):

business_proj/
engineering/
models/
__init__.py
abstract_design.py
abstract_manufacturing.py
abstract_quality_assurance.py
design.py
manufacturing.py
quality_assurance.py

You could also bury the abstract models in their own module using the same
pattern.

Assuming you have 'engineering' as a registered app, your __init__.py would
look


something like this:

from .design import *
from .manufacturing import *
from .quality_assurance import *


This let's Django find all of the models to create the migrations
accordingly.

Imports of these models in other locations can also much more objective:

from engineering.models.design import WidgetOutline

In reality, this also works as an alternative because of the __init__.py
setup:

from engineering.models import WidgetOutline

But that is slightly less verbose. Your call.

Views and forms (and probably URL's) can be organized using the same
pattern. There's no automatic discovery process like models have; views,
forms, and URL's can live anywhere with any legal Python name since they
are manually imported wherever they are needed. Theoretically your entire
project can run inside a single monolithic file.

HTH,

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciWDqrGyYuQSb-hfCc73eG0TVut4-QW3HfA3J_6xLyJCNA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Row based permissions: In DB or App? ... farewell pain

[guettli]

Am Donnerstag, 17. August 2017 16:09:42 UTC+2 schrieb Vijay Khemlani:
>
> How can we propose faster alternatives if we don't know how fast must it 
> be?
>
> Of course it is up to you in the end, but if you are seriously considering 
> exotic options such as using stored procedures at least I would like to 
> know what specific bottlenecks did you stumble upon as I also use 
> object-level permissions extensively in my own project and haven't had any 
> performance issues yet.
>
>
First I had stored procedures on my mind, but I talked to a postgreSQL 
expert and he said that the django ORM would be fast enough in most cases. 
This means
 "has_perm" checking would be finally SQL, something like 
MyModel.objects.filter(...).exists()

The bottle neck I had with guardian is explained on the page below. It is 
not new, it is well known. My case is not different. Up to now we use 
generic foreign keys. That's why I think
numbers won't help here.

http://django-guardian.readthedocs.io/en/stable/userguide/performance.html

 

> If nothing else, your results would prove useful to the community.
>
>

Yes, if I find a solution, I will publish it. This is important for me, 
since the solutions I provide to our customers are mostly
the work of open source libraries. I just put these great peaces together 
with some lines of python code.

I ask myself how to be most useful for the community?


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/eae520bb-6c06-44dc-a0e9-9e616575c6fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Best way to structure a django project with multiple levels of sub apps?

[James Schneider]

On Aug 17, 2017 2:26 PM, "Alexander Joseph" 
wrote:

I'm bullding a larger django project and I'm starting to implement
cookiecutter django after reading through "2 Scoops of Django" but still
have some questions on structuring a project.


I've setup my project, we'll call it 'business_proj'. In business_proj I
started an app called 'accounting' this might have an accounting dashboard
for users in the accounting security group. Now, what if I want to have
apps that belong to accounting, such as 'invoices' and 'purchase_orders'?
Should I create those apps inside my accounting app? Or should I create all
my apps in the main project root? The way I've started doing it is creating
child apps inside of their parent apps but some parent apps are so big that
even this gets messy.


What you are calling 'apps', I would call models. PO's and invoices are
part of larger workflows. There's no technical reason, or even from an
organization perspective, to create that many apps.



Is there a better way to do this? Or are there any pitfalls I need to watch
out for doing it this way? Would it be better to simply make an
'engineering' folder (not module) in my project and store all the
engineering related projects in that folder?Thanks

I think if you better define an app vs. a model, you'll find that you need
only a few apps.

IMHO, an app should represent a logical segment of your project, broad
enough that a majority of the business logic is self-contained and relies
little on other apps. Apps should be very high level.

In your case, you would likely have apps for 'accounting' and
'engineering', along with other distinct units within the business. The
business processes and logic for engineering have little or no overlap with
that of accounting. Same with HR, IT, and so on. That's literally as
detailed as your apps should be, in which case you'll probably end up with
a dozen or so at worst. If it were me, I'd have even less than that with
even larger groupings, maybe something along the lines of 'sales' and
'operations' as the only two apps.

One of the key ideas to help understand here is that a Python file
containing models does not have to be called models.py. In fact, I usually
don't have a models.py at all. Instead, I create a Python module within the
app called 'models', which is nothing more than a directory named 'models'
that contains __init__.py and all of the Python files that contains my
models. Each of those files contains models that are further separated
logically. The structure looks like this (some structure omitted for
brevity):

business_proj/
engineering/
models/
__init__.py
abstract_design.py
abstract_manufacturing.py
abstract_quality_assurance.py
design.py
manufacturing.py
quality_assurance.py

You could also bury the abstract models in their own module using the same
pattern.

Assuming you have 'engineering' as a registered app, your __init__.py would
look

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciWCagCYS6Rpp91uBiS2SbQzthJQ4kLObhrpHk30owN%2BfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A lot of Problems with Migrating (conceptual)

[James Bennett]

On Thu, Aug 17, 2017 at 1:03 PM, Antonis Christofides <
anto...@djangodeployment.com> wrote:

> Second, just to make things clear, the word "migration" has two meanings.
> The original meaning of migration is to switch to another software system
> (e.g. migrate from MySQL to PostgreSQL, or migrate a repository from
> subversion to git). In Django, the term "migration" means something else:
> to update the database to the new schema. In my opinion this terminology is
> wrong and confusing (apparently it comes from Ruby on Rails, but I'm not
> certain), and a better one would have been "dbupdate" or something, but
> since it's migration we'll keep it and you'll have to understand which
> meaning is meant in each case.
>

An important thing worth knowing is that Django's migration framework does
*not* only track the database schema. It also tracks the state of the
Python classes which define the models, including options which do not
affect the underlying DB schema, and making such changes to a model will
create the need for a migration.

This is absolutely necessary, though, because the migration framework
generates a frozen snapshot of the state at each migration, so that you can
access the ORM to do things like data updates via the RunPython migration
operator. If you were to change, say, the default ordering of a model
without also generating a migration to record that change (even though it
has no effect on the database schema), you could run into trouble if you
later removed a field referenced by the old ordering, since one of the
intermediate ORM snapshots would attempt to order queries by a
now-nonexistent column.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAL13Cg_0p1pMtMR2y2eGry1UhQaSRpswntcnMWtHNdPgV1Ph9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A lot of Problems with Migrating (conceptual)

[James Schneider]

On Aug 17, 2017 10:10 AM, "Alexander Joseph" 
wrote:

I'm pretty new to django and I've been having problems with
makemigrations/migrate and I wonder if Im doing things right.

I'm using MySQL backend and after reading in the documentation a little it
sounds like Postgresql might make migrating more painless. Usually my
problems stem from changing table fields, adding new tables/models, and
migrating both on my development server and my production server
(PythonAnywhere)

More than once I've had to drop my database, delete all my migrations, and
start over with initial migrations on the development server. This wont fly
for long on production though of course .. once I actually have users and
data.

I'm wondering if I should switch to postgresql or if theres any further
resources that you might know of that would help me out? Thanks


Switching to a different DBMS will probably not alleviate migration issues,
especially if your project is still in its infancy and your migrations are
all still auto-generated. The migration system was designed to abstract
away the differences between the supported DBMS' so that you as the
developer didn't need to worry about it. Ideally you would be able to
deploy the same set of migrations on any of the supported databases.

Many Django devs have a bias towards Postgres, but much effort is put
toward ensuring that MySQL is supported since that is the only option for
many developers on their hosting platform. Some operations require extra
work in the migrations, and are challenging to implement regardless of the
DB platform. This is likely what you are running into.

I highly doubt MySQL itself is the cause, but it's possible. I'd recommend
posting up some of the errors that you are getting so that the folks here
can help. Might be more beneficial than setting up Postgres and having the
same issues.

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciXA-iNG5cf-q7P9T-cVCjsw44zTgbmBJFM_qBe2c0zdQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.