Re: Rethinking migrations

[charettes]

I have to agree with Marteen.

>From my experience what really slow down the migrate and makemigrations
command is the rendering of model states into concrete model classes. This
is something I concluded from my work on adding the plan object to 
pre_migrate
and post_migrate signals.

As soon as an operation accesses state.apps the rendering kicks in which
triggers the dynamic creation of multiple model classes and the computation
of reverse relationships. There are mechanisms in place to prevent the whole
project model classes from being rendered again when a model state is
altered but if the operation is performed on a model referenced by many
others the relationship chain might force a large number of them to be
rendered again causing massive slow downs.

Markus Holtermann has been working on teaching the migration framework
how to perform database operations without relying on state.apps which 
should
solve the remaining performance issues of the migrate command. In the case
of makemigrations the last remaining issue in the master branch should be 
solved
by stopping to rely on state.apps in RenameModel.state_forwards[1].

Patryk, many improvement landed in 1.9 and 1.10 to speed up the commands
dealing with migrations. Are you still seeing the same slowdown on these 
versions?

Simon

[1] https://github.com/django/django/pull/7468

Le dimanche 6 novembre 2016 00:32:04 UTC+1, Marten Kenbeek a écrit :
>
> On Saturday, November 5, 2016 at 4:53:49 PM UTC+1, Patryk Zawadzki wrote:
>>
>> 1. Dependency resolution that turns the migration dependency graph into 
>> an ordered list happens every time you try to create or execute a 
>> migration. If you have several hundred migrations it becomes quite slow. 
>> I'm talking multiple minutes kind of slow. As you can imagine working with 
>> multiple branches or perfecting your migrations quickly becomes a tedious 
>> task.
>>
>
> Did the dependency resolution actually come up in benchmarks/profiles as a 
> bottleneck? When I optimized and benchmarked the dependency graph code, it 
> had no trouble ordering ~1000 randomly generated migrations with lots of 
> inter-app dependencies in less than a second. I'd be surprised if this had 
> any significant impact on the overall performance of migrations.
>
> An easy way to test this is the `showmigrations` command, which will only 
> generate the graph without any model state changes or model rendering 
> taking place. It does some other things, but nothing that should take in 
> the order of minutes.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4a012e54-fae5-4bba-97a9-f323f38e53bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Marten Kenbeek]

On Saturday, November 5, 2016 at 4:53:49 PM UTC+1, Patryk Zawadzki wrote:
>
> 1. Dependency resolution that turns the migration dependency graph into an 
> ordered list happens every time you try to create or execute a migration. 
> If you have several hundred migrations it becomes quite slow. I'm talking 
> multiple minutes kind of slow. As you can imagine working with multiple 
> branches or perfecting your migrations quickly becomes a tedious task.
>

Did the dependency resolution actually come up in benchmarks/profiles as a 
bottleneck? When I optimized and benchmarked the dependency graph code, it 
had no trouble ordering ~1000 randomly generated migrations with lots of 
inter-app dependencies in less than a second. I'd be surprised if this had 
any significant impact on the overall performance of migrations.

An easy way to test this is the `showmigrations` command, which will only 
generate the graph without any model state changes or model rendering 
taking place. It does some other things, but nothing that should take in 
the order of minutes.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/90cd80d5-fc72-4611-b716-b6d1e5d80e43%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Patryk Zawadzki]

W dniu sobota, 5 listopada 2016 19:57:38 UTC+1 użytkownik Aymeric Augustin 
napisał:
>
> My solution is to throw away and remake all migrations on a regular basis. 
> Then I `TRUNCATE TABLE django_migrations` and `django-admin migrate 
> --fake`. Obviously this isn’t a great solution. 
>
> Since I work mostly on small projects with just a couple developers on 
> staff, doing this every few months suffices to keep the run time below one 
> minute (which is still quite annoying). 
>
> There’s a risk to lose important, manually generated migrations, typically 
> those that create indexes. I diff the schema before / after with apgdiff to 
> avoid such problems. 
>

That's the main problem we're facing. I'm currently leading a project that 
predates dinosaurs and when it was switched from South to Django, all the 
data migrations were just carried over from the old code. They are holy 
gifts from the elder gods and are rich with eldritch symbols. Nobody wants 
to have to copy and paste them every month or so when we decide to redo all 
of the migrations.

There's also the problem of having many long-running (weeks to months) 
feature branches that make it hard to find a point in time where all 
migrations can be safely discarded.

I can also imagine it's much harder to redo initial migrations in projects 
where two-way relations exist between certain applications.

Cheers,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/88ed6c68-1732-49bf-8d6b-3c190729472a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Patryk Zawadzki]

W dniu sobota, 5 listopada 2016 18:40:24 UTC+1 użytkownik Shai Berger 
napisał:
>
> > 2. Dependency resolution is only stable as long as the migration set is 
> > frozen. Sometimes introducing a new migration is enough to break 
> existing 
> > migrations by causing them to execute in a slightly different order. We 
> > often have to backtrack and edit existing migrations and enforce a 
> strict 
> > resolution order by introducing arbitrary dependencies. 
> > 
>
> So, you say you really have implicit dependencies between migrations -- 
> dependencies in substance, which aren't recorded as dependencies. This 
> seems 
> to indicate that you have a lot of manually-written migrations (data 
> migrations?), since the automatically-written ones do include relevant 
> dependencies. This seems odd -- it sounds like you're doing something out 
> of 
> the ordinary. 
>
> This would also explain some of your bad experience with squashing -- 
> indeed, 
> if you have many data migrations, squashing can become much less 
> effective. 
>

Let's not come to conclusions prematurely. Django only supports predicate 
dependencies. You can say "not earlier than after these are applied" but 
that does not mean "immediately after they are applied". Sometimes Django 
tries to run the migration much later. If you have your models scattered 
across a large number of applications (we use apps to gateway entire 
classes of related features) sometimes the late migrations tries to 
reference a column in another model that was long since removed by a much 
later added migration in its respective app.

> 3. Removing an app from a project is a nightmare. You can't migrate to 
> zero 
> > state unless the app is still there. There is no way to add "revert all 
> > migrations for app X" to the migration graph, it's something you need to 
> > run manually. There is no clean way to remove an app that was ever 
> > references in a relation. We were forced to do all kinds of hacks to get 
> > around this. Sometimes it's necessary to create an empty eggshell app 
> with 
> > the same name and copy all migrations there then add necessary data 
> > migrations and finally migrations that remove all the models, indices, 
> > procedures etc. Sometimes people just leave a dead application in 
> > INSTALLED_APPS to not have to deal with this. 
>
> Clear out (maybe even remove) models.py and type "makemigrations", and you 
> get 
> a migration that deletes everything. The answer to getting rid of the 
> historical migrations is squashing, but of course you first need squashing 
> to 
> work properly. 
>

I cannot clear out anything from an app that came from PyPI. That's why I 
mentioned creating fake empty apps that are just containers for their 
migration history. Squashing does nothing to help with that if you have 
another application reference any of those models. Squashing only helps you 
have fewer migrations. If the migrations were always in the correct order, 
the migration engine could collapse them automatically at execution time.

> 4. Squashing migrations is wonky at best. If you create a model in one 
> > migration, alter one of its fields in another and then finally drop the 
> > model sometime later, the squashed migration will have Django try to 
> > execute the alter first and complain about the table not being there. 
> Also 
> > the only reason we need to squash migrations is to prevent problem 1 
> above 
> > from becoming exponentially worse. If migrations were only as slow as 
> the 
> > underlying SQL commands, we'd likely never squash them. 
> > 
>
> If that's so, it's a bug you should report; it's also an issue you can 
> work- 
> around by editing the migration to remove the redundant operation. There 
> are   
> issues with squashing, to be sure, but I don't think this is one of the 
> serious ones. 
>

It's a bug that I will report at some point but I mostly encounter it in 
environments where I can't afford the time needed to properly debug.

> 6. Conflict detection and resolution (migrate --merge) is a make-believe 
> > solution. It just trains people to execute the command without 
> > investigating whether their migration history still makes sense. 
>
> It could be smarter, assuming it understood the content of migrations. We 
> could probably improve it to a point where, for most cases, it would 
> either 
> know to merge automatically or know that there really is a conflict. This 
> would 
> probably not help you if you have a lot of RunPython's in your migrations. 
>

Depends on the project. I really don't care about the framework trying to 
reason about the results of a git merge. Django does not have enough 
understanding of the code and the version control history to do the job of 
the person responsible for the merge. Getting stuff right 60% of the time 
is not reliable enough to depend on it but is reliable enough to get some 
people lazy.

> Some of these I need to dig deeper into and probably file proper tic

Re: Getting full URL using django.urls.base.reverse - PR

[Florian Apolloner]

On Saturday, November 5, 2016 at 6:56:29 PM UTC+1, Sjoerd Job Postmus wrote:
>
> If you go with storing the base domain in the threadlocals, why not go 
> full in and store the request itself in the locals? [1]
>

Because that opens a whole new can of worms, if possible we want less 
threadlocals, not more…
 

> As far as using it in the templates... We have a RequestContext right? So 
> in most cases that should not be an issue I presume.
>

Still leaves the question of how to nicely pass this to the url tag.

But yes, Celery would be a problem, unless we push the request object to 
> celery as well, but I presume that makes mostly no sense except for this.
>

Yeah, I didn't think that through :) 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/84fc3b7a-4263-46bb-a292-e27215952e63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fellow Report - November 5, 2016

[Tim Graham]

Light week due to Django Under the Hood travel and attendance. Busy day of 
sprints today. Jenkins hasn’t yet recovered!

Triaged

---

https://code.djangoproject.com/ticket/27410 - Clarify staticfiles "is 
enabled (default)" in ref (fixed)

https://code.djangoproject.com/ticket/27414 - Document that aliasing 
ForeignKey fields in F expressions returns pk values, not model instances 
(accepted)

https://code.djangoproject.com/ticket/27409 - Issues with admindocs 
docstring links generation (accepted)

https://code.djangoproject.com/ticket/27408 - Make QuerySet.bulk_create() 
populate fields on related models (someday/maybe)

https://code.djangoproject.com/ticket/27407 
-Model.delete(keep_parents=True) should preserve parent reverse 
relationships

https://code.djangoproject.com/ticket/27402 - When using i18n_patterns and 
prefix_default_language=False, 404 page redirects incorrectly (accepted)

https://code.djangoproject.com/ticket/27420 - Oracle DB test user password 
must be quoted if it starts with a number (accepted)

https://code.djangoproject.com/ticket/27427 - Add reply_to argument on 
send_mail wrapper. (wontfix)


Authored

--

https://github.com/django/django/pull/7459 - Completed 
django.utils.timezone test coverage.

https://github.com/django/django/commit/7fe2d8d940f1a02c4754008a27060c4a03e9
 
-  Fixed CVE-2016-9014 -- Validated Host header when DEBUG=True

Reviewed/committed

--

https://github.com/django/django/pull/7162 - Fixed #27063 -- Prevented 
i18n_patterns() from using too much of the URL as the language.

https://github.com/django/django/pull/7452 - Fixed #27400 -- Documented {% 
static %} encoding change in 1.10.

https://github.com/django/django/pull/7431 - Fixed #27363 -- Replaced 
unsafe redirect in SessionMiddleware with SuspiciousOperation.

https://github.com/django/django/pull/7476 - Fixed #27346 -- Stopped 
setting the Content-Length header in ConditionalGetMiddleware.

https://github.com/django/django/commit/da7910d4834726eca596af0a830762fa5fb2dfd9
 
- Fixed CVE-2016-9013 -- Generated a random database user password when 
running tests on Oracle.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ced5b95d-397e-437b-8f88-1fba6fe20eca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Aymeric Augustin]

Hello,

My solution is to throw away and remake all migrations on a regular basis. Then 
I `TRUNCATE TABLE django_migrations` and `django-admin migrate --fake`. 
Obviously this isn’t a great solution.

Since I work mostly on small projects with just a couple developers on staff, 
doing this every few months suffices to keep the run time below one minute 
(which is still quite annoying).

There’s a risk to lose important, manually generated migrations, typically 
those that create indexes. I diff the schema before / after with apgdiff to 
avoid such problems.

This is quite doable but outside the comfort zone of many developers: my 
clients prefer to have me do it even though I documented the steps in detail.

So… yeah, it would be nice to have something more practical, even if it 
requires trading off some purity in the design of migrations.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/84521630-23DC-44CF-B363-8FC913B10084%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Patryk Zawadzki]

W dniu sobota, 5 listopada 2016 17:30:15 UTC+1 użytkownik Andrew Godwin 
napisał:
>
> Hello! I have opinions about this :)
>  
>
>> Possible solution (or "how I'd build it today if there was no existing 
>> code in Django core"):
>>
>> a. Make migrations part of the project and not individual apps. This 
>> takes care of problem 3 above.
>>
>
> It also means it's impossible for apps to ship migrations and define how 
> to upgrade from version to version. I realise that (c) below is part of a 
> proposed solution to this, but how do you propose to match up what's 
> already been run in the database without having names match (and then you 
> just have app migrations by another name)?
>

I would actually insist on keeping the names intact. It means that adding 
an external dependency could inject a migration with a date from the 
previous year but I think that's not a problem as it's guaranteed not to 
conflict with any other migrations.


>> b. Prefix individual migration files with a UTC timestamp 
>> (20161105151023_add_foo) to provide a strict sorting order. This removes 
>> the depsolving requirement and takes care of 1 and 2. By eliminating those 
>> it makes 4 kind of obsolete as squashing migrations would become pointless.
>>
>
> Unfortunately this does not help all the time as computers' clocks aren't 
> necessarily right or in sync, so it would merely be an approximation and 
> you'd still get the occasional clash.
>

You only need your own migrations to be ordered so you can safely assume 
the previous one to be applied before the one you're writing at the moment. 
For two unrelated changes the order pretty much does not matter.
 

> c. Have reusable apps provide migration templates that Django then copies 
>> to my project when "makemigrations" is run.
>>
>
> Would these be lined up with their own timestamp in the single serial 
> migration timeline? Would you have to make sure any of these templates from 
> any app update was copied across and put in the order before you used the 
> new columns?
>

I'd say use proper timestamps. This way two apps can depend on each other 
and the migrations will still get run in proper order.
 

> d. Maintain a separate directory for each database connection.
>>
>
> This I think might be a good idea, though I'd like to see a more 
> generalised idea of "migration sets" and you then then say which alias uses 
> which set (so you can share sets among more than one connection)
>

Agreed.
 

> e. Execute all migrations in alphabetical order (which means by timestamp 
>> first). When an unapplied migration is followed by an applied one, ask 
>> whether to attempt to just apply it or if the user wants to first unapply 
>> migrations that came after it. To me this would work better than 6.
>>
>
> This is basically what South used to do, and it worked reasonably well in 
> either being successful or exploding enough that people noticed. Given that 
> you're proposing per-project migrations, however, people are going to run 
> into this almost constantly, as they will clash significantly more than 
> per-app ones.
>

South was not perfect but I'd say the current solution is not better, it's 
just different. Some of my projects use a lot of long-running feature 
branches so I have an application where every other migration is a merge 
migration with accepted default values. We do try to make migrations 
backwards-compatible where needed but I don't think it's a common scenario 
to add conflicting changes on two feature branches. Most of our conflicts 
can be described as department A added a field they needed while department 
B added a data migration to fix a denormalized field.
 

> Of course we do have migration support in core and it's not compatible 
>> with most of the above list. Any ideas? I think serializing the dependency 
>> solver state and reusing it between runs could be a pretty low hanging 
>> fruit (like "npm shrinkwrap" or yarn's lock file).
>>
>
> I think not only could the dependency solver state be serialised but that 
> it would be a replacement for the datetimes-on-filename proposal in that 
> you could easily pull out a previously-serialised order from disk and then 
> work out what the new ones do.
>
> I am generally not keen on the idea of per-project migrations, though - it 
> makes what's in the database a property of the project, not the app, and 
> that's not how Django has worked traditionally. I think an effort to get a 
> more reliable, exposed global ordering of those individual app migrations 
> would go a long way towards the end goal without having to have migration 
> templates, upgrade instructions, and way more collisions between branches.
>

I do believe that the database is my property and I'd much rather see the 
project code hold reign over its structure. Some problems simply cannot be 
solved by submitting an upstream patch (project-specific or 
backend-specific indexes come to mind).
 

> At the end of the day, though, there's

Re: Getting full URL using django.urls.base.reverse - PR

[Sjoerd Job Postmus]

If you go with storing the base domain in the threadlocals, why not go full in and store the request itself in the locals? [1]
As far as using it in the templates... We have a RequestContext right? So in most cases that should not be an issue I presume.
But yes, Celery would be a problem, unless we push the request object to celery as well, but I presume that makes mostly no sense except for this.
[1] I must confess I greatly prefer being explicit with passing the request around, but sometimes you suddenly need the request object 10 layers deep but nowhere in the middle. Or, you want to use the request object in your AuthenticationBackend (for security counter-measures). For instance, django-brutebuster stores the request in a threadlocal to get the IP address.
On Nov 5, 2016 5:59 PM, Florian Apolloner  wrote:What I was trying to suggest is that the base domain gets stored in one of our threadlocals so that we can generate the full URL without having access to the domain (though I just realize that this wouldn't work in the case of celery etc :/)On Saturday, November 5, 2016 at 5:53:49 PM UTC+1, Mislav Cimperšak wrote:For me use cases were callback urls sent to some 3rd party and of course - emails.Yeah, I wasn't thinking about url tag.I'm not 100% sure I understand what you are suggesting.If we choose to go down the path of using kwarg I could change this in the `reverse` method itself. That way calling `reverse('foo', kwargs={'full': True})` would return a full URL. But that opens a whole other can of issues. For example if someone already has in their code `full` as a parameter for url.On Saturday, November 5, 2016 at 4:40:46 PM UTC+1, Florian Apolloner wrote:To be honest, I think I do not really see the usecase here. I know that sometimes it is required to generate full urls (especially in emails), but quite often in those you do not have access to the request at all. Also, how would this play together with the url tag? Personally I would set the current domain on the local "urlconf" (we have one per thread) and add a "full" kwarg to reverse (and something similar to the urltag).Cheers,FlorianOn Saturday, November 5, 2016 at 3:41:09 PM UTC+1, Mislav Cimperšak wrote:During the sprint in Amsterdam I created a pull request https://github.com/django/django/pull/7484In it I'm adding an optional argument `request` to `django.urls.base.urls.reverse`method.The idea is that using `request` object one can get a fully qualified URL.The idea is taken from Django Rest Framework (https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/reverse.py#L55-L59).I've talked to Tim Graham in person and he said that there are some possibilities that django-hosts will be merged into main django code base with it's own implementation of `reverse` method.To move this issue forward I need an input from core team members regarding my implementation and future plans. If necessary I'll trash my current implementation if it will just bring on more potentials problems in the future.My arguments for suggested implementation is that it's much more simple and easier for the enduser than django-hosts implementation and it's already Tom Christie approved(TM).



-- 
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/613aacf1-95de-43ac-8cc6-a54c09e22aca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.




-- 
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/18df8e3a-79f8-4976-86c3-4779d24a4fc9%40email.android.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Shai Berger]

Hi,

On Saturday 05 November 2016 17:53:49 Patryk Zawadzki wrote:
> 
> I'm typing this from the comfort of Django: Under the Hood sprints so
> please excuse poor grammar and the somewhat chaotic explanations that
> follow. I'm very tired and English is not my mother tongue. This is not a
> DEP but merely a stream of consciousness I'd love to get some feedback on.
> 

I am dealing with some similar issues, but I've reached very different 
conclusions. In much the same spirit, this is not very orderly.


> Here are some of the problems we face when dealing with migrations:
> 
> 1. Dependency resolution that turns the migration dependency graph into an
> ordered list happens every time you try to create or execute a migration.
> If you have several hundred migrations it becomes quite slow. I'm talking
> multiple minutes kind of slow. As you can imagine working with multiple
> branches or perfecting your migrations quickly becomes a tedious task.
> 

I've known this to happen, indeed.

> 2. Dependency resolution is only stable as long as the migration set is
> frozen. Sometimes introducing a new migration is enough to break existing
> migrations by causing them to execute in a slightly different order. We
> often have to backtrack and edit existing migrations and enforce a strict
> resolution order by introducing arbitrary dependencies.
> 

So, you say you really have implicit dependencies between migrations -- 
dependencies in substance, which aren't recorded as dependencies. This seems 
to indicate that you have a lot of manually-written migrations (data 
migrations?), since the automatically-written ones do include relevant 
dependencies. This seems odd -- it sounds like you're doing something out of 
the ordinary.

This would also explain some of your bad experience with squashing -- indeed, 
if you have many data migrations, squashing can become much less effective.

> 3. Removing an app from a project is a nightmare. You can't migrate to zero
> state unless the app is still there. There is no way to add "revert all
> migrations for app X" to the migration graph, it's something you need to
> run manually. There is no clean way to remove an app that was ever
> references in a relation. We were forced to do all kinds of hacks to get
> around this. Sometimes it's necessary to create an empty eggshell app with
> the same name and copy all migrations there then add necessary data
> migrations and finally migrations that remove all the models, indices,
> procedures etc. Sometimes people just leave a dead application in
> INSTALLED_APPS to not have to deal with this.

Clear out (maybe even remove) models.py and type "makemigrations", and you get 
a migration that deletes everything. The answer to getting rid of the 
historical migrations is squashing, but of course you first need squashing to 
work properly.

> 
> 4. Squashing migrations is wonky at best. If you create a model in one
> migration, alter one of its fields in another and then finally drop the
> model sometime later, the squashed migration will have Django try to
> execute the alter first and complain about the table not being there. Also
> the only reason we need to squash migrations is to prevent problem 1 above
> from becoming exponentially worse. If migrations were only as slow as the
> underlying SQL commands, we'd likely never squash them.
> 

If that's so, it's a bug you should report; it's also an issue you can work-
around by editing the migration to remove the redundant operation. There are  
issues with squashing, to be sure, but I don't think this is one of the 
serious ones.

> 5. There's no simple way to roll back all the migrations introduced after a
> particular point in time which is very useful when working with multiple
> feature branches. In my current project dropping the database means having
> to reimport over 200 MB of data snapshots. Switching branches requires me
> to look at branch diffs to determine which migrations to revert.
> 

Yes, this is a real issue, with one modification -- I'd much rather have a good 
way to migrate to a point-in-version-history than to a point-in-time.

This is even more than a development issue -- I've encountered a use-case for 
doing something like this in production: If I want to be able to export an 
object represented by a model (or set of models), by serializing it and saving 
the serialized version; and then I'd want to import it back in after the app 
has progressed -- if I'd want generic support for that, I'd need a way to 
migrate a database to the point where the object was exported, import it, and 
then roll the database forward to the "present".

> 6. Conflict detection and resolution (migrate --merge) is a make-believe
> solution. It just trains people to execute the command without
> investigating whether their migration history still makes sense.
> 

It could be smarter, assuming it understood the content of migrations. We 
could probably improve it to a point where, for most ca

Re: Getting full URL using django.urls.base.reverse - PR

[Florian Apolloner]

What I was trying to suggest is that the base domain gets stored in one of 
our threadlocals so that we can generate the full URL without having access 
to the domain (though I just realize that this wouldn't work in the case of 
celery etc :/)

On Saturday, November 5, 2016 at 5:53:49 PM UTC+1, Mislav Cimperšak wrote:
>
> For me use cases were callback urls sent to some 3rd party and of course - 
> emails.
>
> Yeah, I wasn't thinking about url tag.
>
> I'm not 100% sure I understand what you are suggesting.
>
> If we choose to go down the path of using kwarg I could change this in the 
> `reverse` method itself. That way calling `reverse('foo', kwargs={'full': 
> True})` would return a full URL. But that opens a whole other can of 
> issues. For example if someone already has in their code `full` as a 
> parameter for url.
>
>
>
> On Saturday, November 5, 2016 at 4:40:46 PM UTC+1, Florian Apolloner wrote:
>>
>> To be honest, I think I do not really see the usecase here. I know that 
>> sometimes it is required to generate full urls (especially in emails), but 
>> quite often in those you do not have access to the request at all. Also, 
>> how would this play together with the url tag? Personally I would set the 
>> current domain on the local "urlconf" (we have one per thread) and add a 
>> "full" kwarg to reverse (and something similar to the urltag).
>>
>> Cheers,
>> Florian
>>
>> On Saturday, November 5, 2016 at 3:41:09 PM UTC+1, Mislav Cimperšak wrote:
>>>
>>> During the sprint in Amsterdam I created a pull request 
>>> https://github.com/django/django/pull/7484
>>>
>>> In it I'm adding an optional argument `request` to 
>>> `django.urls.base.urls.reverse`method.
>>> The idea is that using `request` object one can get a fully qualified 
>>> URL.
>>>
>>> The idea is taken from Django Rest Framework (
>>> https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/reverse.py#L55-L59
>>> ).
>>>
>>> I've talked to Tim Graham in person and he said that there are some 
>>> possibilities that django-hosts 
>>>  will be merged into main 
>>> django code base with it's own implementation of `reverse` method.
>>>
>>>
>>> To move this issue forward I need an input from core team members 
>>> regarding my implementation and future plans. If necessary I'll trash my 
>>> current implementation if it will just bring on more potentials problems in 
>>> the future.
>>>
>>>
>>> My arguments for suggested implementation is that it's much more simple 
>>> and easier for the enduser than django-hosts implementation and it's 
>>> already *Tom Christie approved(TM).*
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/613aacf1-95de-43ac-8cc6-a54c09e22aca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Getting full URL using django.urls.base.reverse - PR

[Mislav Cimperšak]

For me use cases were callback urls sent to some 3rd party and of course - 
emails.

Yeah, I wasn't thinking about url tag.

I'm not 100% sure I understand what you are suggesting.

If we choose to go down the path of using kwarg I could change this in the 
`reverse` method itself. That way calling `reverse('foo', kwargs={'full': 
True})` would return a full URL. But that opens a whole other can of 
issues. For example if someone already has in their code `full` as a 
parameter for url.



On Saturday, November 5, 2016 at 4:40:46 PM UTC+1, Florian Apolloner wrote:
>
> To be honest, I think I do not really see the usecase here. I know that 
> sometimes it is required to generate full urls (especially in emails), but 
> quite often in those you do not have access to the request at all. Also, 
> how would this play together with the url tag? Personally I would set the 
> current domain on the local "urlconf" (we have one per thread) and add a 
> "full" kwarg to reverse (and something similar to the urltag).
>
> Cheers,
> Florian
>
> On Saturday, November 5, 2016 at 3:41:09 PM UTC+1, Mislav Cimperšak wrote:
>>
>> During the sprint in Amsterdam I created a pull request 
>> https://github.com/django/django/pull/7484
>>
>> In it I'm adding an optional argument `request` to 
>> `django.urls.base.urls.reverse`method.
>> The idea is that using `request` object one can get a fully qualified URL.
>>
>> The idea is taken from Django Rest Framework (
>> https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/reverse.py#L55-L59
>> ).
>>
>> I've talked to Tim Graham in person and he said that there are some 
>> possibilities that django-hosts 
>>  will be merged into main 
>> django code base with it's own implementation of `reverse` method.
>>
>>
>> To move this issue forward I need an input from core team members 
>> regarding my implementation and future plans. If necessary I'll trash my 
>> current implementation if it will just bring on more potentials problems in 
>> the future.
>>
>>
>> My arguments for suggested implementation is that it's much more simple 
>> and easier for the enduser than django-hosts implementation and it's 
>> already *Tom Christie approved(TM).*
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/881afdc9-94fd-4c1a-95a2-0db1e4a03a81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Andrew Godwin]

Hello! I have opinions about this :)


> Possible solution (or "how I'd build it today if there was no existing
> code in Django core"):
>
> a. Make migrations part of the project and not individual apps. This takes
> care of problem 3 above.
>

It also means it's impossible for apps to ship migrations and define how to
upgrade from version to version. I realise that (c) below is part of a
proposed solution to this, but how do you propose to match up what's
already been run in the database without having names match (and then you
just have app migrations by another name)?


>
> b. Prefix individual migration files with a UTC timestamp
> (20161105151023_add_foo) to provide a strict sorting order. This removes
> the depsolving requirement and takes care of 1 and 2. By eliminating those
> it makes 4 kind of obsolete as squashing migrations would become pointless.
>

Unfortunately this does not help all the time as computers' clocks aren't
necessarily right or in sync, so it would merely be an approximation and
you'd still get the occasional clash.


>
> c. Have reusable apps provide migration templates that Django then copies
> to my project when "makemigrations" is run.
>

Would these be lined up with their own timestamp in the single serial
migration timeline? Would you have to make sure any of these templates from
any app update was copied across and put in the order before you used the
new columns?


>
> d. Maintain a separate directory for each database connection.
>

This I think might be a good idea, though I'd like to see a more
generalised idea of "migration sets" and you then then say which alias uses
which set (so you can share sets among more than one connection)


>
> e. Execute all migrations in alphabetical order (which means by timestamp
> first). When an unapplied migration is followed by an applied one, ask
> whether to attempt to just apply it or if the user wants to first unapply
> migrations that came after it. To me this would work better than 6.
>

This is basically what South used to do, and it worked reasonably well in
either being successful or exploding enough that people noticed. Given that
you're proposing per-project migrations, however, people are going to run
into this almost constantly, as they will clash significantly more than
per-app ones.


>
> Of course we do have migration support in core and it's not compatible
> with most of the above list. Any ideas? I think serializing the dependency
> solver state and reusing it between runs could be a pretty low hanging
> fruit (like "npm shrinkwrap" or yarn's lock file).
>

I think not only could the dependency solver state be serialised but that
it would be a replacement for the datetimes-on-filename proposal in that
you could easily pull out a previously-serialised order from disk and then
work out what the new ones do.

I am generally not keen on the idea of per-project migrations, though - it
makes what's in the database a property of the project, not the app, and
that's not how Django has worked traditionally. I think an effort to get a
more reliable, exposed global ordering of those individual app migrations
would go a long way towards the end goal without having to have migration
templates, upgrade instructions, and way more collisions between branches.

At the end of the day, though, there's a reason I made the schema editing
separate from the migration runners - you can re-use all the nasty work in
the schema editing interface and just replace the other part. This huge
change is the sort of thing I'd want to see working and proven before we
considered changing core, preferably as a third-party app, but of course
I'd like to talk through potential smaller changes first, rather than
throwing out the entire system.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqLBdv-0LGB6T1FS7eSxxKwjpfrVibXUT7KS-v9WCYK9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: RFC: DEP 7 - dependency policy

[Patryk Zawadzki]

W dniu sobota, 5 listopada 2016 13:24:28 UTC+1 użytkownik Jacob Kaplan-Moss 
napisał:
>
> Hi all -
>
> DEP 7 proposes a new dependency policy. In a nutshell, the policy is: 
> Python packaging is good now. Django can have dependancies.
>
> For full details, please check out the DEP: 
> https://github.com/django/deps/blob/master/draft/0007-dependency-policy.rst
>
> I'd appreciate any comments and feedback y'all have before I submit this 
> to the technical board for review. In particular, there are a couple of 
> things I'd like feedback on:
>
> - Are my criteria for "maturity" appropriate? Will they cover use-cases we 
> want to cover?
>
 
What are your concerns here? Couldn't Django eventually either take over or 
vendor-in the project if it ever becomes unresponsive?

Cheers,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/af985f1f-7f37-4527-9cf7-6603ed992627%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Rethinking migrations

[Patryk Zawadzki]

Greetings, Jazz Guitarists,

I've briefly talked about this with Markus and he mentioned that the 
subject was already brought up by Tyson Clugg but I think it deserves a 
proper discussion here.

I'm typing this from the comfort of Django: Under the Hood sprints so 
please excuse poor grammar and the somewhat chaotic explanations that 
follow. I'm very tired and English is not my mother tongue. This is not a 
DEP but merely a stream of consciousness I'd love to get some feedback on.

Here are some of the problems we face when dealing with migrations:

1. Dependency resolution that turns the migration dependency graph into an 
ordered list happens every time you try to create or execute a migration. 
If you have several hundred migrations it becomes quite slow. I'm talking 
multiple minutes kind of slow. As you can imagine working with multiple 
branches or perfecting your migrations quickly becomes a tedious task.

2. Dependency resolution is only stable as long as the migration set is 
frozen. Sometimes introducing a new migration is enough to break existing 
migrations by causing them to execute in a slightly different order. We 
often have to backtrack and edit existing migrations and enforce a strict 
resolution order by introducing arbitrary dependencies.

3. Removing an app from a project is a nightmare. You can't migrate to zero 
state unless the app is still there. There is no way to add "revert all 
migrations for app X" to the migration graph, it's something you need to 
run manually. There is no clean way to remove an app that was ever 
references in a relation. We were forced to do all kinds of hacks to get 
around this. Sometimes it's necessary to create an empty eggshell app with 
the same name and copy all migrations there then add necessary data 
migrations and finally migrations that remove all the models, indices, 
procedures etc. Sometimes people just leave a dead application in 
INSTALLED_APPS to not have to deal with this.

4. Squashing migrations is wonky at best. If you create a model in one 
migration, alter one of its fields in another and then finally drop the 
model sometime later, the squashed migration will have Django try to 
execute the alter first and complain about the table not being there. Also 
the only reason we need to squash migrations is to prevent problem 1 above 
from becoming exponentially worse. If migrations were only as slow as the 
underlying SQL commands, we'd likely never squash them.

5. There's no simple way to roll back all the migrations introduced after a 
particular point in time which is very useful when working with multiple 
feature branches. In my current project dropping the database means having 
to reimport over 200 MB of data snapshots. Switching branches requires me 
to look at branch diffs to determine which migrations to revert.

6. Conflict detection and resolution (migrate --merge) is a make-believe 
solution. It just trains people to execute the command without 
investigating whether their migration history still makes sense.


Some of these I need to dig deeper into and probably file proper tickets. 
For example I have an idea on how to fix 4 but it would make 1 even slower.

I took some time to get a good long look at what other ORMs are doing. The 
graph-based dependency solving approach is rather uncommon. Most systems 
treat migrations as part of the project rather than the packages it uses.


Possible solution (or "how I'd build it today if there was no existing code 
in Django core"):

a. Make migrations part of the project and not individual apps. This takes 
care of problem 3 above.

b. Prefix individual migration files with a UTC timestamp 
(20161105151023_add_foo) to provide a strict sorting order. This removes 
the depsolving requirement and takes care of 1 and 2. By eliminating those 
it makes 4 kind of obsolete as squashing migrations would become pointless.

c. Have reusable apps provide migration templates that Django then copies 
to my project when "makemigrations" is run.

d. Maintain a separate directory for each database connection.

e. Execute all migrations in alphabetical order (which means by timestamp 
first). When an unapplied migration is followed by an applied one, ask 
whether to attempt to just apply it or if the user wants to first unapply 
migrations that came after it. To me this would work better than 6.

f. Migrating to a timestamp solves 5.


Of course we do have migration support in core and it's not compatible with 
most of the above list. Any ideas? I think serializing the dependency 
solver state and reusing it between runs could be a pretty low hanging 
fruit (like "npm shrinkwrap" or yarn's lock file).

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, sen

Re: Getting full URL using django.urls.base.reverse - PR

[Florian Apolloner]

To be honest, I think I do not really see the usecase here. I know that 
sometimes it is required to generate full urls (especially in emails), but 
quite often in those you do not have access to the request at all. Also, 
how would this play together with the url tag? Personally I would set the 
current domain on the local "urlconf" (we have one per thread) and add a 
"full" kwarg to reverse (and something similar to the urltag).

Cheers,
Florian

On Saturday, November 5, 2016 at 3:41:09 PM UTC+1, Mislav Cimperšak wrote:
>
> During the sprint in Amsterdam I created a pull request 
> https://github.com/django/django/pull/7484
>
> In it I'm adding an optional argument `request` to 
> `django.urls.base.urls.reverse`method.
> The idea is that using `request` object one can get a fully qualified URL.
>
> The idea is taken from Django Rest Framework (
> https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/reverse.py#L55-L59
> ).
>
> I've talked to Tim Graham in person and he said that there are some 
> possibilities that django-hosts  
> will 
> be merged into main django code base with it's own implementation of 
> `reverse` method.
>
>
> To move this issue forward I need an input from core team members 
> regarding my implementation and future plans. If necessary I'll trash my 
> current implementation if it will just bring on more potentials problems in 
> the future.
>
>
> My arguments for suggested implementation is that it's much more simple 
> and easier for the enduser than django-hosts implementation and it's 
> already *Tom Christie approved(TM).*
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/83ff37c9-afb0-4c0e-8547-b0a7aa4fc5c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Language Specific collations in different databases

[Alan Kesselmann]

It looks like Mysql pretty much works like postgres.

Same Collate function works just fine, when i create database with utf8 
charset and then use collations like utf8_estonian_ci for 
estonian, utf8_general_ci for english and utf8_swedish_ci for swedish. I 
guess mysql experts can comment on if i should use mysql like this or not, 
but the Collate class seems to be working with the sql that was generated 
by Django:
SELECT ... FROM ... ORDER BY ... COLLATE "utf8_swedish_ci" DESC



On Saturday, November 5, 2016 at 5:19:44 PM UTC+2, Alan Kesselmann wrote:
>
> Hi!
>
> So during DUTH sprints i took on an issue that i created over 3 years ago: 
> https://code.djangoproject.com/ticket/21181...
>
> The actual fix or improvement for my case right now actually really 
> simple: 
>
> class Collate(Func):
> function ='COLLATE'
> template = '(%(expressions)s) COLLATE "%(function)s"'
>
> And then using it like this:
> ordering = Collate(
> F('first_name'),
> function=collation)
> qs = qs.order_by(ordering.desc())
>
> It was all actually given as answer here: 
> http://stackoverflow.com/questions/18935712/queryset-sorting-specifying-column-collation-for-django-orm-query
>
> Going over this and making it work for the other databases (other than 
> postgres), i need some help, because i know next to nothing about sqlite3, 
> mysql and oracle. This is where i could use little help.
>
> To get additional collations working with postgresql i first need to 
>  create actualy locale files in my own computer like this:
>
>
> sudo locale-gen et_EE.UTF-8
>
> sudo update-locale
>
>
> And then run the command in my database:
> CREATE COLLATION "et_EE" (LOCALE = "et_EE.utf8")
>
> And then i can use this collate function which will give me correct syntax 
> like:
> SELECT ... FROM ... ORDER BY ... COLLATE "et_EE" DESC
>
> So this is where i could use your help - can someone provide me 
> information about. Perhaps some of you can, with the cost of few minutes, 
> save me some experimenting and trial-n-error time, by answering these 
> questions:
> 1) how to do all these steps in sqlite3 (if at all possible and if not 
> then what should be fallback/error, using Collate with sqlite3)
> 2) how to do all these steps in mysql (if at all possible and if not then 
> what should be fallback/error, using Collate with mysql)
> 3) how to do all these steps in oracle (if at all possible and if not then 
> what should be fallback/error, using Collate with oracle)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fc92a834-1571-4969-8b74-e9e6a2b3f892%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Language Specific collations in different databases

[Alan Kesselmann]

Hi!

So during DUTH sprints i took on an issue that i created over 3 years 
ago: https://code.djangoproject.com/ticket/21181...

The actual fix or improvement for my case right now actually really simple: 

class Collate(Func):
function ='COLLATE'
template = '(%(expressions)s) COLLATE "%(function)s"'

And then using it like this:
ordering = Collate(
F('first_name'),
function=collation)
qs = qs.order_by(ordering.desc())

It was all actually given as answer 
here: 
http://stackoverflow.com/questions/18935712/queryset-sorting-specifying-column-collation-for-django-orm-query

Going over this and making it work for the other databases (other than 
postgres), i need some help, because i know next to nothing about sqlite3, 
mysql and oracle. This is where i could use little help.

To get additional collations working with postgresql i first need to 
 create actualy locale files in my own computer like this:


sudo locale-gen et_EE.UTF-8

sudo update-locale


And then run the command in my database:
CREATE COLLATION "et_EE" (LOCALE = "et_EE.utf8")

And then i can use this collate function which will give me correct syntax 
like:
SELECT ... FROM ... ORDER BY ... COLLATE "et_EE" DESC

So this is where i could use your help - can someone provide me information 
about. Perhaps some of you can, with the cost of few minutes, save me some 
experimenting and trial-n-error time, by answering these questions:
1) how to do all these steps in sqlite3 (if at all possible and if not then 
what should be fallback/error, using Collate with sqlite3)
2) how to do all these steps in mysql (if at all possible and if not then 
what should be fallback/error, using Collate with mysql)
3) how to do all these steps in oracle (if at all possible and if not then 
what should be fallback/error, using Collate with oracle)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d4aec91e-3293-444e-aab5-f41cc97c59cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: New deferred loading implementation (Was: [Review Request] Added support for database delegated fields (like AutoField))

[Ben Cole]

As discussed with many core team members (Simon Charette, Josh Smeaton, 
Marc Tamlyn, Tim Graham) at DUTH 2016 sprints, myself and Joachim 
Jablon have proposed a new, simpler solution to this. 
See https://code.djangoproject.com/ticket/27446

The proposal therefore is to add a `readonly` option to the base `Field` 
> class that when `True` would strip these fields from being compiled to SQL 
> during `INSERT`s and `UPDATE`s. This allows for a very simple change that 
> covers all possible write queries that Django may perform (including 
> bulk_*).
> There exists a proof of concept 
> https://github.com/novafloss/django-readonly-field


PR to follow soon...

On Friday, 12 February 2016 12:32:11 UTC, Anssi Kääriäinen wrote:
>
> On Thursday, February 11, 2016 at 2:41:44 PM UTC+2, Florian Apolloner 
> wrote:
>>
>> Oh, I somewhat missread and though there would be a new DEFERRED 
>> argument, the backwards issue is easy enough though:
>>
>>  * Unless I miss something, YourModel.__init__ is Model.__init__ if the 
>> user didn't change it -> pass is DEFERRED
>>  * If the user changed it check for model._meta.new_style_deferred and 
>> continue accordingly
>>  * Raise a warning if the __init__ is a custom one and new_style_deffered 
>> is not set…
>>
>
> If we are going to introduce a backwards compat system for this, then I 
> think I want to get rid of calling Model.__init__ at all when loading from 
> DB. We get faster model initialization, and conceptually loading from DB is 
> like unpickling which (correctly) doesn't call __init__.
>
> However, based on the comments in the PR, I think we are going to just 
> document the change to __init__ and skip deprecation.
>
>  - Anssi
>  
>
>>
>> On Thursday, February 11, 2016 at 1:38:44 PM UTC+1, Florian Apolloner 
>> wrote:
>>>
>>>
>>>
>>> On Thursday, February 11, 2016 at 10:51:59 AM UTC+1, Anssi Kääriäinen 
>>> wrote:

 Before doing any further work on this we should decide if the 
 Model.__init__() problem is bad enough to stop this cleanup, and if so, do 
 anybody have any ideas of how to avoid the problem?

>>>
>>> I do not think Model.__init__() is anywhere near public API, add it to 
>>> the release notes and be done with it, worst case add a try/except around 
>>> it…
>>>
>>> Cheers,
>>> Florian 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d34db1f8-db9e-45ef-a383-3573986677b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Getting full URL using django.urls.base.reverse - PR

[Mislav Cimperšak]

During the sprint in Amsterdam I created a pull request 
https://github.com/django/django/pull/7484

In it I'm adding an optional argument `request` to 
`django.urls.base.urls.reverse`method.
The idea is that using `request` object one can get a fully qualified URL.

The idea is taken from Django Rest Framework 
(https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/reverse.py#L55-L59).

I've talked to Tim Graham in person and he said that there are some 
possibilities that django-hosts  will 
be merged into main django code base with it's own implementation of 
`reverse` method.


To move this issue forward I need an input from core team members regarding 
my implementation and future plans. If necessary I'll trash my current 
implementation if it will just bring on more potentials problems in the 
future.


My arguments for suggested implementation is that it's much more simple and 
easier for the enduser than django-hosts implementation and it's already *Tom 
Christie approved(TM).*

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3ea6110b-0ddd-4f64-ac99-8854756157f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RFC: DEP 7 - dependency policy

[Andrew Godwin]

I think the "maturity" criteria are pretty sensible, though I am slightly
concerned about the potential for a project to be effectively unmaintained
even though there's someone's name on it who are active elsewhere.

Do you think there's a sensible way we could outline a few checks for what
it means to be maintained? It's not something I think we could make hard
and fast rules for, but a sort of checklist for us to use to check
dependencies every release might be useful.

Andrew

On Sat, Nov 5, 2016 at 1:23 PM, Jacob Kaplan-Moss 
wrote:

> Hi all -
>
> DEP 7 proposes a new dependency policy. In a nutshell, the policy is:
> Python packaging is good now. Django can have dependancies.
>
> For full details, please check out the DEP: https://github.com/
> django/deps/blob/master/draft/0007-dependency-policy.rst
>
> I'd appreciate any comments and feedback y'all have before I submit this
> to the technical board for review. In particular, there are a couple of
> things I'd like feedback on:
>
> - Are my criteria for "maturity" appropriate? Will they cover use-cases we
> want to cover?
>
> - Do we need more policy/process around dealing with potential abandonment
> issues?
>
> Thanks,
>
> Jacob
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CAK8PqJEMoT-v5SqLzauC6XtxQ%
> 2BAXSHJEUOSWcGody3r9i73f9w%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upRg4qm06XHQy%2BATQcuBfpk_y-XbS0Bsp4NfVn0Ww%3DprQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Channels Update, November 2016

[Andrew Godwin]

Hi everyone,

It's been almost a year since the Channels funding project started, and a
little while since it became an official Django project, and it's been hard
to know what the state of it is at times.

I'm going to start doing monthly updates on how Channels is progressing and
what's going on, and in this first update additionally reflect on the last
twelve months of work and what's happened since we applied to Mozilla for
funding.

Funding Retrospective
-

There was some delays in getting the funds through from Mozilla thanks to a
series of issues on their end and then our end, but work started almost
straight away anyway and invoices came later once everything was sorted out.

The initial few months were mostly getting the API and channel layers built
out, and getting Channels to a stable state where it could serve WebSocket
and HTTP requests. By about March this was there, and around this time I
proposed merging it into 1.10 but I retracted that proposal after some
valid feedback - more on that and the future of channels below.

The months since have been a series of polishing and small improvements as
we get feedback from both internal sources, such as the load testing we
funded using some of the MOSS money, as well as people running Channels in
development or production and filing bugs and feeding back when things went
well.

I've not been as good as I could have been at getting others involved under
the proposal system; around ten people have approached me interested in
working on a proposal but only five of those have got as far as sending one
in, leaving more of the work on me as I would like. Things are fortunately
looking up for getting a second paid maintainer on board, however, and I
hope to share more about this soon.

We've spent $60,000 of the $150,000 that we got from the fund; we're a bit
behind where I'd like to be at this point due to some of the early delays
but we have around another $20,000 in proposals/invoices coming up soon so
it feels like about the right rate of spend.

The plan is to continue to use the fund to keep Channels polishing going
(especially making sure it works well at larger scale), and to start adding
on more protocols and features (see below for more about that).

Current State
-

Let's start with where the project is at the moment. I'm currently prepping
for a 1.0 release and the first officially stable API, with a few
exceptions (data binding will be kept as an "alpha" API for now).

The 1.0 release will have a couple of major changes (these will be more
fully explained in the release notes):

 * WebSockets will have an accept/reject phase rather than always
accepting. This removes the need for strict ordering, but some code will
need a change as it's partially backwards-incompatible, and it's not
currently possible to provide a shim to fix this.

 * Sending from consumers is now delayed until after the consumer exits so
they are "atomic" (and discouraging long-running consumers). This is
backwards-compatible as long as consumers were being used as described, and
not being held open on some external trigger inside.

The third breaking change I am considering is changing how message sending
is provided inside Channels so that it's possible to modify via decorators
and superclasses - current ideas include providing a send() callable as a
second argument to consumers, or attaching it on the message object.

This might also enable a backwards-compatible shim for the websocket
acceptance change if it's implemented, which is another reason to consider
it.

Inclusion in Django
---

The effort to "include" Channels in Django started with a proposed merge
into 1.10, which was rejected with some sensible arguments and ultimately
failed. I did not think merging into 1.11 made sense, partially as it would
be a while before it was stable enough that it would be accepted and
particularly as 1.11 is an LTS.

(Some of the objections raised in the inclusion discussion have resulted in
changes in Channels' design or optimisations in the relevant code; I think
it was right to not put it in at that point, in retrospect.)

As a result, I worked on the "official projects" DEP to carve out a place
to have projects that are Django-the-project not Django-the-core-codebase,
and this seems to have gone down well (and opened the door to having more
of these in future as we consider how to move Django forwards)

Channels is now happily an official project, and there's no hurry to
include it so far; neither I nor other core team members I've talked to
know for sure if we'd want to include it in future or not, especially as
it's been a long-standing goal of ours to make Django more modular (but
Channels might end up being low level enough that it makes sense to merge).

It will partially be down to performance concerns, too; if Channels were to
be part of base Django it would need to not cause a performance impact in a
default installation.

RFC: DEP 7 - dependency policy

[Jacob Kaplan-Moss]

Hi all -

DEP 7 proposes a new dependency policy. In a nutshell, the policy is:
Python packaging is good now. Django can have dependancies.

For full details, please check out the DEP:
https://github.com/django/deps/blob/master/draft/0007-dependency-policy.rst

I'd appreciate any comments and feedback y'all have before I submit this to
the technical board for review. In particular, there are a couple of things
I'd like feedback on:

- Are my criteria for "maturity" appropriate? Will they cover use-cases we
want to cover?

- Do we need more policy/process around dealing with potential abandonment
issues?

Thanks,

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAK8PqJEMoT-v5SqLzauC6XtxQ%2BAXSHJEUOSWcGody3r9i73f9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.