date:20170215

Re: Should Model.save() fix incorrect types that happen to save correctly?

2017-02-15 Thread Robert Roskam

+1, documenting in this case seems to be the most appropriate. 

Robert Roskam


On Monday, February 13, 2017 at 10:51:59 AM UTC-5, Adam Johnson wrote:
>
> What do you think? Absent a better suggestion, we could document this 
>> pitfall so that there's something to point to when related tickets come in.
>
>
> +1, as Aymeric points out the more complex fields probably won't work with 
> coercion-on-set.
>
> On 13 February 2017 at 15:27, charettes > 
> wrote:
>
>> > What do you think? Absent a better suggestion, we could document this 
>> pitfall so that there's something to point to when related tickets come in.
>>
>> I also think this is the most appropriate solution.
>>
>> Le lundi 13 février 2017 09:57:49 UTC-5, Tim Graham a écrit :
>>>
>>> Once in a while, there's a ticket about this behavior:
>>>
>>> m = Model(decimal='12.9')
>>> m.save()
>>> self.assertEqual(m.decimal, '12.9')
>>> m.refresh_from_db()
>>> self.assertEqual(m.decimal, Decimal('12.9'))
>>>
>>> That is, you can create a model with an incorrect type and it won't be 
>>> fixed until you refresh the object from the database.
>>>
>>> Most recent complaint, "it is only a basic expectation that the DB layer 
>>> and the Application layer will correspond to each-other after performing 
>>> save, which is in other words, syncing your state with the DB. 
>>> Personally, this bug (one way binding between application and db on save) 
>>> broke many of my tests and took a lot of my time." [0] See [1] for another 
>>> manifestation of this.
>>>
>>> I think calling to_python() in Model.save() would have unacceptable 
>>> performance consequences for little benefit considering that it's a 
>>> reasonable expectation for developers to provide the correct type. Further, 
>>> you can use full_clean() to coerce to the correct types:
>>>
>>> m = Model(decimal='12.9')
>>> m.full_clean()
>>> self.assertEqual(m.decimal, Decimal('12.9'))
>>>
>>> What do you think? Absent a better suggestion, we could document this 
>>> pitfall so that there's something to point to when related tickets come in.
>>>
>>> [0] https://code.djangoproject.com/ticket/27825
>>> [1] https://code.djangoproject.com/ticket/24028
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/4e59974e-9916-434e-8840-4f42996e5052%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Adam
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4728e4e7-2810-441a-a729-109f84253215%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to merge django-csp into contrib

2017-02-15 Thread Tim Graham

Yes, if a ticket goes weeks or months without activity, it's unlikely 
someone is working on it, so it's fine to reassign.

On Wednesday, February 15, 2017 at 9:11:01 PM UTC-5, Robert Roskam wrote:
>
> Hey All,
>
> So it's over a year later, and even though there is consensus, this ticket 
> (https://code.djangoproject.com/ticket/15727) appears to have had no 
> progress.
>
> Would it be OK if someone else were to pick up this ticket and move it 
> forward?
>
> Robert Roskam
>
> On Sunday, December 6, 2015 at 11:22:20 AM UTC-5, Thomas Grainger wrote:
>>
>> Yeah this makes a lot more sense in SecurityMiddleware
>>
>> On Sunday, 6 December 2015 10:01:54 UTC, Florian Apolloner wrote:
>>>
>>>
>>>
>>> On Sunday, December 6, 2015 at 10:13:46 AM UTC+1, Aymeric Augustin wrote:

 I understand that the “just merge django-csp” approach minimizes 
 effort. However the result doesn’t feel well integrated with Django. I 
 left 
 a few comments that all boil down to “this isn’t how we would have done if 
 we’d started from a clean slate” and quickly gave up when I noticed that 
 trend.

>>>
>>> Ditto, I'd like to see this in core and as part of SecurityMiddleware.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d1f10383-7543-4049-a7b4-080d011bd7d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to merge django-csp into contrib

2017-02-15 Thread Robert Roskam

Hey All,

So it's over a year later, and even though there is consensus, this ticket 
(https://code.djangoproject.com/ticket/15727) appears to have had no 
progress.

Would it be OK if someone else were to pick up this ticket and move it 
forward?

Robert Roskam

On Sunday, December 6, 2015 at 11:22:20 AM UTC-5, Thomas Grainger wrote:
>
> Yeah this makes a lot more sense in SecurityMiddleware
>
> On Sunday, 6 December 2015 10:01:54 UTC, Florian Apolloner wrote:
>>
>>
>>
>> On Sunday, December 6, 2015 at 10:13:46 AM UTC+1, Aymeric Augustin wrote:
>>>
>>> I understand that the “just merge django-csp” approach minimizes effort. 
>>> However the result doesn’t feel well integrated with Django. I left a few 
>>> comments that all boil down to “this isn’t how we would have done if we’d 
>>> started from a clean slate” and quickly gave up when I noticed that trend.
>>>
>>
>> Ditto, I'd like to see this in core and as part of SecurityMiddleware.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3c475cca-1c43-40e5-8a09-5899d8732821%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Use signing backend for signed_cookie session engine

2017-02-15 Thread Danielle Madeley

Hi all,

Was looking at implementing a custom signing backend via a HSM to sign and 
validate my Django sessions without knowing the key.

It seems that the functions signing.loads() and signing.dumps() force you 
to use TimestampSigner [1] rather than calling get_cookie_signer(). This 
has the interesting side effect that response.set_signed_cookie() does go 
via the signing backend but the signed_cookie session engine is signed with 
a different backend.

Is there any reason the loads() and dumps() functions don't use the signing 
backend?

[1] https://github.com/django/django/blob/master/django/core/signing.py#L127

--danni

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b6f92f36-54f8-4371-96d0-7ac420d89838%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

2017-02-15 Thread charettes

Hi Raphael,

I've been working on making the optimizer a bit smarter recently and came 
to the
same conclusion as you concerning the "left" and "right" optimizations.

This should be possible to solve by allowing `Operation.reduce()` to return 
the
full list of operations it replaces by appending `in_betwen` before or 
after the
operations it combines to perform "left" or "right" optimizations.

In the mean time I have a solution that makes the optimizer perform a bit 
better
that I'd be glad to get your feedback on[1]. One thing bugging me was that
`RemoveField` operations didn't have any references to definition of the 
field
they were removing, making optimization "through" them unsafe as they could 
be
operating on a related field.

Cheers,
Simon

[1] https://github.com/django/django/pull/7999/files

Le mercredi 15 février 2017 07:22:11 UTC-5, rap...@makeleaps.com a écrit :
>
> Hello everyone,
>
>I spent a couple hours last night trying to improve the migration 
> squasher optimizer (migrations were taking almost 15 minutes in CI). I came 
> up with a couple ideas for anyone interested in improvements:
>  
>  1- Having an interactive mode for squashing would be interested. 
> Currently, when squashing migrations, I do the following: 
>
>- Generate an initial squash
>- Edit it (namely, move around operations to get more optimizations to 
>work)
>- remove the "replaces" tag, then rerun migration squashing to 
>"re-optimize"
>- repeat until I get something I like, then add the original 
>"replaces" tag
>
>It would be cool if instead, the process were (with a flag):
>
>- Generate an initial squash, but have the process wait for 
>confirmation to "commit" this squash as final (though writing out the file)
>- Edit the file, and tell the process to try re-optimizing with the 
>same file (getting around the "no-squash of squashes" rule)
>- Potentially, allow us to also step back
>
>  For example, the "squashmigrations" command output could look like:
>
> generated 0001_squashed_mig.py
>> optmize migration[yN]? 
>> 
>> regenerated 0001_squashed_mig.py
>> ( 20 operations -> 10 operations)
>> optimize migration[Ynr]? 
>> 
>> regenerated 0001_squashed_mig.py
>> ( No change in operation count)
>> optimize migration[Ynr]? 
>> 
>> rolled back to previous version
>> optimize migration[Ynr]? 
>> 
>> Saved migration
>>
>
>
> A simpler version of this command would simply be to add an 
> "optimizemigration" command that just reads in a single migration and 
> optimizes the operations, without touching any of the squashiness. 
>
>
>  2- The reducer might be a bit too pessimistic
>
>  Currently, the optimizer lets "reduce" operations (that take 2 operations 
> and return 0,1, or 2 operations, or None if nothing can be change) do 
> whatever they want. Because of that, if you have [A, B, C ,D] and B depends 
> on A, you can't reduce A and C because the reduction might remove A.
>
>  In reality there are two kinds of reduction operations that we could be 
> taking into account:
>
>- reducing "left". if you have [A, B, C, D] (for example, A is a 
> CreateModel , C is an AddField for the same model),  and you can reduce A 
> and C into just A' (A with C), giving [A', B, D], then it doesn't matter 
> that B depends on A. 
>
> The thing that matters is if C depends on B (for example, C adds a foreign 
> key to a model created in B). This is actually already encoded in the 
> CreateMode + AddField reduction, but is perhaps a more general case.
>
> In a sense, reducing A and C "to the left" means that we're bringing A and 
> C closer together only by moving C. This is a major part of the potential 
> reductions that the current optimizer is missing.
>
>- reducing right. If you have [A, B, C, D] (for example, A is a 
> CreateModel, C is a RemoveModel for the same model), and you can reduce A 
> and C into just C' (C with A), giving [B, C', D], then it does matter that 
> B depends on A. C can't depend on B (assuming causality holds in our 
> universe)
>
> This is the current mechanism, essentially. If B depends on A, then you 
> can't move A past B. 
>
>  Removing both operations is a special case of reducing right (You can 
> make C' into a no-op).
>
> I had monkeypatched a special case of  reducing left (taking CreateModel, 
> AddField of different models and swapping them . For example 
> [CreateModel(A), CreateModel(B), AddField(A.foo)] -> [CreateModel(A), 
> AddField(A.foo), CreateModel(B)]) and got decent results, but I think 
> making the optimization code express these two concepts separately would 
> catch even more of the optimizations I saw that the optimizer didn't.
>
>
> I hope some of this is useful 
>  
>   Raphael
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developer

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Andrew Godwin

>
>
> You're essentially speaking about 2 things here, in my opinion:
>
> 1. Adding a new feature for interactive squash
> 2. Improving the MigrationOptimizer
>
> I certainly see a point for 2. Not sure how much for 1. Anyway, your
> reasoning for 2 sounds great! I'd be more than happy if you want to get
> this into Django! Can you create a respective ticket and drop your
> explanation in there, please :)
>
>
I am also definitely in support of 2), and I can see 1) being useful for
migration sets that have a lot of operations the optimiser won't touch by
default (e.g. SQL), but it's probably more work to be less generally useful.

Andrew


>
> On Wednesday, February 15, 2017 at 1:22:11 PM UTC+1, rap...@makeleaps.com
> wrote:
>>
>> Hello everyone,
>>
>>I spent a couple hours last night trying to improve the migration
>> squasher optimizer (migrations were taking almost 15 minutes in CI). I came
>> up with a couple ideas for anyone interested in improvements:
>>
>>  1- Having an interactive mode for squashing would be interested.
>> Currently, when squashing migrations, I do the following:
>>
>>- Generate an initial squash
>>- Edit it (namely, move around operations to get more optimizations
>>to work)
>>- remove the "replaces" tag, then rerun migration squashing to
>>"re-optimize"
>>- repeat until I get something I like, then add the original
>>"replaces" tag
>>
>>It would be cool if instead, the process were (with a flag):
>>
>>- Generate an initial squash, but have the process wait for
>>confirmation to "commit" this squash as final (though writing out the 
>> file)
>>- Edit the file, and tell the process to try re-optimizing with the
>>same file (getting around the "no-squash of squashes" rule)
>>- Potentially, allow us to also step back
>>
>>  For example, the "squashmigrations" command output could look like:
>>
>> generated 0001_squashed_mig.py
>>> optmize migration[yN]?
>>> 
>>> regenerated 0001_squashed_mig.py
>>> ( 20 operations -> 10 operations)
>>> optimize migration[Ynr]?
>>> 
>>> regenerated 0001_squashed_mig.py
>>> ( No change in operation count)
>>> optimize migration[Ynr]?
>>> 
>>> rolled back to previous version
>>> optimize migration[Ynr]?
>>> 
>>> Saved migration
>>>
>>
>>
>> A simpler version of this command would simply be to add an
>> "optimizemigration" command that just reads in a single migration and
>> optimizes the operations, without touching any of the squashiness.
>>
>>
>>  2- The reducer might be a bit too pessimistic
>>
>>  Currently, the optimizer lets "reduce" operations (that take 2
>> operations and return 0,1, or 2 operations, or None if nothing can be
>> change) do whatever they want. Because of that, if you have [A, B, C ,D]
>> and B depends on A, you can't reduce A and C because the reduction might
>> remove A.
>>
>>  In reality there are two kinds of reduction operations that we could be
>> taking into account:
>>
>>- reducing "left". if you have [A, B, C, D] (for example, A is a
>> CreateModel , C is an AddField for the same model),  and you can reduce A
>> and C into just A' (A with C), giving [A', B, D], then it doesn't matter
>> that B depends on A.
>>
>> The thing that matters is if C depends on B (for example, C adds a
>> foreign key to a model created in B). This is actually already encoded in
>> the CreateMode + AddField reduction, but is perhaps a more general case.
>>
>> In a sense, reducing A and C "to the left" means that we're bringing A
>> and C closer together only by moving C. This is a major part of the
>> potential reductions that the current optimizer is missing.
>>
>>- reducing right. If you have [A, B, C, D] (for example, A is a
>> CreateModel, C is a RemoveModel for the same model), and you can reduce A
>> and C into just C' (C with A), giving [B, C', D], then it does matter that
>> B depends on A. C can't depend on B (assuming causality holds in our
>> universe)
>>
>> This is the current mechanism, essentially. If B depends on A, then you
>> can't move A past B.
>>
>>  Removing both operations is a special case of reducing right (You can
>> make C' into a no-op).
>>
>> I had monkeypatched a special case of  reducing left (taking CreateModel,
>> AddField of different models and swapping them . For example
>> [CreateModel(A), CreateModel(B), AddField(A.foo)] -> [CreateModel(A),
>> AddField(A.foo), CreateModel(B)]) and got decent results, but I think
>> making the optimization code express these two concepts separately would
>> catch even more of the optimizations I saw that the optimizer didn't.
>>
>>
>> I hope some of this is useful
>>
>>   Raphael
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@goo

Re: Some thoughts about improving migration squashing

2017-02-15 Thread raphael

Markus: Now that I've written a command to optimize a single migration 
file, I think that it's sufficient for the "squash, edit, optimize" 
workflow that I was doing before. It's more about offering people to get 
their squashing done well until our optimizer becomes omniscient. 

Florian: Having the command run on all migrations should be 
straightforward, I'll look into that!

Tim: It's not untested! I tested it locally by running it ;) I'll write up 
a couple test cases and some documentation.


I've posted one ticket  for 
the management command to pass a selected migration through the optimizer, 
and another ticket  for 
improvements to the optimizer itself.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/686a3cd9-c15c-4619-a2f7-5aa9f9978910%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Markus Holtermann

Thanks Raphael, that's a pretty good write up!

You're essentially speaking about 2 things here, in my opinion:

1. Adding a new feature for interactive squash
2. Improving the MigrationOptimizer

I certainly see a point for 2. Not sure how much for 1. Anyway, your 
reasoning for 2 sounds great! I'd be more than happy if you want to get 
this into Django! Can you create a respective ticket and drop your 
explanation in there, please :)

Cheers,

/Markus

On Wednesday, February 15, 2017 at 1:22:11 PM UTC+1, rap...@makeleaps.com 
wrote:
>
> Hello everyone,
>
>I spent a couple hours last night trying to improve the migration 
> squasher optimizer (migrations were taking almost 15 minutes in CI). I came 
> up with a couple ideas for anyone interested in improvements:
>  
>  1- Having an interactive mode for squashing would be interested. 
> Currently, when squashing migrations, I do the following: 
>
>- Generate an initial squash
>- Edit it (namely, move around operations to get more optimizations to 
>work)
>- remove the "replaces" tag, then rerun migration squashing to 
>"re-optimize"
>- repeat until I get something I like, then add the original 
>"replaces" tag
>
>It would be cool if instead, the process were (with a flag):
>
>- Generate an initial squash, but have the process wait for 
>confirmation to "commit" this squash as final (though writing out the file)
>- Edit the file, and tell the process to try re-optimizing with the 
>same file (getting around the "no-squash of squashes" rule)
>- Potentially, allow us to also step back
>
>  For example, the "squashmigrations" command output could look like:
>
> generated 0001_squashed_mig.py
>> optmize migration[yN]? 
>> 
>> regenerated 0001_squashed_mig.py
>> ( 20 operations -> 10 operations)
>> optimize migration[Ynr]? 
>> 
>> regenerated 0001_squashed_mig.py
>> ( No change in operation count)
>> optimize migration[Ynr]? 
>> 
>> rolled back to previous version
>> optimize migration[Ynr]? 
>> 
>> Saved migration
>>
>
>
> A simpler version of this command would simply be to add an 
> "optimizemigration" command that just reads in a single migration and 
> optimizes the operations, without touching any of the squashiness. 
>
>
>  2- The reducer might be a bit too pessimistic
>
>  Currently, the optimizer lets "reduce" operations (that take 2 operations 
> and return 0,1, or 2 operations, or None if nothing can be change) do 
> whatever they want. Because of that, if you have [A, B, C ,D] and B depends 
> on A, you can't reduce A and C because the reduction might remove A.
>
>  In reality there are two kinds of reduction operations that we could be 
> taking into account:
>
>- reducing "left". if you have [A, B, C, D] (for example, A is a 
> CreateModel , C is an AddField for the same model),  and you can reduce A 
> and C into just A' (A with C), giving [A', B, D], then it doesn't matter 
> that B depends on A. 
>
> The thing that matters is if C depends on B (for example, C adds a foreign 
> key to a model created in B). This is actually already encoded in the 
> CreateMode + AddField reduction, but is perhaps a more general case.
>
> In a sense, reducing A and C "to the left" means that we're bringing A and 
> C closer together only by moving C. This is a major part of the potential 
> reductions that the current optimizer is missing.
>
>- reducing right. If you have [A, B, C, D] (for example, A is a 
> CreateModel, C is a RemoveModel for the same model), and you can reduce A 
> and C into just C' (C with A), giving [B, C', D], then it does matter that 
> B depends on A. C can't depend on B (assuming causality holds in our 
> universe)
>
> This is the current mechanism, essentially. If B depends on A, then you 
> can't move A past B. 
>
>  Removing both operations is a special case of reducing right (You can 
> make C' into a no-op).
>
> I had monkeypatched a special case of  reducing left (taking CreateModel, 
> AddField of different models and swapping them . For example 
> [CreateModel(A), CreateModel(B), AddField(A.foo)] -> [CreateModel(A), 
> AddField(A.foo), CreateModel(B)]) and got decent results, but I think 
> making the optimization code express these two concepts separately would 
> catch even more of the optimizations I saw that the optimizer didn't.
>
>
> I hope some of this is useful 
>  
>   Raphael
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bcc73cef-e6f8-4aaf-b5a9-4592895b3b2e%40googlegroups.com.
For more options, visit

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Markus Holtermann

What might be interesting to look into when squashing all migrations in one 
app would be to assume no migrations would exist. That could then result in 
only 2 migrations which could run through the optimizer (as opposed to 
let's say 20 migrations with many more operations).

/Markus

On Wednesday, February 15, 2017 at 3:46:01 PM UTC+1, Florian Apolloner 
wrote:
>
> Fwiw I think by default it could/should try to optimize all migrations of 
> an app, manually specifying the migration name should be optional.
>
> On Wednesday, February 15, 2017 at 2:00:54 PM UTC+1, rap...@makeleaps.com 
> wrote:
>>
>> I ended up having some time today, so wrote up a management command for 
>> the first suggestion!
>>
>> I called it "optizemigration"
>>
>>
>> >>> ./manage.py optimizemigration appname 0001_squashed
>>   # snipped django startup noise
>>   Optimized from 9 operations to 4 operations
>>
>> Optimized migration /Users/rtpg/proj/projname/projname/appname/migrations
>> /0001_squashed_20170215.py
>>
>>
>> This reads in the migration file, runs the migration optimizer, and then 
>> outputs to the same file. Writing it has paid off almost immediately for me.
>>
>> Those who are interested can take a look here 
>> 
>> .
>>
>> How much testing/coverage requirements are there for management commands 
>> like these?
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/24896164-cf2f-4eb9-8d87-3932f2c30e78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Florian Apolloner

Fwiw I think by default it could/should try to optimize all migrations of 
an app, manually specifying the migration name should be optional.

On Wednesday, February 15, 2017 at 2:00:54 PM UTC+1, rap...@makeleaps.com 
wrote:
>
> I ended up having some time today, so wrote up a management command for 
> the first suggestion!
>
> I called it "optizemigration"
>
>
> >>> ./manage.py optimizemigration appname 0001_squashed
>   # snipped django startup noise
>   Optimized from 9 operations to 4 operations
>
> Optimized migration /Users/rtpg/proj/projname/projname/appname/migrations/
> 0001_squashed_20170215.py
>
>
> This reads in the migration file, runs the migration optimizer, and then 
> outputs to the same file. Writing it has paid off almost immediately for me.
>
> Those who are interested can take a look here 
> 
> .
>
> How much testing/coverage requirements are there for management commands 
> like these?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6055f360-b916-4bd3-8b9a-eea3768c1241%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Tim Graham

Hi Raphael,

It looks like a similar idea was proposed in 
https://groups.google.com/d/topic/django-developers/C1L-NhyQYG4/discussion. 
I don't think a ticket was ever created, so you can do that.

100% test coverage is required. Why would we accept untested code? ;-)

On Wednesday, February 15, 2017 at 8:00:54 AM UTC-5, rap...@makeleaps.com 
wrote:
>
> I ended up having some time today, so wrote up a management command for 
> the first suggestion!
>
> I called it "optizemigration"
>
>
> >>> ./manage.py optimizemigration appname 0001_squashed
>   # snipped django startup noise
>   Optimized from 9 operations to 4 operations
>
> Optimized migration /Users/rtpg/proj/projname/projname/appname/migrations/
> 0001_squashed_20170215.py
>
>
> This reads in the migration file, runs the migration optimizer, and then 
> outputs to the same file. Writing it has paid off almost immediately for me.
>
> Those who are interested can take a look here 
> 
> .
>
> How much testing/coverage requirements are there for management commands 
> like these?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c9d9a4d2-49ab-448b-8e5c-13ea3795c2dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

2017-02-15 Thread raphael

I ended up having some time today, so wrote up a management command for the 
first suggestion!

I called it "optizemigration"


>>> ./manage.py optimizemigration appname 0001_squashed
  # snipped django startup noise
  Optimized from 9 operations to 4 operations

Optimized migration /Users/rtpg/proj/projname/projname/appname/migrations/
0001_squashed_20170215.py


This reads in the migration file, runs the migration optimizer, and then 
outputs to the same file. Writing it has paid off almost immediately for me.

Those who are interested can take a look here 

.

How much testing/coverage requirements are there for management commands 
like these?


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b4924688-1001-4a2b-a8de-20e6a10ff83f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Some thoughts about improving migration squashing

2017-02-15 Thread raphael

Hello everyone,

   I spent a couple hours last night trying to improve the migration 
squasher optimizer (migrations were taking almost 15 minutes in CI). I came 
up with a couple ideas for anyone interested in improvements:
 
 1- Having an interactive mode for squashing would be interested. 
Currently, when squashing migrations, I do the following: 

   - Generate an initial squash
   - Edit it (namely, move around operations to get more optimizations to 
   work)
   - remove the "replaces" tag, then rerun migration squashing to 
   "re-optimize"
   - repeat until I get something I like, then add the original "replaces" 
   tag

   It would be cool if instead, the process were (with a flag):

   - Generate an initial squash, but have the process wait for confirmation 
   to "commit" this squash as final (though writing out the file)
   - Edit the file, and tell the process to try re-optimizing with the same 
   file (getting around the "no-squash of squashes" rule)
   - Potentially, allow us to also step back

 For example, the "squashmigrations" command output could look like:

generated 0001_squashed_mig.py
> optmize migration[yN]? 
> 
> regenerated 0001_squashed_mig.py
> ( 20 operations -> 10 operations)
> optimize migration[Ynr]? 
> 
> regenerated 0001_squashed_mig.py
> ( No change in operation count)
> optimize migration[Ynr]? 
> 
> rolled back to previous version
> optimize migration[Ynr]? 
> 
> Saved migration
>


A simpler version of this command would simply be to add an 
"optimizemigration" command that just reads in a single migration and 
optimizes the operations, without touching any of the squashiness. 


 2- The reducer might be a bit too pessimistic

 Currently, the optimizer lets "reduce" operations (that take 2 operations 
and return 0,1, or 2 operations, or None if nothing can be change) do 
whatever they want. Because of that, if you have [A, B, C ,D] and B depends 
on A, you can't reduce A and C because the reduction might remove A.

 In reality there are two kinds of reduction operations that we could be 
taking into account:

   - reducing "left". if you have [A, B, C, D] (for example, A is a 
CreateModel , C is an AddField for the same model),  and you can reduce A 
and C into just A' (A with C), giving [A', B, D], then it doesn't matter 
that B depends on A. 

The thing that matters is if C depends on B (for example, C adds a foreign 
key to a model created in B). This is actually already encoded in the 
CreateMode + AddField reduction, but is perhaps a more general case.

In a sense, reducing A and C "to the left" means that we're bringing A and 
C closer together only by moving C. This is a major part of the potential 
reductions that the current optimizer is missing.

   - reducing right. If you have [A, B, C, D] (for example, A is a 
CreateModel, C is a RemoveModel for the same model), and you can reduce A 
and C into just C' (C with A), giving [B, C', D], then it does matter that 
B depends on A. C can't depend on B (assuming causality holds in our 
universe)

This is the current mechanism, essentially. If B depends on A, then you 
can't move A past B. 

 Removing both operations is a special case of reducing right (You can make 
C' into a no-op).

I had monkeypatched a special case of  reducing left (taking CreateModel, 
AddField of different models and swapping them . For example 
[CreateModel(A), CreateModel(B), AddField(A.foo)] -> [CreateModel(A), 
AddField(A.foo), CreateModel(B)]) and got decent results, but I think 
making the optimization code express these two concepts separately would 
catch even more of the optimizations I saw that the optimizer didn't.


I hope some of this is useful 
 
  Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6042857b-fd48-467a-badc-1cf5305ecac4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Model translation and meta options

2017-02-15 Thread James Pic

On Wed, Feb 15, 2017 at 12:21 AM, Adam Johnson  wrote:

> Sorry for the terrible pun here, but I'd like to suggest the
> meta-feature... allowing 3rd party apps to add their own options to Meta
> classes. If there was a sensible API for this (or if Django just copied all
> attributes defined in Meta onto _meta blindly), Django wouldn't have to
> add the translatable option without defining its behaviour.  Instead the
> third party apps could all define their own and use that.
>

I recon that was my initial thought too (this is also something we could
use for ModelForms BTW). While that would also work, I still think that
trying to make model data intl support as great as intl support everywhere
else in Django is a good idea - but I'm a non English native so I guess I
have strong feelings about this: without model intl I just can't use django
to communicate with my people. I still /believe/ "Django admin should at
least support translation of data, somehow, at some point". If we can have
a consensus on the requirement itself then it will be easier I think to
decide which first step to take to get there.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op1-T3a7hcXoQBenQYd3kRMkGgwi7OTyxE1UAc59feAS_YA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Should Model.save() fix incorrect types that happen to save correctly?

Re: Proposal to merge django-csp into contrib

Re: Proposal to merge django-csp into contrib

Use signing backend for signed_cookie session engine

Re: Some thoughts about improving migration squashing

Re: Some thoughts about improving migration squashing

Re: Some thoughts about improving migration squashing

Re: Some thoughts about improving migration squashing

Re: Some thoughts about improving migration squashing

Re: Some thoughts about improving migration squashing

Re: Some thoughts about improving migration squashing

Re: Some thoughts about improving migration squashing

Some thoughts about improving migration squashing

Re: Model translation and meta options

14 matches

Site Navigation

Mail list logo

Footer information