Re: [off topic] Re: models.CalculatedField feature

[Anssi Kääriäinen]

Even simple comparisons can be difficult to simulate in Python. As an example, 
some databases treat empty strings as null. Still, autogenerating the Python 
side of simple computations is an interesting idea. At least worth a try.

There are some hard problems if trying to push all computations to the db. 
First problem is that this won't work for unsaved models. Second, if the value 
isn't fetched on initial select, the database values might change. This is 
almoust the same problem as when changing values on Python side. Finally, 
accessing the db just to do a simple computation isn't a good from performance 
perspective.

For using Python in the DB, on PostgreSQL Python isn't available in hosted 
environments as Python can access the db machine.

I'd take a look on what it requires to implement this feature as 3rd party 
addition, and then do the required changes to core.

- Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ed9c5893-44a9-4fda-bd97-1ff409de69c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Database "execute hooks" for instrumentation

[Anssi Kääriäinen]

Would it make sense to use the same technique used for HTTP request/response 
middleware? That is, the hook would look a bit like this:

def simple_execute_hook(execute):
# One-time configuration and initialization.
def execute_hook(sql, params, many, context):
# Code to be executed for each cursor.execute() call.
# If many = True, the final call will be execute_many.
# The context parameter might contain stuff like used
# connection.
execute(sql, params, many, context)
# Code to be executed after the SQL has been ran.
return execute_hook

You would then add the hook with the connection's context manager.

The reason I'm asking is that this way the coding style would be immediately 
familiar if you have used the request/response middlewares.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e6098f14-0f0a-4ae6-b094-bfd9e2ccb51a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Having a MongoDB connector for Django

[Anssi Kääriäinen]

My 2 cents on this...

I don't think it would be an unachievable goal to write a MongoDB backend for 
Django's ORM. However, it wouldn't support any relational feature, and likely 
would also need to skip support for some other common features, for example 
AutoField is actually hard to support on MongoDB. The Django ecosystem is very 
much written on an assumption that the underlying datastore is relational, so 
this would mean that even if you have MongoDB backend, you can't use it with 
most contrib models for example.

Even if you can't use the backend with the full ecosystem, such a backend might 
be very useful for some use cases. My guess is that the most common case would 
be usage of MongoDB as an additional data store along your relational database.

I believe that most users requesting a MongoDB backend actually would want to 
see a backend which is a drop in replacement for the SQL backends. 
Unfortunately, due to the differences in optimal database schema design between 
relational and document oriented data stores, this is both a bad idea, and 
almost impossible to implement.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7f6391f8-e477-4573-b7f9-06634d1b7957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom Join Conditions

[Anssi Kääriäinen]

I really like the .annotate() version. The way this feature works with 
.annotate() is easy to understand if you understand how existing .annotate() 
functions work. But, if we go with this approach, it might look a bit strange 
if we don't do an automatic select_related(). On the other hand, there are 
cases where you definitely don't want to do a select_related(), like if the 
relation annotation is only used in further aggregation. So, maybe there should 
be opt-out, or maybe we just don't do any automatic select_related for this 
case.

I do like the .annotate() version for other reasons, too. There are variants of 
this feature which would work nicely with the annotate style, that is either 
annotation querysets for objects that do *not* have a relation to the original 
queryset, or annotation raw SQL directly.

 - Anssi
PS. It seems the Google Groups UI acts a bit strange for me today. I hope this 
post does arrive to the group in a readable format...

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4ceb0715-bd6b-4860-a6e6-77dc9ecabae6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Optimizing out unused annotations from count queries

[Anssi Kääriäinen]

On Monday, August 21, 2017 at 11:28:35 AM UTC+3, Tom Forbes wrote:
>
> > Interestingly enough, just doing a .filter(m2m_relation__foo='bar') 
> might change the results
>
> Is that not because it's a filter? Would ". 
> annotate(x=F('m2m_relation__foo'))" change the results in any way? Sorry if 
> I'm not following.
>

What's happening here is that Django generates an unconstrained join to the 
m2m_relation. Unsurprisingly that generates duplicate rows to the queryset. 
For example:

class Publisher:
name = CharField()

class Book:
 publisher = ForeignKey(Publisher, related_name='books')

print(Publisher.objects.annotate(book_id=F('books__id')).query)
OUT: SELECT publisher.id, book.id as book_id FROM publisher JOIN book ON 
book.publisher_id = publisher.id

Now, if there are two books fo any publisher, that publisher will be 
outputted twice in the resulting queryset.

Normally you wouldn't want to do this, but even if you do this by accident, 
count() and len(qs) should return the same number. And, for .values() 
querysets this can be actually useful in some cases.

Note that it's possible to remove the select entry for the annotation, even 
if the join can't be removed in this case.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d9f2cded-ef37-4852-8845-1fcc126457ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Optimizing out unused annotations from count queries

[Anssi Kääriäinen]

On Sunday, August 20, 2017 at 2:48:23 AM UTC+3, Josh Smeaton wrote:

> Thanks for making me elaborate, because I'm probably wrong. I'll step 
> through what my thinking was though.
>
> '.annotate(local=F('related__field'))' will join to the related table. If 
> `related` is a nullable foreign key, then a join would do an implicit 
> filter, removing the row from the result set if it had no relation, and 
> reducing the count. 
>
> But Django models a nullable foreign key join as a LEFT JOIN which 
> wouldn't filter the row out and wouldn't reduce the count.
>
> Multivalue relationships (reverse foreign key and m2m joins) would need 
> some care though I think. I'm not sure if the same annotation above works 
> on m2m joins or not.
>

Interestingly enough, just doing a .filter(m2m_relation__foo='bar') might 
change the results. For m2m relations, any non-aggregate annotation is not 
safe to remove.

I believe our test cases are not fully covering here, so removing the 
annotations will be a bit risky. If there are cases where removal is known 
to be safe, it's of course a good idea to remove anything extra from the 
query.

 - Anssi 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/51a5c1ad-e071-482b-aa3a-184dd4145413%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Automatic prefetching in querysets

[Anssi Kääriäinen]

On Thursday, August 17, 2017 at 11:30:45 PM UTC+3, Aymeric Augustin wrote:
>
> Hello, 
>
> > On 17 Aug 2017, at 14:48, Luke Plant > 
> wrote: 
> > 
> > On 16/08/17 23:17, Aymeric Augustin wrote: 
> >> It should kick in only when no select_related or prefetch_related is in 
> effect, to avoid interfering with pre-existing optimizations. It's still 
> easy to construct an example where it would degrade performance but I don't 
> think such situations will be common in practice. 
> > 
> > I think Ansii's example is a non-contrived and potentially very common 
> one where we will make things worse if this is default behaviour, 
> especially for the non-expert programmer this behaviour is supposed to 
> help: 
> > 
> > 
> > if qs: 
> > qs[0].do_something() 
> > 
> > (where do_something() access relations, or multiple relations). We will 
> end up with *multiple* large unnecessary queries instead of just one. The 
> difference is that the first one is easy to explain, the remainder are much 
> harder, and will contribute even more to the "Django is slow/ORMs are bad" 
> feeling. 
>
>
> I wasn't convinced by this example because it's already sub-optimal 
> currently. `if qs` will fetch the whole queryset just to check if it has at 
> least one element. 
>
> Assuming there's 1000 elements in qs, if you don't care about a 1000x 
> inefficiency, I don't expect you to care much about two or three 1000x 
> inefficiencies...
>

I agree that this example isn't particularly worrying. It's something an 
experienced developer wouldn't do. On the other hand, we are aiming at 
making things simpler for non-experienced developers.

To me the worrying part here is that we really don't have any data or 
experience about if the cure will be worse than the disease. Likely not, 
but what do we gain by taking risks here?

Maybe we should just add the queryset method. This is the smallest atomic 
task that can be done. Even if there's only the queryset method available, 
it's possible to enable prefetches per model by using a Manager.

 - Anssi


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c9d9dca0-f71d-48dd-9985-17ef8b8ec95d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Automatic prefetching in querysets

[Anssi Kääriäinen]

On Wednesday, August 16, 2017 at 2:27:11 PM UTC+3, Josh Smeaton wrote:
>
> It won't affect experienced users. They'll read the release notes, see 
>> that this change has been implemented, and either go and delete a bunch of 
>> prefetch_related() calls, grumble a bit and turn auto-prefetch off globally 
>> or just file it away as another fact they know about the Django ORM.
>>
>
> This is, for me, the biggest point in favour of having it opt out. It'll 
> barely affect anyone who is really experienced, but can provide such a 
> large upside for others.
>

There is huge potential for this feature going very bad for performance. 
For example, this common anti-pattern would immediately cause interesting 
issues:

if qs:
qs[0].do_something()

then if do_something() does access a couple of relations, you'll load data 
for the whole queryset where you only need it for the first one. So, at 
least make this opt-in instead of opt-out.

Doing this for "select_related" cases on opt-in basis seems like a 
plausible idea. Doing this for reverse foreign key or m2m cases where you 
need to use prefetch_related seems complex to implement, and just for that 
reason I'd avoid the prefetch related case at least initially.

Note that it should be possible to implement this fully as 3rd party 
project if we add an easy way to customise how related object fetching is 
done. I'm not sure if we can add an easy way for that.

By the way, a very efficient test for N+1 queries issue is to do something 
like this:

def test_nplusone_queries(self):
self.create_object()
with self.count_queries() as cnt:
self.client.get('object_listing/')
self.create_object()
with self.assertNumQueries(cnt):
self.client.get('object_listing/')

that is, you test the same list view with one and two objects. If there was 
an easy way to write tests like this, that would make n+1 queries problems 
a lot easier to detect. Maybe something to make this kind of test trivial 
to write could be in Django? Something like 
self.assertListViewScales(object_creator, 'view_name')?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1dcd8559-5f99-4635-b1d1-959bea066192%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Support for database level on delete cascade 21961

[Anssi Kääriäinen]

On Tuesday, July 25, 2017 at 5:34:33 AM UTC+3, Nick Stefan wrote:
>
> 5. Q: implicit foreign keys between child to parent in model inheritance?
> A: We could forbid models.DB_CASCADE on these types of relations?
>

There's actually a variation of this that seems a bit hairy to solve 
properly. For models:

class Category:
pass

class Child:
pass

class Parent(Child):
category = models.ForeignKey(Category, on_delete=DB_CASCADE)

Then, when you delete a category instance, the associated parent instance 
will get deleted, but the child instance won't (as there's no foreign key 
from child to parent, we can't make this work in the DB without using 
triggers).

Now, the problem here is that this actually leaves the database in an 
inconsistent state. I guess the solution might be to document this, add a 
warning to checks done on startup, and let users deal with the 
inconsistency if they so decide.

We could solve these problems by including an O2O from both parent to child 
and from child to parent, and making them both DB_CASCADE. That's not a 
good idea for other reasons (complexity of saving a new instance to the 
database for one). If we want to make these cases work seamlessly directly 
in the database, it's likely better to add in triggers. It wouldn't be 
horribly complex code to write. The hardest part would be making sure the 
triggers are kept in sync when doing migrations.

For the other parts your proposal does make a lot of sense to me. Don't 
fall in the trap of complexities in parent -> child cascade. Just add 
warnings and that's it.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3b9085e1-68c9-484e-af0f-aa330f2f074b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding generated common table expressions

[Anssi Kääriäinen]

On Thursday, April 6, 2017 at 11:53:32 AM UTC+3, Marc Tamlyn wrote:
>
> Regarding Anssi's comments about SubQuery, we do now have that in core as 
> of 1.11 [0]. It does look like an .attach() approach might actually have 
> been a nicer version of this, but on the other hand it's currently 
> implementable solely with the Expressions API. It seems like the OuterRef 
> is very similar to your queryset.ref(). An even nicer approach using attach 
> could be to say qs.attach(q1=some_qs).filter(a=F('q1__b'))?
>

Hmmh, we have one form of SubQuery, but that's actually for SELECT clause, 
not for FROM clause. I believe the same class won't work for the CTE or 
subquery in FROM clause case.

As for the attach(), seems like a really nice syntax. We do need something 
for generating the join clause for the JOIN. If you look at an example:
q1 = Book.objects.values('author_id').annotate(avg_price=Avg('price'))
q2 = Author.objects.attach(q1=q1)
it needs to create something like:
WITH q1 AS (
SELECT author_id, avg(price) FROM book GROUP BY author_id
)
SELECT author.id, author.name
   FROM author
   LEFT JOIN q1 ON author.id = q1.author_id;

Or, equivalently without the CTE:

SELECT author.id, author.name
   FROM author
   LEFT JOIN ( SELECT author_id, avg(price) FROM book GROUP BY author_id) 
ON author.id = q1.author_id;

Now, the main points are:
   1. There is no need to design this to be about CTEs. That just limits 
the feature from backends that don't have CTEs without any real benefit. 
>From Django's perspective the two above queries are the same.
   2. We do need something for the JOIN ON condition. In some cases Django 
could guess this, but there needs to be an explicit way to express the join 
condition.

If we allow usage of expressions from the attached queryset, but don't try 
to go for cases where model instance are created from the attached 
queryset, this will be both possible to implement without having to write a 
change-everything patch, and this will also be a really nice feature.

For recursive CTEs, I'd leave that strictly as a later step. The only thing 
we need to check right now is that we don't do something that prevents a 
good recursive CTEs implementation later on.

 - Anssi

>
> Looking forwards to seeing a DEP!
>
> [0] 
> https://docs.djangoproject.com/en/1.11/ref/models/expressions/#subquery-expressions
>
> On 22 March 2017 at 01:32, Ashley Waite  > wrote:
>
>> Here's the code changes I've made, noting that some of them were to shove 
>> in a generalised VALUES clause that mocks being a queryset, so that it 
>> plays with the same interface.
>>
>>
>> https://github.com/django/django/compare/master...ashleywaite:cte-dev#files_bucket
>>
>> I've had a glance at cte-trees/cte-forest and once general CTEs are 
>> worked out expanding that to include recursive CTEs wouldn't be too 
>> difficult, and that would greatly simplify the implementation of cte-forest 
>> to the extent that it might be viable as a django data/reference type.
>>
>> - Ashley
>>
>>
>> On Saturday, March 18, 2017 at 8:28:53 PM UTC+11, Josh Smeaton wrote:
>>>
>>> Thanks for bringing this up Ashley, and for all of the detail you 
>>> provided. I'd certainly like to see CTEs make their way into Django, 
>>> provided we could come up with a nice enough API. From the look of it, 
>>> you've already got something that works with an okay API so I'm hopeful.
>>>
>>> I'd be very interested in seeing your POC too if you're able to share.
>>>
>>> From looking very briefly at django-cte-trees it doesn't aim to support 
>>> user defined CTEs for anything other than recursive queries. I'd be 
>>> interested in seeing, as part of a DEP, how CTE inclusion in django core 
>>> could support the cte-trees project from an API perspective.
>>>
>>> On Friday, 17 March 2017 22:28:17 UTC+11, Ashley Waite wrote:

 Hey all,


 I'd like to suggest adding Common Table Expression (CTE) query 
 generation as a feature to Django.

 I've been working on a project that required manipulation of many 
 records at once, and as with many ORMs found that this wasn't an ideal 
 use-case in Django. As the rest of our code base and related projects are 
 in Django, there was a strong preference to find a way to do it and keep 
 to 
 the same model-is-the-truth design.

 I first did this by writing some hackish functions using raw querysets 
 and generating my own CTE based queries, but it lacked ideal flexibility 
 and maintainability. So I've now written some modifications into my Django 
 to do this in a more Django-esque way and think that this functionality 
 would be beneficial within the project itself, but am unsure exactly where 
 to start the conversation about that.


 *Why generate CTE based queries from querysets?*

 By allowing querysets to be attached to each other, and setting 
 appropriate WHERE clauses, arbitrary and nested SQL que

Re: Adding signals to bulk update/create operations

[Anssi Kääriäinen]

On Friday, March 31, 2017 at 10:50:54 AM UTC+3, Todor Velichkov wrote:
>
> @Anssi, thank you for your response.
> I really haven't think about threat safety, but maybe its because in our 
> specific usage it's close to impossible to happen.
>
> What do you think about this:
> 1) Put everything into a transaction
> 2) Before update, count the queryset.
> 3) Fire the pre_update with the queryset, and the count.
> 4) Do the actual update -> get the updated rows
> 5) Assert the updated rows is equal to the queryset.count().
> 6) Revert the transaction if there is a difference.
>
> No pk fetching, this is left to the be implemented by the listener if he 
> needs it.
>

I'm afraid this won't work well enough. It does work for many use cases, 
but in cases where concurrency is an issue, you'd get some loud failures, 
and worse, you could get cases where things work silently wrong (one item 
removed, one added concurrently). These cases would be near-impossible to 
debug.

The reason why we don't have update signals is that it's hard to find a 
solution that allows users to run actions when an instance is updated, 
works correctly in concurrent cases and with large amount of objects and 
doesn't turn bulk update in to a loop of save() calls.

Maybe the best approach is to look for a solution where users can customise 
update behaviour for remote models if they so wish. Then you could write 
your own update signals solution, one that doesn't need to care about all 
of the above constraints.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fd81d210-a2d6-4e7a-9eb0-d3f8f078a032%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding signals to bulk update/create operations

[Anssi Kääriäinen]

The problem with passing the queryset is that it's possible that some 
object is added to or removed from the queryset between the pre_update and 
actual update execution. To avoid this the execution should go somewhere 
along the lines of:
   1) if there is pre_update or post_update do stages 2-5, if not, update 
as with current code
   2) fetch primary keys for models to be updated to a set, with 
.select_for_update() applied
   3) fire pre_update, give the primary key set as argument
   4) do the update against a queryset with .filter(pk__in=pk_set)
   5) fire post_update with primary key set as argument

This way the pre and post update signals will execute against a fixed set 
of instances. The bad part is that this can significantly slow down the 
.update() call, but to get actually safe signals, I don't see a way around 
that.

 - Anssi

On Friday, March 31, 2017 at 3:51:34 AM UTC+3, Tim Graham wrote:
>
> There's an accepted ticket about adding pre_update and post_update 
> signals: https://code.djangoproject.com/ticket/21461. From a quick 
> glance, I think this is what you're proposing.
>
> On Thursday, March 30, 2017 at 4:28:00 PM UTC-4, Todor Velichkov wrote:
>>
>> Consider the following piece of code:
>>
>> @receiver(pre_save, sender=MyModel)
>> def my_handler(sender, **kwargs):
>> instance = kwargs['instance']
>> if instance.verified:
>> do_something(instance)
>> else:
>> do_else(instance)
>>
>>
>>
>> Its good, because it keeps `MyModel` decoupled from `do_something`
>>
>> But there is one flaw. If we do:
>>
>> MyModel.objects.filter(verified=False).update(verified=True)
>>
>> we are screwed, `do_something` is not executed, and our models get 
>> out-of-sync.
>>
>> If we try to get smart and manually fire the pre_save signal for each 
>> instance, we are gonna have a hard time.
>> Its gonna be slow.
>> And its gonna be memory inefficient.
>>
>> We already experienced it in our app.
>>
>> So our new approach is like this:
>>
>> pre_bulk_update = Signal(providing_args=["queryset", "update_kwargs"
>> ])
>> post_bulk_update = Signal(providing_args=["update_kwargs",])
>>
>> pre_bulk_create = Signal(providing_args=["objs", "batch_size"])
>> post_bulk_create = Signal(providing_args=["objs", "batch_size"])
>>
>>
>> class MyModelQuerySet(models.QuerySet):
>> def update(self, **kwargs):
>> pre_bulk_update.send(sender=self.model, queryset=self, 
>> update_kwargs=kwargs)
>> res = super(MyModelQuerySet, self).update(**kwargs)
>> # The queryset will be altered after the update call
>> # so no reason to send it.
>> post_bulk_update.send(sender=self.model, update_kwargs=kwargs
>> )
>> return res
>>
>> def bulk_create(self, objs, batch_size=None):
>> pre_bulk_create.send(sender=self.model, objs=objs, batch_size
>> =batch_size)
>> res = super(MyModelQuerySet, self).bulk_create(objs, 
>> batch_size)
>> post_bulk_create.send(sender=self.model, objs=objs, 
>> batch_size=batch_size)
>> return res
>>
>>
>> class MyModel(models.Model):
>> #...
>> objects = MyModelQuerySet.as_manager()
>>
>>
>>
>> This gives us a nice interface to handle all kind of changes regarding 
>> `MyModel`
>> Our example usage looks like this:
>>
>>
>> @receiver(pre_save, sender=MyModel)
>> def my_handler(sender, **kwargs):
>> instance = kwargs['instance']
>> if instance.verified:
>> do_something(instance)
>> else:
>> do_else(instance)
>>
>>
>> @receiver(pre_bulk_update, sender=MyModel)
>> def my_bulk_update_handler(sender, **kwargs):
>> update_kwargs = kwargs['update_kwargs']
>> if 'verified' not in update_kwargs:
>> # no change im interested in
>> # no need to take any action
>> return
>>
>> queryset = kwargs['queryset']
>> pks_to_be_updated = queryset.values_list('pk', flat=True)
>> if update_kwargs['verified']:
>> do_something_bulk_update_implementation(pks_to_be_updated)
>> else:
>> bulk_update_do_else_implementation(pks_to_be_updated)
>>
>>
>> @receiver(pre_bulk_create, sender=MyModel)
>> def my_bulk_create_handler(sender, **kwargs):
>> objs = kwargs['objs']
>> group_1 = []
>> group_2 = []
>> for obj in objs:
>> if obj.verified:
>> group_1.append(obj)
>> else:
>> group_2.append(obj)
>>
>> if group_1:
>> do_something_bulk_create_implementation(group_1)
>> if group_2:
>> bulk_create_do_else_implementation(group_2)
>>
>>
>>
>> I think this turns out to be a very clean approach.
>> It help us use the most optimal strategy to handle the change.
>> So I'm sharing this with the community to check your feedback.
>> I believe if this gets into the Django Internals, it can be a very 
>> powerful tool.
>> It will lose power as a 3rd party app.
>>
>

-- 
You received this message 

Re: Deprecate is_superuser, is_staff and is_active

[Anssi Kääriäinen]

If you want to query for each user with given permission, then the query 
seems to be this (unfortunately not tested as I don't have a nice database 
to test this against):

users = User.objects.filter(
Q(is_superuser=True) |

Q(pk__in=Group.objects.filter(permissions=permission).values('user__pk')) |

Q(pk__in=Permission.objects.filter(id=permission.pk).values('user__pk')))

I can't see how removing the is_superuser attribute would make this query 
more efficient. And, it's likely the query you were using is slow because 
it doesn't use subqueries. It would be nice to make Django use subqueries 
automatically, but for queryset API reasons that's hard to do.

 - Anssi

On Tuesday, March 28, 2017 at 1:53:00 PM UTC+3, guettli wrote:
>
>
>
> Am Montag, 27. März 2017 16:11:06 UTC+2 schrieb Melvyn Sopacua:
>>
>> On Friday 24 March 2017 04:31:32 guettli wrote:
>>
>> > I know this is a crazy idea, and it will get the "won't fix" tag very
>>
>> > soon.
>>
>> > 
>>
>> > Nevertheless I want to speak it out.
>>
>> > 
>>
>> > My use case: Get a queryset of users who have a given permission.
>>
>>  
>>
>> I'm still thinking about this use case.
>>
>> Cause it's not for normal use - you don't normally use permissions as 
>> data, you use permissions to deny / allow access to data.
>>
>> So, you're already in the "specialized" corner.
>>
>>  
>>
>
>
> I can use more words, to make this use case more colorful.
>
> Imagine a approval workflow: Only some users are allowed to do the 
> approval.
>
> An invoice instance needs to be forwarded to someone with the matching 
> permissions.
>
> This way I want to provide a html form where the current user can choose 
> the user with the matching
> permissions.
>
>
>
>
>  
>
>> > At the moment this query is complex and results in performance issues,
>>
>>  
>>
>> For a small install this works fine. When performance becomes an issue 
>> because there's a large number of users and groups, my first instinct is to 
>> regroup users so that no permission exceptions exist anymore at the user 
>> level.
>>
>> I'd add all superusers to the "wheel" group ("root", "superuser", 
>> whatever floats your boat).
>>
>> Now the query to get this info no longer requires the user model for 
>> conditions and you can simply query the group manager.
>>
>> It is what groups are for and it'll be always faster get all the matching 
>> groups as one set and combine it with all the users that have the 
>> permission assigned directly.
>>
>> The "one query" requirement conflicts with your performance requirement.
>>
>>
>
> Unfortunately the current sql query has several outer joins and takes 
> several seconds even with only very few data in the database.
>
> We have a work-around this. The basic part is published here: 
> https://code.djangoproject.com/ticket/27260#comment:10
>
>
>
>  
>
>>  
>>
>> -- 
>>
>> Melvyn Sopacua
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f38eb15f-4e5f-45b8-8fa9-be4e749bc86c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite key development

[Anssi Kääriäinen]

Just my late 2 cents to this...

First, I wouldn't put too much weight on the DEP proposals. Looking back to 
them now, I don't think they went to the right direction.

For Michal Petrucha's work, it was really close to being merged some years 
ago. The problem was that migrations happened at the same time, and that 
resulted in need of substantial changes to Michal's work, and unfortunately 
this never happened. So, I'd put a lot of weight on this work.

If I'd were to take on this task, I'd try to develop the feature with the 
idea of merging the work in multiple batches. For example this might be a 
possible plan:
  1. Introduce virtual/composite field, only CREATE TABLE migrations 
support at this point, not documented as public API
  2. Add support for primary key and unique composite fields, again 
non-public
  3. Work on foreign keys, again non-public
  4. Migrations support
  5. Make the work public API (that is, document the work)

This would give a chance to get the work done in incremental pieces. Doing 
this all in one go is such a large task that there is a big risk of not 
getting this done at all.

Whatever approach you take, I really hope you'll be able to work on this 
and get this long waited feature in to Django.

 - Anssi

On Wednesday, March 1, 2017 at 7:08:35 AM UTC+2, Asif Saifuddin wrote:
>
> Hi Julien,
>
> I will publish it very soon.
>
> Thanks
>
> On Wednesday, March 1, 2017 at 5:58:50 AM UTC+6, Julien Phalip wrote:
>>
>> Hi Asif,
>>
>> That sounds good. On first look I did have some reservations about some 
>> of the design details in the current DEP, especially around the observer 
>> pattern for data binding. But I’m going to have to dig deeper into the 
>> implementation to get a clearer idea.
>>
>> It’d be great if you could publish your work-in-progress at some point so 
>> we can discuss the approach in more detail.
>>
>> Thanks,
>>
>> Julien
>>
>> On Mon, Feb 27, 2017 at 9:03 PM, Asif Saifuddin  wrote:
>>
>>> Hi Julian,
>>>
>>> I have been also reviewing and studying the previous works, deps, 
>>> discussions, codes and tickets. I have also been working to prepare a new 
>>> dep based on the previous works.
>>>
>>> Like what Michal said, from my observation, I found the works and 
>>> approaches of Michal is quite better and it's possible to break the work 
>>> down into smaller parts to implement gradually.
>>>
>>> I am not quite sure how much work or which approaches of  Thomas 
>>> Stephenson in 2015 could be mixed or needed with Michal's approach. In my 
>>> humble opinion Michal spent more time in working on this and I also found 
>>> his insights and suggestions on this topic more sensible.
>>>
>>> I would also like to work on this. My Dep is not yet ready for a push.
>>>
>>> You could certainly review and give you input on the Dep. Some core 
>>> members suggested me to finalize a Dep before div into code.
>>>
>>> Let me know what your thoughts.
>>>
>>> Regards,
>>>
>>> Asif
>>>
>>> I have also contact with 
>>>
>>> On Tuesday, November 29, 2016 at 6:10:38 PM UTC+6, Craig Kerstiens wrote:

 Hi all,

 My company (Citus Data) is interested in sponsoring some Django work. 
 In particular work on support for composite primary keys. From what I 
 understand this wouldn't be the first time the work has been explored and 
 it sounds like it has a number of intricacies to it (
 https://github.com/django/deps/blob/master/draft/0191-composite-fields.rst 
 and 
 https://github.com/django/deps/blob/master/draft/0192-standalone-composite-fields.rst).
  
 Our hope here is that it would be done in line with something that could 
 eventually become part of an official Django release vs. a one-off work 
 around. 

 While we know there's no guarantee of it being accepted, we'd love to 
 find someone with some knowledge of all the existing areas that would have 
 to be touched and has some experience contributing to Django core to 
 improve that likelihood. As part of the work, we would want the two 
 existing DEPs to be completed and work towards getting those accepted by 
 the broader community. And hopefully it goes without saying, but we'd 
 fully 
 expect all the work to done in the open similar to other Django 
 development. 

 If you're interested in doing this work please reach out as we'd love 
 to discuss further. And if we have enough interest we'll be doing a full 
 CFP for the work to try to keep the process as fair as possible. 

 ~ Craig

 -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-de

Re: Adding generated common table expressions

[Anssi Kääriäinen]

+1 for the INSERT and UPDATE parts.

For the .attach() method, I wonder if it would be better to work on having 
generic subquery support in the ORM instead of targeting CTEs specifically. 
>From Django's perspective there isn't any big difference between:
WITH fooquery AS (SELECT * FROM foo WHERE ...)
SELECT * FROM baz JOIN fooquery ON ...
and
SELECT * FROM baz JOIN (SELECT * FROM foo WHERE ...) fooquery ON ...

The great thing about the subquery form is that it works on every database, 
and to my knowledge the subquery support is surprisingly standard on all 
database Django needs to care about.

If we want full support for either of the above cases, then we need some 
way to tell to the Django's ORM what the fooquery (either as subquery or 
with CTE) produces. For simple cases this won't be complex (say, for 
.values() it's just a list of expressions), but what to do when the 
subquery/CTE contains .select_related() for example? Then we'd have a 
situation where the subquery produces a list of expressions but those 
expressions might refer to different models in the CTE.

So, the starting point should be to allow only "simple" queries in 
.attach(). This means that the attached query must be either .values() 
queryset, or a queryset containing expressions from single model only (no 
.extra(), .annotate() or select_related() added). In addition I think 
.attach() should support subqueries instead of using only CTEs. We get 
support on every backend instead of PostgreSQL only for pretty much the 
same amount of effort.

 - Anssi

On Saturday, March 18, 2017 at 11:28:53 AM UTC+2, Josh Smeaton wrote:
>
> Thanks for bringing this up Ashley, and for all of the detail you 
> provided. I'd certainly like to see CTEs make their way into Django, 
> provided we could come up with a nice enough API. From the look of it, 
> you've already got something that works with an okay API so I'm hopeful.
>
> I'd be very interested in seeing your POC too if you're able to share.
>
> From looking very briefly at django-cte-trees it doesn't aim to support 
> user defined CTEs for anything other than recursive queries. I'd be 
> interested in seeing, as part of a DEP, how CTE inclusion in django core 
> could support the cte-trees project from an API perspective.
>
> On Friday, 17 March 2017 22:28:17 UTC+11, Ashley Waite wrote:
>>
>> Hey all,
>>
>>
>> I'd like to suggest adding Common Table Expression (CTE) query generation 
>> as a feature to Django.
>>
>> I've been working on a project that required manipulation of many records 
>> at once, and as with many ORMs found that this wasn't an ideal use-case in 
>> Django. As the rest of our code base and related projects are in Django, 
>> there was a strong preference to find a way to do it and keep to the same 
>> model-is-the-truth design.
>>
>> I first did this by writing some hackish functions using raw querysets 
>> and generating my own CTE based queries, but it lacked ideal flexibility 
>> and maintainability. So I've now written some modifications into my Django 
>> to do this in a more Django-esque way and think that this functionality 
>> would be beneficial within the project itself, but am unsure exactly where 
>> to start the conversation about that.
>>
>>
>> *Why generate CTE based queries from querysets?*
>>
>> By allowing querysets to be attached to each other, and setting 
>> appropriate WHERE clauses, arbitrary and nested SQL queries can be 
>> generated. Where the results of the queries are only necessary for the 
>> execution of following queries this saves a very substantial amount of time 
>> and database work. Once these features exist, other functionality can also 
>> transparently use these to generate more efficient queries (such as large 
>> IN clauses).
>>
>> This allows several powerful use cases I think Django would benefit from:
>>
>>
>> *Large 'IN' clauses*, can be implemented as CTEs reducing expensive 
>> lookups to a single CTE INNER JOIN. For sets of thousands to match from 
>> tables of millions of records this can be a very substantial gain.
>>
>>
>> *Composite 'IN' conditions,* where multiple fields must match and you're 
>> matching against a large set of condition rows. In my usage this was "where 
>> the md5/sha hashes match one of the million md5/sha tuples in my match 
>> set". This is simply a CTE JOIN with two clauses in the WHERE.
>>
>>
>> *Nested data creation*, where the parent doesn't yet exist. Django 
>> doesn't currently do this as the primary keys are needed, and this makes 
>> normalised data structures unappealing. Using INSERTs as CTEs that supply 
>> those keys to following statements means that entire nested data structures 
>> of new information can be recreated in the database at once, efficiently 
>> and atomically.
>>
>>
>> *Non-uniform UPDATE*s, such that a modified set of objects can all be 
>> updated with different data at the same time by utilising a CTE values 
>> statement JOINed to the UPDATE statem

Re: changing the default MySQL isolation level to READ COMMITTED

[Anssi Kääriäinen]

For reference there are some discussions about this 
in https://code.djangoproject.com/ticket/13906. The short version of the 
discussions is that MySQL is very interesting when it comes to transaction 
isolation levels and things like SELECT FOR UPDATE...

I think it would be ok to allow changing the isolation level, set the 
default to read committed, and document that if you use some other 
isolation level you are on your own. We could of course allow using 
different isolation levels on other databases, too. PostgreSQL's true 
serializable transactions for example are extremely nice for some 
application domains.

  -Anssi

On Friday, January 13, 2017 at 5:44:31 PM UTC+2, Tim Graham wrote:
>
> In the end, I don't use MySQL but if a similar change were made for 
> PostgreSQL, I would find the current approach more annoying (bothering me 
> about changing defaults over two release cycles) than cautious.
>
> I'm also uncertain that duplicating a deprecation warning in a system 
> check is a good precedent to set. Adam, is the current implementation what 
> you envisioned when you made your comment and what do you see as the 
> advantages of that?
>
> Aymeric, I assume "I’d love if this fix made it into 1.10" was a typo for 
> 1.11. Anyway, the "Allowed transaction isolation level to be chosen on 
> MySQL" commit isn't controversial. I'll try to get at least that in.
>
> On Friday, January 13, 2017 at 10:11:26 AM UTC-5, Adam Johnson wrote:
>>
>>  aside from some very performance-sensitive websites that already worked 
>>> around Django’s bugs
>>
>>
>> Tbf it's true I already added READ-COMMITTED at work
>>
>>  I’d love if this fix made it into 1.10
>>
>>
>>
>>  
>>
>> On 13 January 2017 at 15:05, Aymeric Augustin <
>> aymeric.au...@polytechnique.org> wrote:
>>
>>> Hello,
>>>
>>> I think there’s been little feedback because this issue is very, very 
>>> far from the concerns of most Django users — aside from some very 
>>> performance-sensitive websites that already worked around Django’s bugs and 
>>> will add the configuration line needed to keep whatever the isolation level 
>>> they chose.
>>>
>>> If I was trying to merge that change, I think I’d just document the 
>>> backwards incompatibility and call it a day. But as everyone knows, I’m 
>>> taking backwards compatibility less seriously than most.
>>>
>>> Given the uncertainty around the consequences of this change, I don’t 
>>> think Shai’s approach is out of place, even though it creates an overhead 
>>> for every Django project using MySQL.
>>>
>>> Since Shai is leading the effort and considering the general lack of 
>>> feedback on this issue, I think it’s fair to let him make the final call 
>>> and keep the deprecation path if he thinks that’s safer.
>>>
>>> Regardless, I’d love if this fix made it into 1.10, let’s not delay it 
>>> because we’re worried of being too cautious :-)
>>>
>>> -- 
>>> Aymeric.
>>>
>>>
>>> On 13 Jan 2017, at 14:52, Tim Graham  wrote:
>>>
>>> I guess three days is too little time to get a consensus on this. At 
>>> this point I'm thinking to defer this from 1.11.
>>>
>>> On Wednesday, January 11, 2017 at 9:50:27 AM UTC-5, Patryk Zawadzki 
>>> wrote:

 To add some context, REPEATABLE READ can break get_or_create among 
 other things (not sure how many invalid tickets Django gets related to 
 that).

>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/42212944-d7e7-4c33-8c34-8e235a7d515a%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/2D7B4E2A-30EF-4201-9272-C5430825E123%40polytechnique.org
>>>  
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout

Re: Django ORM query syntax enhancement

[Anssi Kääriäinen]

+1 from me, too.

I still think we should be able to get into a point where you can run 
queries like:
Q.user.firstname.collate('fi').lower() == 'anssi'

So, not only can you call regular transforms (like lower), but also 
transforms that take arguments. Other use cases are for example substring:
Q.user.firstname.substring(from=5) == 'akaariai'

And of course, this would be super useful for things like drilling into 
JSON data.

 - Anssi

On Friday, October 7, 2016 at 6:20:29 PM UTC+3, Robert Roskam wrote:
>
> +1 from me. I really like this approach to help making my queries more DRY.
>
> I also like how lightweight the library is altogether. Well done!
>
>
> For those looking for a direct link, here you go: 
> https://github.com/Nepherhotep/django-orm-sugar
>
>
>
> On Thursday, October 6, 2016 at 1:04:56 PM UTC-4, Alexey Zankevich wrote:
>>
>> Hey all,
>>
>> Just want to announce recent changes in Django ORM Sugar library, which 
>> might be useful in future Django query syntax enhancement (if ever happens).
>> 1. Now it supports indexes and slices:
>>
>> >>> Q.data.owner.other_pets[0].name='Fishy'
>> Q(data__owner__other_pets__0__name='Fishy')
>> >>> Q.tags[0] == 'thoughts'
>> Q(tags__0='thoughts')
>> >>> Q.tags[0:2].contains(['thoughts']) 
>> Q(tags__0_2__contains=['thoughts'])
>>
>>
>> 2. Previously, it was only possible to call defined method (like it is 
>> done for *is_not_null()* function) or registered with decorator. Now if 
>> you try to call unknown function of sugar Q, it will internally pass 
>> function name and append it as *__functionname *to lookup field:
>>
>> >>> Q.user.username.contains('Rodriguez')
>> Q(user__username__contains='Rodriguez')
>>
>>
>> There is no such function "contains" anymore in sugar Q, however, it just 
>> adds *__contains* to lookup field and passes parameter to it.
>>
>> 3. It will pass a tuple to lookup if multiple arguments passed:
>>
>> >>> Q.user.create_datetime.range(d1, d2)
>> Q(user__create_datetime__range=(d1, d2))
>>
>>
>> I think the library is at the final state now and isn't going to get new 
>> substantial features, but responses are highly appreciated.
>>
>> Regards,
>> Alexey
>>
>>
>> On Sunday, August 16, 2015 at 4:18:26 PM UTC+3, Alexey Zankevich wrote:
>>>
>>> Hi all,
>>>
>>> This topic is related to the current ORM query syntax with underscores.
>>> There are lots of arguing related to it, anyway it has pros and cons.
>>>
>>> Let's take a concrete example of querying a model:
>>>
>>> >>> 
>>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>>>
>>>
>>> Pros:
>>>
>>> 1. The syntax is easy to understand
>>> 2. Can be extended with custom transforms and lookups
>>>
>>> However, there are several cons:
>>>
>>> 1. Long strings is hard to read, especially if we have fields with 
>>> underscores.
>>> It's really easy to make a mistake by missing one:
>>>
>>> >>> 
>>> GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)
>>>
>>> Not easy to catch missing underscore between user and profile, is it? 
>>> Even
>>> though, it's not easy to say whether it should be "user_profile" 
>>> attribute or
>>> user.profile foreign key.
>>>
>>> 2. Query strings can't be reused, thus the approach violates DRY 
>>> principle.
>>> For example, we need to order results by last_login_date:
>>>
>>> >>> 
>>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) \
>>> .order_by('user__profile__last_login_date')
>>>
>>> We can't keep user__profile_login_date as a variable as in the first 
>>> part of the
>>> expression we use a keyword argument, meanwhile in the second part - 
>>> just a 
>>> string. And thus we just have to type query path twice.
>>>
>>> 3. Lookup names not natural to Python language and require to be 
>>> remembered or
>>> looked up in documentation. For example, "__gte" or "__lte" lookups tend 
>>> to be
>>> confused with "ge" and "le" due to similarity to methods "__ge__" and 
>>> "__le__".
>>>
>>> 4. Lookup keywords limited to a single argument only, very inconvenient 
>>> when
>>> necessary to filter objects by range.
>>>
>>> I was thinking a lot trying to solve those issues, keeping in mind Django
>>> approaches. Finally I came up with solution to extend Q objects with dot
>>> expression syntax:
>>>
>>> >>> GameSession.objecs.filter(Q.user.profile.last_login_date >= 
>>> yesterday)
>>>
>>> Q is a factory instance for old-style Q objects. Accessing attribute by 
>>> dot
>>> returns a child factory, calling factory will instantiate old-style Q 
>>> object.
>>>
>>> >>> Q
>>> 
>>>
>>> >>> Q.user.profile
>>> 
>>>
>>> >>> Q(user__name='Bob')
>>> 
>>>
>>> It overrides operators, so comparing factory with value returns a 
>>> related Q
>>> object:
>>>
>>> >>> Q.user.name == 'Bob'
>>> 
>>>
>>> Factory has several helper functions for lookups which aren't related to 
>>> any
>>> Python operators directly:
>>>
>>> >>> Q.user.name.icontains('Bob')
>>> 
>>>
>>> And helper to get query path as s

Re: Django Integration

[Anssi Kääriäinen]

On Tue, May 10, 2016 at 4:54 AM, Andrew Godwin  wrote:
> My concern around using the Mozilla money to refine it was that it was
> applied for under the pretense this would be core Django, though if that's
> still our intention and we keep it external for now I don't see too much of
> a problem arising there.

I don't see a problem here - the main opposition seems to be about
having the code in Django 1.10. This puts a lot of pressure on Tim,
and there isn't enough time to field-test the design. It also feels
wrong that we approach many feature additions with saying that the
design should be proven in a 3rd party app, but there hasn't yet been
enough time to prove the channels design.

Tim's feeling about this is very important to me. He is doing an
awesome job reviewing and committing patches and ensuring releases
happen on time. In my opinion he should have control of what happens
near feature freeze. If he doesn't have that control, it will make it
impossible for him to ensure we stay on release schedule.

I'm going to try to avoid writing more about this. My opinion is
hopefully clear enough already. If the decision is to ship this in
1.10, then the available time should be used to make this feature as
good as possible, not on this discussion.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GRgBGwhuEqhBf%3D%3DYyJr%2BF51dVo4CqerGPBNLfwu02hZA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making Django more PaaS-friendly

[Anssi Kääriäinen]

On Tue, May 10, 2016 at 1:48 AM, Carl Meyer  wrote:
> Hi Anssi,
>
> On 05/09/2016 06:53 AM, Anssi Kääriäinen wrote:
> I'm curious to hear more about this - could you give some example DEPs
> where this was a problem? In the case of the two DEPs I was most closely
> involved with (multiple template engines and rethinking middleware), I
> don't think this is an accurate description of what happened. Both those
> DEPs were written relatively early in the design process and evolved as
> we learned from discussion and implementation.
> They were done quite similarly to how the PEPs I've worked on were done.
>
> I'm also curious about your use of the word "enforced." I don't really
> see the DEP process as something that needs "enforcement" (and I'm
> curious where you see it as having been lacking in "enforcement"). IMO
> the only relevant "enforcement," really, is that the implementation of a
> DEP which is rejected by the technical board (has never happened yet)
> should not be merged.

It's not so much the DEPs that have been written (though the channels
DEP is going to be post-design instead of supporting the design). It's
more about those features that don't have a DEP at all. For example
database schemas and subquery expressions come to mind (these are
ongoing work, PRs available at GitHub). There are various design
choices and the features are somewhat significant, yet the design
hasn't been at all DEP driven.

Also, having DEP requested for channels only at this point, and Andrew
(a technical board member) assuming a DEP wasn't required should point
out there might be a problem. This is not to criticize Andrew but the
way DEPs are sometimes required, sometimes not.

I'm not seeing a problem with the DEP idea in general, with any
particular DEP or with any single feature having or not having a DEP.
I'm criticizing the approach where sometimes patches are called out
randomly for DEP, often at a point where the DEP is just documenting
what has been already decided, not supporting the decision process
itself.

I probably picked the wrong thread to complain about this - if we are
going to want DEPs for features similar to the ideas discussed in this
thread, then the timing for requesting a DEP was accurate.

  - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1E9axuBW5y673LSAE3-2%2BUg45JcPL%2BNLQJVsUrjQ3aLzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making Django more PaaS-friendly

[Anssi Kääriäinen]

I'm very much afraid we are in a situation where we have a documented
DEP process which isn't actually enforced, nor does it document what
we actually do.

Python's PEP process is useful as the document evolves while the
design evolves. What we do with DEPs is different, we first design
something, then we document the design decision, often mostly because
DEPs are a required document. We don't get what we could from DEPs as
they are written too late.

Could we try an approach where we *first* require a DEP, then have the
design discussion?

 - Anssi

On Mon, May 9, 2016 at 2:42 PM, Shai Berger  wrote:
> On Monday 09 May 2016 05:06:47 James Bennett wrote:
>> Whee, this thread got big!
>>
>> The takeaway I'm getting here is that we should be careful about what we
>> adopt into core and when; I'm going to take a couple days and write up some
>> notes on what I think a good conservative approach would look like, and ask
>> for feedback on that.
>>
>
> Looking at what we just had in other threads, this is probably DEP-worthy;
> even if the results end up being a relatively small modification, it would be
> helpful to record why we decided to reject the other suggestions.
>
> Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1HrKje7%3DQLi-51sw4ZRSY3jFE1VTu_BKsL1y4u2EiVRJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Integration

[Anssi Kääriäinen]

On Thursday, May 5, 2016, Andrew Godwin  wrote:
>
> Do you have a link to the presentation about them removing it?
>

https://youtu.be/839rskyJaro around 34 minutes and onwards, another version
is https://youtu.be/2dG5HeM7MvA at 24 minutes.

They were tackling a bit different problem, so their lessons might not
apply. Most importantly they weren't aiming for websockets at all, just for
high availability and throughput for normal HTTP traffic. On the other
hand, the architecture of broxy itself is pretty much the same as Daphne,
and they felt they had problems with it.

The way we have approached recent feature additions is that we let them
prove themselves outside of core. I think the crucial question is why
Channels can't do this, that is why can't we wait some time and let actual
production deployments prove the design?

I know South is an example of why some features have a hard time living
outside core. But DRF is an example that a feature crucial to modern web
development can work very well outside of core.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1FDXaa2kMAo69NO7ji-Cpd4sdTwDWgaysx9NRGH%2BGH%2BSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Integration

[Anssi Kääriäinen]

On Thursday, May 5, 2016, Russell Keith-Magee 
wrote:

> I will admin that I haven’t been paying *close* attention to Andrew’s work
> - I’m aware of the broad strokes, and I’ve skimmed some of the design
> discussions, but I haven’t been keeping close tabs on things. From that
> (admittedly weak) position, the only counterargument that I’ve heard are
> performance concerns.
>

I haven't brought this up before, but security is something we should
discuss pre-merge.

What I'm mainly worried about is malicious clients intentionally trying to
choke the channels layer. I guess the approaches for DoS attack would fall
under these categories:
  1. Try to generate large responses and read those response slowly.
  2. Fire a large request, don't read the response.
  3. Try to cause exceptions in various parts of the stack - if the worker
never writes to the response channel, what will happen?

There are always DoS vectors, but checking there aren't easy ones should be
done. The main concern with channels is potential resource leak.

I found accidentally some presentations that seem relevant to thus
discussion. I recently watched some presentations about high availability
at Braintree. There are two presentations available, one from 2013 by Paul
Gross, where he explains their approach to HA, and one from 2015 by Lionel
Barrow, explaining what changed. Both are very interesting and highly
recommended.

The 2013 presentation introduces one key piece to HA at braintree, called
Broxy. Broxy basically serves HTTP the same way as Daphne - write requests
to tedis, and wait for response again through Redis.

The 2015 representation explains what changed. They removed Broxy because
it turned out to be conceptually complex and fragile. It might be their
implementation. But there is certain level of both complexity and possible
fragility about the design. On the other hand their story pretty much
verifies that the design does scale.

All in all I'd feel a lot more confident if there were large production
sites using the channels system, so that we shouldn't theorize on the
excellence of the approach. Are there already production deployments out
there?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1FoOSiGr0w_3JP8UYE-_owqOnS59d%2BOjp8eTWt4Qh48SQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: unique_together does not work as expected with nullable fields

[Anssi Kääriäinen]

If you really, really want an unique index that allows just a single
value, you might want to try unique index on (a, b, c) where c is not
null, and another unique index on (a, b) where c is null. That might
give the results you are looking for, though I haven't tested this.

SQL's nulls are weird, but in this case the feature is actually very
useful, especially for single column indexes. Unfortunately this
causes an abstraction leak, but we'll just have to live with that.

 - Anssi

On Fri, Apr 29, 2016 at 11:04 AM, Aymeric Augustin
 wrote:
> Hello,
>
> In SQL, defining a unique index on a nullable column will only enforce
> unicity of non-null values. This behavior seems more useful than allowing
> exactly one null value.
>
> Your example adds two more columns to the index. Other than that, it’s the
> exact same situation.
>
> In my opinion, the conclusion here is “users of Django still need to know a
> little bit about SQL”. I don’t think this warrants making code changes. We
> could add something to the documentation.
>
> --
> Aymeric.
>
> On 28 Apr 2016, at 20:34, Rich Rauenzahn  wrote:
>
>
> I just got bitten by this today, finding a duplicate row where I didn't
> expect one.  I haven't been able to find an existing Django bug.
>
> It's a popular topic on stack overflow:
>
> http://stackoverflow.com/questions/17510261/django-unique-together-constraint-failure
> http://dba.stackexchange.com/questions/9759/postgresql-multi-column-unique-constraint-and-null-values
>
>
> This is apparently an expected (and standardized) thing in SQL that ('A',
> 'B', NULL) is unique to ('A', 'B', NULL) as NULL is never equal to another
> NULL.
>
> There is a workaround at the SQL level of ...
>
> CREATE UNIQUE INDEX ab_c_null_idx ON my_table (id_A, id_B) WHERE id_C IS
> NULL;
>
>
> I'm wondering if this ought to at least be addressed in a runtime warning,
> or at least documentation in unique_together -- and I'm hoping that perhaps
> a Django level workaround could be devised to explicitly ask for unique
> indexes accommodating null values.
>
> For myself, I'm writing a unittest to fail if any of my unique_together's
> have a nullable field and using a specific value as my "null" value for now.
>
> Thoughts?  Has this come up before?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/566d247e-4aae-429e-9cc3-2544c82ce9a3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/92297BB3-F01E-42C7-8F18-8FCB32F927EC%40polytechnique.org.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1FEZMY%3Dj-o0YZary-LPpeifaYAqaQJjSb1apo1R8E9oVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Change executed raw mysql queries to return timezone aware datetimes.

[Anssi Kääriäinen]

I'm mildly in favor of b), seems like most projects would want that anyways.

If there is a really easy way for picking tz conversions per cursor,
then that could be a good approach, too. Something like cursor =
connection.cursor(convert_tz=True) might work. I know then MySQL would
have a bit different API than the rest of the backends for cursor
creation, but the SQL executed is going to be backend specific in any
case...

 - Anssi

On Fri, Apr 29, 2016 at 10:16 AM, Aymeric Augustin
 wrote:
> Hello Bryce,
>
> Oops! I didn’t realize that conversions were a parameter of the database
> connection on MySQL :-( So, yes, it is technically possible to implement the
> solution you’re describing.
>
> However I’m not positive that we should make this change. There’s an
> argument to be made that database settings should only affect the ORM and
> other Django components.
>
> `from django.db import connection; connection.cursor()` is supposed to give
> you a database cursor using the credentials from your settings file, but
> apart from that, it’s just a regular database cursor.  Django’s wrapper is
> expected to be transparent. I believe it’s important, as much as is
> practical, to preserve the ability to interact with the database without
> interference from Django.
>
> Also reverting backwards-incompatible changes in the following release is
> painful for users… Even if the first backwards-incompatible change is a sunk
> cost at this point, reverting it would be particularly vexing for affected
> users who updated their code.
>
> To sum up, the question we’re reaching is:
>
> “When USE_TZ =True, should cursor.execute() with a Django cursor return (a)
> naive datetimes like a plain MySQLdb cursor (b) aware datetimes in UTC like
> the Django ORM?"
>
> Who has an opinion on this question?
>
> Best regards,
>
> --
> Aymeric.
>
> On 29 Apr 2016, at 00:48, Bryce Drennan  wrote:
>
> I'm wondering if we're using the term "global" slightly differently.
>
> Yes, prior to the changes removing converters, Django had these converters
> globally applied to all mysql connections that it created. However, they
> didn't apply to mysql connections created outside of django, nor did django
> have to implement the converters globally like it did. Timezone support
> could be implemented at the connection level instead of globally being
> applied to all mysql connections.
>
> I have made this proof of concept to illustrate two distinct mysql
> connections, one which adds the timezone conversion, and one which does not:
> https://gist.github.com/brycedrennan/fc5f1cb08afe6dba034e3185368a7e2a/revisions
>
> It seems reasonable to me that connections created by Django would have
> Django specific functionality (like returning timezone aware datetimes).  If
> a specific DB should not have this functionality then we could setup the
> converter to not be applied to those connections.  If specific non-django
> code requires naive datetimes then it can use a non-django db connection.
> Does that make sense? I am not super familiar with the django codebase so I
> worry I'm missing something.
>
>
> On Monday, April 25, 2016 at 2:40:42 PM UTC-7, Aymeric Augustin wrote:
>>
>> Hi Bryce,
>>
>> #19738 is unrelated to the problem you’re describing. It was fixed in
>> d9521f66 which removed adapters i.e. transformations of parameters passed to
>> the database.
>>
>> You’re talking about the removal of converters i.e. transformations of
>> values returned by the database. It is discussed in #23820 and was committed
>> in ec186572.
>>
>> When I implemented time zone support in Django 1.4, conversion of values
>> could happen in several places. This was improved in Django 1.8 with
>> e9103402.
>>
>> This unified approach to conversions allowed to expand the functionality
>> of the ORM which now tracks the type of returned values, even computed ones.
>>
>> That said, this design cannot work well for raw queries because Django
>> doesn’t know the type of returned values, so it cannot apply a suitable
>> converter. You get whatever the DB-API module gives you. In a sense, this
>> behavior is sensible: Django shouldn’t interfere with values provided by raw
>> queries.
>>
>> This is where I introduced the backwards incompatibility you’re talking
>> about:
>>
>> https://github.com/django/django/commit/ec186572e6cfde4cd4bc1491ff552c5d32211d9f#diff-dbe1511db54b8fff2cb72a199cf37b4cL90
>>
>> (By the way, this is where MySQL converters used to be registered
>> globally. I’m not sure how you came to the conclusion that they were
>> registered per connection.)
>>
>> The thread on this mailing list you linked to describes clearly how I
>> ended up there:
>> https://groups.google.com/d/msg/django-developers/Ci_cs0KTagM/DchJIB8-vRsJ
>>
>> I acknowledged the backwards incompatibility in the releases notes:
>>
>> https://github.com/django/django/commit/ec186572e6cfde4cd4bc1491ff552c5d32211d9f#diff-c0f4f3a0c26f12ebc2e41904af710f4dR421
>>
>> I wrote in a later commit that val

Re: typeshed stubs for mypy

[Anssi Kääriäinen]

I think we can use type annotations in comments, and I do think this
could be useful in some cases as a hint to autocomplete systems in
IDEs.

What I'm afraid is that if we add annotations somewhere, we'll have
really hard time determining when to stop when requests for more type
hints arrive. So, unless we have easy to follow rules for when to
reject type hints, I'm with Florian - too much work for little gain.
(Or in other words, it is really easy to follow a rule where we reject
type hints always).

I've tested mypy a little, and while it might be really useful for
larger Django projetcts, it isn't there yet.

 - Anssi

On Thu, Apr 21, 2016 at 12:26 PM, Florian Apolloner
 wrote:
> stub files would do it too, but all in all it seems much work for not much
> gain.
>
> On Thursday, April 21, 2016 at 2:20:44 AM UTC+2, Tim Graham wrote:
>>
>> My understanding is that adding type annotations must wait until we drop
>> support for Python 2 which will happen after we cut the stable/1.11.x branch
>> in January 2017.
>>
>> On Wednesday, April 20, 2016 at 7:28:25 PM UTC-4, Pradip Caulagi wrote:
>>>
>>> Hello,
>>>
>>> Is there any plans to add type annotations to Django itself?  Or to
>>> provide typeshed stubs so that we can run mypy on django projects?
>>>
>>> Thanks,
>>> Pradip
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/a4dc54e9-52a4-46ce-86a7-1b8eccfac10a%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1FyyG1PCOMwyNeO%3Dn__MaGLRWXm1CHxto8_ZZfR042Z_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add documentation to address OWASP Top 10?

[Anssi Kääriäinen]

It is notable that if the number of items is a secret (say, you don't
want to reveal how many sales items you have), just having information
about sequential numbers is bad. In that case you should use UUID,
which the documentation could point out.

On Wed, Apr 6, 2016 at 4:35 AM, Josh Smeaton  wrote:
> I like the idea of addressing the OWASP top 10. Further, I think the advice
> of obscuring keys is wrong. The problem is actually addressed in the OWASP
> Top 10[0]
>
> 4 Insecure Direct Object References:
> A direct object reference occurs when a developer exposes a reference to an
> internal implementation object, such as a file, directory, or database key.
> Without an access control check or other protection, attackers can
> manipulate these references to access unauthorized data.
>
> The proper solution is *Access Controls* like the Permissions system
> provides. If you're going to rely on obscurity (one time file downloads for
> example), then you want to do so with a UUID or some kind of cryptohash (I
> haven't verified the particulars, don't take my this comment as security
> advice). That's not appropriate for something like a comments system.
>
> [0] https://www.owasp.org/index.php/Top10#OWASP_Top_10_for_2013
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/0ca48548-6c6f-4d80-b166-ff49487ff3cb%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1FWzGhDmmDvJO4bFajMR0fyubjQoP4-%3DgPQwBn7nSpMpw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Vendoring multipledispatch

[Anssi Kääriäinen]

+1 to allowing dependencies.

For multipledispatch, I'd like to first verify the following items:
  1. Does the author consider the API for multipledispatch stable?
  2. Are the rules in multipledispatch what we want? If we vendor it,
we can easily change the matching rules.

On Tue, Apr 5, 2016 at 2:01 AM, Florian Apolloner  wrote:
>
>
> On Monday, April 4, 2016 at 3:58:09 PM UTC+2, Donald Stufft wrote:
>>
>> Without looking at this specific thing too closely, maybe it’s time for
>> Django to gain a required dependency instead of bundling or reinventing
>> everything?
>
>
> +I_do_not_know_how_much
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9470fda1-67ab-4d92-b819-597d364fb460%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1H0LBvO4yKu35H67VK-LaSnizpF2h1xDYyGptf88W8Bdg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: bulk_create for multi table inheritance support

[Anssi Kääriäinen]

For MySQL InnoDB tables the behavior is described here:
http://dev.mysql.com/doc/refman/5.1/en/innodb-auto-increment-handling.html,
see "Gaps in auto-increment values for “bulk inserts”" item. So, you
are safe as long as the lock mode is correct, and assuming the lock
mode is the default is OK for Django.

For SQLite there can't be concurrent inserts (tables are locked
exclusively when writing), so you can just check the max id from the
table after insert and calculate the values from that.

There are still all the 3rd party databases (mssql for example), so
even if all core backends can support the multi-table bulk create,
you'll need to take in account that this isn't necessarily the case
for all possible backends.

Personally I think we should just rename the method to fast_create()
and allow it to fall back to single row at time behavior. This way you
could safely call it for any objects on any database and know the
results will be OK. I don't see much point in throwing errors when
batch insert isn't possible. If batch insert isn't possible, the user
needs to insert the values one row at a time in any case.

 - Anssi

On Tue, Mar 29, 2016 at 11:47 AM, Podrigal, Aron
 wrote:
> Hi,
>
> I worked on supporting bulk_insert for multi table inheritance here [1] I
> would like to get some feedback.
>
> bulk_create can be used in one of 2 ways
>
> 1) If you already have parent records in the database and want to
> bulk_insert into the child table only.
> 2) There isn't any parent records, and need to create them as well.
>
> In the first case, all database backends can support it. It simply requires
> the user to set the `parent_ptr` attributes for all child instances, and
> then do  bulk_create.
> In the second case it gets tricky and it cannot be supported for all
> databases for any model as it requires a way to get all the ids from the
> inserted parent records. Postgres is the most flexible in that case and
> supports bulk_insert for any table and any type of field because it supports
> the RETURNING clause so we can always retrieve the ids for the inserted
> rows. For sqlite we can only support bulk_create if the model does not have
> a parent with an AutoField.
> for MySQL I think we can rely on `LAST_INSERT_ID()` which will return the
> first ID out of the rows count inserted, so we can than generate a list of
> IDs of `range(last_insert_id, last_insert_id + count_rows)`.
>
> Can anyone confirm if we can rely on MySQL last_insert_id to be consistent
> and and without gaps for all records inserted at once?
>
> Thanks.
>
> [1]
> https://github.com/django/django/compare/master...ar45:allow_bulk_insert_multi_inheritance
>
>
> --
> Aron Podrigal
> -
> '101', '1110010', '110', '1101110'   '101', '110',
> '1100100', '1110010', '1101001', '1100111', '111', '1101100'
>
> P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CANJp-yh7SXmC1a3T3RZjdDApdAqo6Op06DsQCqOQ0LhbmVhY5g%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1EfyCqE37hbdB%3DvyTvCRmJ6zqspSfng1-kxn1Evwtr_Lg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Anssi Kääriäinen]

This being MySQL I wouldn't be surprised if changing the isolation
level would introduce new problems. Also, user code might rely on
Django using repeatable read. If we introduce the change in to stable
releases, we risk breaking sites that work perfectly well currently.
To me this is against our backwards compatibility rules which state
that we avoid doing such changes except when completely unavoidable.

 - Anssi

On Mon, Mar 21, 2016 at 10:40 AM, Shai Berger  wrote:
> My recommendation to backport is based on the observation that the peculiar
> REPEATABLE READ behavior is highly conductive to data loss in the presence
> of concurrency, combined with a sense that it is not very well known; I find
> it much more likely that the change will fix broken code than break really
> working code.
>
> On 21 במרץ 2016 09:59:25 GMT+02:00, "Anssi Kääriäinen" 
> wrote:
>>
>> I'm strongly -1 on changing the default isolation level in a minor
>> release. We can recommend users switch the level and complain loudly
>> if they don't. But just changing the isolation level has potential for
>> breaking working code.
>>
>>  - Anssi
>>
>> On Mon, Mar 21, 2016 at 9:27 AM, Shai Berger  wrote:
>>>
>>>  First of all, I would like to say that I strongly support the move to
>>> READ
>>>  COMITTED, including backporting it to 1.8.x.
>>>
>>>  But we also need to explain: REPEATABLE READ is a higher transaction
>>> isolation
>>>  level than READ COMMITTED. If you have problematic code, it should lead
>>> to
>>>  more deadlocks and/or transactions failing at commit time (compared to
>>> READ
>>>  COMMITTED), not to data loss. The reason we get data losses is MySql's
>>> unique
>>>  interpretation of REPEATABLE READ. If you're interested in the details
>>> (and if
>>>  you use MySql, you should be), read on.
>>>
>>>  With MySql's REPEATABLE READ, the "read" operations -- SELECT statements
>>> --
>>>  indeed act like they act in the usual REPEATABLE READ: Once you've read
>>> some
>>>  table, changes made to that table by other transactions will not be
>>> visible
>>>  within your transaction. But "write" operations -- UPDATE, DELETE,
>>> INSERT and
>>>  the like -- act as if they're under READ COMMITTED, affecting (and
>>> affected by)
>>>  changes committed by other transactions. The result is, essentially,
>>> that
>>>  within a transaction, the reads are not guaranteed to be consistent with
>>> the
>>>  writes [1].
>>>
>>>  In particular, in the bug[2] that caused this discussion, we get the
>>> following
>>>  behavior in one transaction:
>>>
>>>  (1) BEGIN TRANSACTION
>>>
>>>  (2) SELECT ... FROM some_table WHERE some_field=some_value
>>>  (1 row returned)
>>>
>>>  (3) (some other transactions commit)
>>>
>>>  (4) SELECT ... FROM some_table WHERE some_field=some_value
>>>  (1 row returned, same as above)
>>>
>>>  (5) DELETE some_table WHERE some_field=some_value
>>>  (answer: 1 row deleted)
>>>
>>>  (6) SELECT ... FROM some_table WHERE some_field=some_value
>>>  (1 row returned, same as above)
>>>
>>>  (7) COMMIT
>>>  (the row that was returned earlier is no longer in the
>>> database)
>>>
>>>  Take a minute to read this. Up to step (5), everything is as you would
>>> expect;
>>>  you should find steps (6) and (7) quite surprising.
>>>
>>>  This happens because the other transactions in (3) deleted the row that
>>> is
>>>  returned in (2), (4) & (6), and inserted another one where
>>>  some_field=some_value; that other row is the row that was deleted in
>>> (5). The
>>>  row that this transaction selects was not seen by the DELETE, and hence
>>> not
>>>  changed by it, and hence continues to be visible by the SELECTs in our
>>>  transaction. But when we commit, the row (which has been deleted) no
>>> longer
>>>  exists.
>>>
>>>  I have expressed elsewhere my opinion of this behavior as a general
>>> database
>>>  feature, and feel no need to repeat it here; but I think that, if
>>> possible, it
>>>  is Django's job as a framework to protect its u

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Anssi Kääriäinen]

I'm strongly -1 on changing the default isolation level in a minor
release. We can recommend users switch the level and complain loudly
if they don't. But just changing the isolation level has potential for
breaking working code.

 - Anssi

On Mon, Mar 21, 2016 at 9:27 AM, Shai Berger  wrote:
> First of all, I would like to say that I strongly support the move to READ
> COMITTED, including backporting it to 1.8.x.
>
> But we also need to explain: REPEATABLE READ is a higher transaction isolation
> level than READ COMMITTED. If you have problematic code, it should lead to
> more deadlocks and/or transactions failing at commit time (compared to READ
> COMMITTED), not to data loss. The reason we get data losses is MySql's unique
> interpretation of REPEATABLE READ. If you're interested in the details (and if
> you use MySql, you should be), read on.
>
> With MySql's REPEATABLE READ, the "read" operations -- SELECT statements --
> indeed act like they act in the usual REPEATABLE READ: Once you've read some
> table, changes made to that table by other transactions will not be visible
> within your transaction. But "write" operations -- UPDATE, DELETE, INSERT and
> the like -- act as if they're under READ COMMITTED, affecting (and affected 
> by)
> changes committed by other transactions. The result is, essentially, that
> within a transaction, the reads are not guaranteed to be consistent with the
> writes [1].
>
> In particular, in the bug[2] that caused this discussion, we get the following
> behavior in one transaction:
>
> (1) BEGIN TRANSACTION
>
> (2) SELECT ... FROM some_table WHERE some_field=some_value
> (1 row returned)
>
> (3) (some other transactions commit)
>
> (4) SELECT ... FROM some_table WHERE some_field=some_value
> (1 row returned, same as above)
>
> (5) DELETE some_table WHERE some_field=some_value
> (answer: 1 row deleted)
>
> (6) SELECT ... FROM some_table WHERE some_field=some_value
> (1 row returned, same as above)
>
> (7) COMMIT
> (the row that was returned earlier is no longer in the 
> database)
>
> Take a minute to read this. Up to step (5), everything is as you would expect;
> you should find steps (6) and (7) quite surprising.
>
> This happens because the other transactions in (3) deleted the row that is
> returned in (2), (4) & (6), and inserted another one where
> some_field=some_value; that other row is the row that was deleted in (5). The
> row that this transaction selects was not seen by the DELETE, and hence not
> changed by it, and hence continues to be visible by the SELECTs in our
> transaction. But when we commit, the row (which has been deleted) no longer
> exists.
>
> I have expressed elsewhere my opinion of this behavior as a general database
> feature, and feel no need to repeat it here; but I think that, if possible, it
> is Django's job as a framework to protect its users from it, at least as a
> default.
>
> On Monday 21 March 2016 02:25:37 Cristiano Coelho wrote:
>> What performance changes can you expect doing this change? It is probably
>> that default on MySQL for a good reason.
>
> The Django project is usually willing to give up quite a lot of performance in
> order to prevent data losses. I agree that this default on MySql is probably
> for a reason, but I don't think it can be a good reason for Django.
>
> Have fun,
> Shai.
>
> [1] https://dev.mysql.com/doc/refman/5.7/en/innodb-consistent-read.html
> [2] https://code.djangoproject.com/ticket/26347

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1EOwq7CC2bF6nL7-GuHPv_RJ_cf_giCtchtk2gzYYMr8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ GSOC 2016 ] providing standard interface for NoSQL databases

[Anssi Kääriäinen]

There are two goals when it comes to document DB support. First goal
is to allow usage of models written specifically for document oriented
databases. This is something that should be possible to achieve even
if there are some hard problems to solve. The other goal is to allow
usage of any Django model with document oriented databases just like
you would use it with relational databases. This goal is extremely
hard to achieve.

The reason the latter one is hard is that for example ForeignKey and
ManyToManyField do not make much sense with document oriented
databases. In addition most Django models use auto-incrementing
primary keys, and pretty much no distributed database supports those
properly. So, if you take a random Django model (say
django.contrib.auth.models.Permission) it will not be easy to fit that
model into document oriented databases. If I recall correctly, one of
the big problems for the GSoC project Russell mentioned was support
for any Django model with document oriented databases.

I believe there would be lots of uses for models which are written
specifically for document oriented databases. So +1 to the general
idea.

 - Anssi


On Wed, Mar 16, 2016 at 3:10 AM, Russell Keith-Magee
 wrote:
>
> It would also be worthwhile looking into the prior art on this topic. We had
> a GSoC project for non-relational data stores a few years back, and came to
> the conclusion that it probably wasn’t viable. You can dig into the archives
> to find out why.
>
> If you’re interested in adding non-relational support to Django, my
> inclination is that you’ll get more traction proposing a specific backend -
> I outlined what would be involved in my DjangoCon US talk last year:
>
> https://www.youtube.com/watch?v=VgM0qmpHDiE
>
> Yours,
> Russ Magee %-)
>
> On Wed, Mar 16, 2016 at 5:50 AM, Curtis Maloney  wrote:
>>
>> It sounds like by "NoSQL DBS" you mean specifically "Document store DBMS".
>> Is this correct?
>>
>> (I'm sure most people know my rant about how "NoSQL" is a misnomer, and
>> it's really "Non-relational" that people mean - it's not SQL per se they're
>> avoiding, it's the relational model)
>>
>> The NoSQL/Non-rel name is really too vague, as it covers "anything that's
>> not SQL / Relational"... which can cover key/value, document, graph, log,
>> and even filesystem or email, when it comes down to it.
>>
>> I think it will help if you clarify what model you _do_ target, not which
>> you do _not_.
>>
>> All that said/vented, I agree it would be nice to have a common
>> abstraction layer atop various document stores.  The trick now will be
>> finding sufficient commonalities.
>>
>> I feel that the ORM's relation-span/transforms/lookups syntax would be
>> equally viable on nested documents.
>>
>> One of the benefits of an abstraction layer is allowing other tools to
>> generically interact with these new objects.  What plans do you have for how
>> to support non-flat documents in, for instance, Forms?
>>
>>
>> --
>> Curtis
>>
>> On 16/03/16 05:29, girish ramnani wrote:
>>>
>>>
>>> Hello,
>>>
>>> I would like to propose providing a standard interface for NoSQL
>>> databases . Some the points to consider are:
>>>
>>>  1. The /Fields /which are used to define the schema of the sql
>>> databases can also be used to provide in No sql databases.
>>>  2. In NoSQL dbs ,developers usually store data in denormalized form. so
>>> in two related collections a meta tag /denormal = True /can be
>>> provided. Which would make that collection a child of the other
>>> model ( related ). In normal scenario the collections will be
>>> connected using _id used as a foreign key.
>>>  3. providing contrib.mongo with Field such as GridFSField for providing
>>> support for GridFS
>>>  4. considering the database transaction transaction blocking ( PyMongo
>>> provides blocking api for mongo )
>>>  5. creating Mixins such as /FlexibleMixin /which would allow Models to
>>> have dynamic fields. i.e directly insert fields which are not listed
>>> in the Model schema.
>>>
>>>
>>> there are lot more things (Migrations, settings etc) to consider but
>>> before that needed to know that I am going in the right direction or
>>> not?. Please suggest.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com
>>> .
>>> To post to this group, send email to django-developers@googlegroups.com
>>> .
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>>
>>> https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com
>>>
>>> 

Re: Proposal on Custom Indexes - Google Summer of Code

[Anssi Kääriäinen]

On Thu, Mar 10, 2016 at 8:17 AM, Markus Holtermann
 wrote:
> Hi Akshesh,
>
> thank you for your proposal! Sounds like a good plan.
>
> On Thursday, March 10, 2016 at 8:16:10 AM UTC+11, akki wrote:
>
>>> Once the index is added to a model, the model can inject itself into the
>>> index, or the schema editor can use the originating model to get data it
>>> needs. No need to pass the model into the index type I think. Regarding the
>>> name, I'm unsure if I prefer a name=Index() or an Index(name='') approach.
>>> It'd be good to see that choice in your proposal with a couple of pros and
>>> cons.
>>
>>
>> Well, since we are defining these indexes in a list (to be stored in
>> Meta.indexes) I would go with the Index(name='') approach.
>
>
> I'd prefer the Index(name='') approach
>
> class MyModel(models.Model):
> field1 = models.CharField()
> field2 = models.CharField()
>
> class Meta:
> indexes = [
> Index('field1', 'field2', name='some_index_across_two_columns'),
> 'field2',  # Would be the same as Index('field2'), an index on
> this one column -- name derived from field name: field2_index
> Index(ToLower('field1')),  # A functional index on field1 --
> name derived from field name and functions: field1_tolower_index
> ]
>
> That way you should also be able to check for duplicate index names.

How about

class MyModel(models.Model):
field1 = models.CharField()
field2 = models.CharField()

field1_idx = models.Index('field1')
field2_lower_idx = models.Index(ToLower('field2'))

The main advantage is that the same syntax can be used for custom
indexes, constraints, composite fields and so on. Internally we could
of course have the same representation for indexes as before.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1Eet%3DzPTKXJJpuRy8b0tWF%2BaA9-QaT%2BzfF%3DGV6wAd3N3g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal on Custom Indexes - Google Summer of Code

[Anssi Kääriäinen]

I was assuming that the Index class would be responsible for
generating the whole SQL. Seems like this isn't the case.

I assumed something like this would happen when a new index is added:
   1. User adds a new BTreeIndex to a model
   2. The makemigrations command detects a change and creates a
migration file with AddIndex operation
   3. The migrate command calls BTreeIndex.database_forwards, the
database_forwards method generates the needed SQL ('CREATE INDEX
foobar ON thetable(thecol)')
   4. The migration framework runs the generated SQL

This way would let the BTreeIndex class generate the full SQL.

If I am not mistaken, the plan is to generate the SQL mostly in the
migration operations, that is the user can't alter the full SQL by
creating custom Index classes. So, #3 above would be replaced by the
AddIndex operation detecting changes in the BTreeIndex class and
generating the needed SQL. The main point is that the AddIndex
operation isn't under user control, so parts of the generated SQL is
essentially hard-coded into Django.

Giving users full control over the generated SQL would allow contrib
or 3rd party apps to create fully custom index classes for their
needs. The expressions/transforms/lookups approach has shown this is a
powerful approach. As an example, it is unlikely we will add BRIN
index support to Django core, but it could be added to
django.contrib.postgres.

It is worth considering if we could allow users to have full control
of the generated SQL. It might be a bit too much for one GSoC program,
so I'm fine with a decision to use the hard-coded approach.

 - Anssi

On Sun, Mar 13, 2016 at 10:24 AM, akki  wrote:
>
>>
>> What about calling the attribute something like "constraints" similar to
>> how it's done in SQLAlchemy [1]? For now the attribute can just contain a
>> list of Index() instances but that would also lay grounds for supporting
>> check constraints and other related table level options.
>
> Since we are just allowing instances of Index class and it's subclasses
> right now, I don't think there is much gain in naming it "constraints". It
> might even confuse the user.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/090de3a1-63da-4897-9bb6-a7997a9b9610%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1Egv1R7XOA-bG9RPM4iRsCY88szCE7KPj%3Dzmq%3DGFJeBGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal on Custom Indexes - Google Summer of Code

[Anssi Kääriäinen]

On Wed, Mar 9, 2016 at 11:16 PM, akki  wrote:
>>
>> Of course, this is a great thing to have. But I wonder if we can do
>> something to prevent pushing non-indexes to the indexes attribute of
>> Meta. Having Meta.indexes = [CheckTypeConstraint(foo__in=['foo',
>> 'bar'])] doesn't read correctly.
>
>
> Perhaps it would be best not to mix constraints and indexes. I would like to
> know your thoughts on stopping to create indexes explicitly for unique
> constraints because all database do that automatically because that only
> gives a performance degradation as far as I know.

The point is that if the Index class generates the full SQL for the
index creation, then the Index classes can be used to run arbitrary
changes to the database. Having the ability to run arbitrary SQL is
great, but this will lead to users having all sorts of non-indexes in
the indexes meta attribute. I'd like to avoid that if possible.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1H%3DHQpYBVB24Pfe7NokhsCU0gnjAykV2GOo%2BO%3D2ig-PTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal on Custom Indexes - Google Summer of Code

[Anssi Kääriäinen]

If the CREATE INDEX part of the generated SQL isn't hard-coded
somewhere outside the Index class, the exact same mechanism can be
used to run other DDL commands, like ALTER TABLE  ADD
CONSTRAINT check_type_choices CHECK (type IN ('foo', 'bar'));

Of course, this is a great thing to have. But I wonder if we can do
something to prevent pushing non-indexes to the indexes attribute of
Meta. Having Meta.indexes = [CheckTypeConstraint(foo__in=['foo',
'bar'])] doesn't read correctly.

 - Anssi

On Wed, Mar 9, 2016 at 4:20 AM, Josh Smeaton  wrote:
> Hi Akshesh,
>
> The proposal looks really good so far. I haven't gone through it in depth
> though, but I'll set aside some time to try and do so. A few comments:
>
> - Using the schema editor rather than sqlcompiler is the right choice.
>
> - The earlier part of your proposal seems to focus on allowing postgres
> users to define more complex indexes that they previously haven't been able
> to use. Oracle is then highlighted also for allowing functional indexes. I
> think one of the biggest "why's" of this project is to allow users to define
> custom indexes. It doesn't matter what kind of index. They get access to the
> stuff that creates them. That should be highlighted before all else.
>
> - Should the type of index (BTree, Spatial, etc) be an index type in and of
> itself, or should it be a property of a specific index? My knowledge here is
> limited, so I'm not sure if there are certain constraints on which kind of
> indexes can use spatial/btree/partial storage types. If certain index types
> are very heavily restricted, then it probably makes sense to highlight them
> (as you've done) as a top level class rather than an attribute of indexes in
> general. If most indexes can be arranged with different storage types
> though, it'd probably make more sense to expose that functionality as an
> attribute of all indexes.
>
> - "Index() subclasses will also have a supports(backend) method which would
> return True or False depending on whether the backend supports that index
> type." Backends (operations.py) already supports
> check_expression_support(self, expression). You can bake index support into
> these methods from within each backend. Especially helpful for third party
> backends, as they won't need to monkey patch the index classes. The only
> issue I can see with reusing check_expression_support is the overhead of all
> other expressions being checked. I don't think that's a big concern, but
> it's worth being conscious of.
>
> - "Index classes will take 3 arguments fields, model and name - the later
> two being optional.". Index classes should accept a list of expressions, not
> just field names. This will allow trivial support of functional indexes
> without a specific functional index class. You'll also benefit from existing
> expression subclasses which process field names as F() expressions. F()
> expressions may have to be changed (or a new concept introduced) which can
> resolve the field, but doesn't try to do any joining.
>
> fullname_index=Index(ToLower('first_name'), ToLower('last_name')) -> create
> index _fullname_index on (Lower(first_name),
> Lower(last_name));
>
> Once the index is added to a model, the model can inject itself into the
> index, or the schema editor can use the originating model to get data it
> needs. No need to pass the model into the index type I think. Regarding the
> name, I'm unsure if I prefer a name=Index() or an Index(name='') approach.
> It'd be good to see that choice in your proposal with a couple of pros and
> cons.
>
>
> Ok, onto the questions you actually asked.
>
> 1) I don't think it's necessary to have an IndexTogether type.
> Index('field_a', 'field_b') should be sufficient I think. If it's not
> sufficient, call out why. I'm also in favour of translating any
> index_together definitions into appropriate Index types. Then SchemaEditor
> only needs to work with Index expressions and things remain simpler at the
> boundary between model definition and migrations.
>
> 2) Again I think it's a good idea to internally create index classes based
> on older (unique=True) definitions. Unique may not have to have it's own
> index type as it could (maybe) be a property of an Index. Again, I'll leave
> that choice up to you, but it'd be good to see the pros/cons briefly
> discussed.
>
> Again, really good proposal!
>
> On Wednesday, 9 March 2016 01:46:02 UTC+11, akki wrote:
>>
>> Hi
>>
>> My name is Akshesh Doshi (akki). I am a student at Indian Institute Of
>> Technology, Roorkee (IITR). I have been contributing to Django for quite
>> some time now and my experience has been really great till now. I found the
>> community to be very welcoming and have learnt a lot in this period of time.
>>
>> With this spirit I would like to work on the idea of "Custom Indexes"  and
>> extend it as my proposal for Google Summer of Code 2016.
>>
>> I have started preparing my proposal and here is the initial draft of it.
>> I would like

Re: Copy-from-base model inheritance.

[Anssi Kääriäinen]

The recommended way to do what you are after is:

class AbstractBase(models.Model):
f1 = models.IntegerField()

class Meta:
abstract = True

class Base(AbstractBase):
 pass

class Derived(AbstractBase):
f2 = models.IntegerField()


If you can't use the above way, then the way forward would be something like:

class Base(models.Model):
f1 = models.IntegerField()

class Derived(make_abstract(Base)):
f2 = models.IntegerField()

It might be possible to create the make_abstract() method outside of
Django. If not, we might be able to improve internals so that
make_abstract() would be easy to write outside of Django.

I don't think we want this inbuilt, as the make_abstract() model
design is inferior to using real abstract model as base for both
models.

 - Anssi

On Wed, Mar 2, 2016 at 2:30 AM, Joakim Saario  wrote:
> Hello!
>
> Wouldn't it be great to be able to inherit django models like any other
> python class? I.e not like the default multi-table inheritance.
>
> "How would this differ from an abstract model" you may ask. Well, it is sort
> of like an abstract model inheritance, except for the abstract part where
> the base model isn't registered
> and need to have abstract = True
>
> For example
>
> class NormalModel(models.Model):
> foo = models.CharField(max_length=10)
> bar = models.CharField(max_length=10)
>
> class CopiedBaseModel(NormalModel):
> buzz = models.CharField(max_length=10)
>
> class Meta:
> copy_from_base = True
>
>
> Would be equivalent to
>
> class NormalModel(models.Model):
>
> foo = models.CharField(max_length=10)
> bar = models.CharField(max_length=10)
>
> class CopiedBaseModel(NormalModel):
> foo = models.CharField(max_length=10)
> bar = models.CharField(max_length=10)
> buzz = models.CharField(max_length=10)
>
>
> My original use case for this was with django-cms, which didn't play well
> with the multi-table inheritance when I needed to extend a built in plugin
> of theirs. So I ended up copying the whole model.
> which annoyed me that it was the only way to do it.
>
> This also isolates the models from each other in constrast to multi-table
> inheritance, as the base class is just copied over to the new class, it
> shares this functionality with abstract models. Think of it as a kind of
> reverse abstract model. I wrote some code for this and actually reused much
> of the code used for abstract models. (see attatched patch)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/62b0b093-c7e5-4abb-a797-63277b31d27c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GEtPXYdA4c8Uv8Ss4YLHJ1XH%2Bv35NYH1uCEg2WGpm_Xw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

New deferred loading implementation (Was: [Review Request] Added support for database delegated fields (like AutoField))

[Anssi Kääriäinen]

On Thursday, February 11, 2016 at 2:41:44 PM UTC+2, Florian Apolloner wrote:
>
> Oh, I somewhat missread and though there would be a new DEFERRED argument, 
> the backwards issue is easy enough though:
>
>  * Unless I miss something, YourModel.__init__ is Model.__init__ if the 
> user didn't change it -> pass is DEFERRED
>  * If the user changed it check for model._meta.new_style_deferred and 
> continue accordingly
>  * Raise a warning if the __init__ is a custom one and new_style_deffered 
> is not set…
>

If we are going to introduce a backwards compat system for this, then I 
think I want to get rid of calling Model.__init__ at all when loading from 
DB. We get faster model initialization, and conceptually loading from DB is 
like unpickling which (correctly) doesn't call __init__.

However, based on the comments in the PR, I think we are going to just 
document the change to __init__ and skip deprecation.

 - Anssi
 

>
> On Thursday, February 11, 2016 at 1:38:44 PM UTC+1, Florian Apolloner 
> wrote:
>>
>>
>>
>> On Thursday, February 11, 2016 at 10:51:59 AM UTC+1, Anssi Kääriäinen 
>> wrote:
>>>
>>> Before doing any further work on this we should decide if the 
>>> Model.__init__() problem is bad enough to stop this cleanup, and if so, do 
>>> anybody have any ideas of how to avoid the problem?
>>>
>>
>> I do not think Model.__init__() is anywhere near public API, add it to 
>> the release notes and be done with it, worst case add a try/except around 
>> it…
>>
>> Cheers,
>> Florian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1190f1dd-dda9-445b-8a1c-396584553dd1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Review Request] Added support for database delegated fields (like AutoField)

[Anssi Kääriäinen]

On Wednesday, February 3, 2016 at 8:47:16 AM UTC+2, Anssi Kääriäinen wrote:

> For the update_fields change, I think we can do that completely 
> separately. The same goes for changing the way how deferred fields are 
> implemented.
>

I created a proof-of-concept pull request for improved deferred fields 
loading. See https://github.com/django/django/pull/6118. In short, it is 
faster and cleaner (no more dynamically created subclasses for deferred 
loading!) than the previous implementation, but we have a couple of 
backwards incompatibilities:
1) Model.__init__() will be called with value DEFERRED for those fields 
which are deferred. Previously those fields were not given to __init__() at 
all.
2) The class level attribute Model._deferred must be removed (we can't 
have class level attribute as we don't have dynamic classes any more)
3) A potential name clash for fields when one has a class method and 
field with clashing names.

Of these the Model.__init__() case could be problematic for those users who 
have overridden __init__().

Before doing any further work on this we should decide if the 
Model.__init__() problem is bad enough to stop this cleanup, and if so, do 
anybody have any ideas of how to avoid the problem?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ced9d758-4cfe-4012-b7fc-37aaf221a83e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Query on BooleanField with values other than True and False, bug or intentional?

[Anssi Kääriäinen]

I'm with Tim here - we shouldn't alter the behavior at this point.

On Thu, Feb 4, 2016 at 8:17 AM, Chris Foresman  wrote:
> I don't think that's the same thing. We're talking about interfacing with a
> system that has clearly delineated values.
>>
>> The boolean type can have several states: "true", "false", and a third
>> state, "unknown", which is represented by the SQL null value.[1]
>
>
> While the database accepts a variety of optional string literals to insert
> true or false, none of them are truly comparable to Python's truth-y or
> false-y values. For instance, psycopg2 only uses True, False, and None to
> represent TRUE, FALSE, and NULL in the database.[2] That's why values passed
> to filter on a BooleanField are cast to bool.
>
> While allowing some slop might be considered more "Pythonic" is some cases,
> I feel "explicit is better then implicit" in this case given that SQL
> doesn't allow boolean columns to be set in an implicit "Pythonic"
> truth-y/false-y way.
>
>
> [1](http://www.postgresql.org/docs/9.1/static/datatype-boolean.html)
> [2](http://initd.org/psycopg/docs/usage.html#constants-adaptation)
>
>
> On Tuesday, February 2, 2016 at 4:23:37 AM UTC-6, HM wrote:
>>
>> I don't think enforcing it would be pythonic.
>>
>> Once upon a time, Python lacked True and False, all you had were
>> truthy and falsey and maybe per-project/team constants:
>>
>> http://python-history.blogspot.no/2013/11/story-of-none-true-false.html
>>
>> I remember the arguments when bool() was added. Those against were
>> worried that it would send a signal that would lead to a slippery
>> slope where truthy and falsey would disappear because people would
>> insist that "it's pretty sloppy to not enforce an explicit boolean
>> value and can lead to subtle bugs".
>>
>> Well..
>>
>>
>> On 31 January 2016 at 10:34, Adam Johnson  wrote:
>> > Just to play devil's advocate... you're all worrying about one simple
>> > case;
>> > there are infinite variants of it with subtle bugs, for example imagine
>> > the
>> > same situation but with Value:
>> >
>> > Whatever.object.filter(is_active=Value('false'))
>> >
>> > The ORM can't predict the type of a Value call - pretty much by
>> > definition,
>> > since it's a raw value passed to the database. So you might be able to
>> > fix
>> > BooleanField for a few cases, but you can't fix them all..
>> >
>> >
>> >
>> > On Friday, January 29, 2016 at 10:38:52 PM UTC, Chris Foresman wrote:
>> >>
>> >> Sorry, sbrandt noted the issue of subtle bugs, not Maxime.
>> >>
>> >>
>> >> On Friday, January 29, 2016 at 4:37:51 PM UTC-6, Chris Foresman wrote:
>> >>>
>> >>> I have to agree here; it's pretty sloppy to not enforce an explicit
>> >>> boolean value and can lead to subtle bugs. In addition to the one
>> >>> mentioned
>> >>> by Maxime, consider the case of a nullable boolean field:
>> >>>
>> >>> >>> bool(None)
>> >>> False
>> >>>
>> >>> Maybe that field has a better converter for possible values and
>> >>> explicitly allows `None`, but I think it would be fairly trivial to
>> >>> add a
>> >>> stricter check and pretty easy to fix code that's not backwards
>> >>> compatible
>> >>> with find/replace.
>> >>>
>> >>>
>> >>> On Friday, January 22, 2016 at 3:18:45 PM UTC-6, Maxime Lorant wrote:
>> 
>>  At least, the behaviour is predictable : it uses `bool(value)`. I
>>  believe it is not a bug but more a question about the input
>>  definition:
>>  should we allow non boolean value? I don't see any problem here
>>  accepting a
>>  string as a `True` value, it is the job of the user to ensure the
>>  value is
>>  castable to a boolean.
>> 
>>  The same behaviour is found on `IntegerField` for example:
>>  `Model.objects.filter(pk="foo")` raises `ValueError: invalid literal
>>  for
>>  int() with base 10: 'foo'` which implies the ORM tried to cast 'foo'
>>  as an
>>  integer.
>> 
>>  On Friday, January 22, 2016 at 8:51:41 PM UTC+1, Kaveh wrote:
>> >
>> > Today I discovered I can use strings and numbers to query on
>> > BooleanFields.
>> >
>> > Let's say we have the following model:
>> >
>> > class Whatever(models.Model):
>> > name = models.CharField(max_length=200)
>> > is_active = models.BooleanField()
>> >
>> > The following queries return all instances which their `is_active`
>> > field is True:
>> >
>> > Whatever.object.filter(is_active=True)
>> > Whatever.object.filter(is_active='some_random_text')
>> > Whatever.object.filter(is_active=777)
>> >
>> > and the followings return the ones which are False:
>> >
>> > Whatever.object.filter(is_active=False)
>> > Whatever.object.filter(is_active='')
>> > Whatever.object.filter(is_active=0)
>> >
>> > Is this behaviour intentional or is it a bug? Should queries on
>> > BooleanFields accept strings and numbers?
>> >
>> > --
>> > You received this message because you are subscri

Re: Replacing the contrib.sites Site model with a setting?

[Anssi Kääriäinen]

Could we make the sites portion of Django an interface? In settings you 
give something like SITES_PROVIDER = 'myapp.sites.MultiTenancySite', and 
that provider then needs to fulfill a given API. This way we wouldn't need 
to offer anything complex in-built, but users would be free to do whatever 
they want to.

 - Anssi

On Saturday, January 30, 2016 at 11:11:08 AM UTC+2, Aymeric Augustin wrote:
>
> This is a valid use case, but not one django.contrib.sites (officially) 
> supports, since it requires a constant SITE_ID setting. 
>
> While Tim's suggestion doesn't cause a regression, it doesn't make this 
> pattern easier to implement either. 
>
> It's definitely worth considering. 
>
> -- 
> Aymeric. 
>
> > Le 30 janv. 2016 à 08:01, Max Arnold > a 
> écrit : 
> > 
> > What if this list of sites needs to be changed dynamically without app 
> restart (multitenancy)? 
> > 
> >> On Fri, Jan 29, 2016 at 12:45:02PM -0800, Tim Graham wrote: 
> >> In another thread about adding a "scheme" field to the Site model [1], 
> I 
> >> floated the idea of moving the data stored by the Site model into a 
> setting: 
> >> 
> >> I've sometimes thought that the Site model violates the principle that 
> you 
> >> shouldn't put configuration in your database. I guess there's some 
> >> usefulness to having a ForeignKey to the site, but... would it be 
> feasible 
> >> to offer a SITES setting that could be used instead? e.g. 
> >> 
> >> SITES = { 
> >>1: {'scheme': 'http', 'domain': example.com, 'name': 'My Site'}, 
> >>... 
> >> } 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Django developers  (Contributions to Django itself)" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com . 
> > To post to this group, send email to django-d...@googlegroups.com 
> . 
> > Visit this group at https://groups.google.com/group/django-developers. 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/20160130070141.GA2770%40otg.dm.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2014647b-aa3d-412f-8fc3-a0bc646e9a4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Review Request] Added support for database delegated fields (like AutoField)

[Anssi Kääriäinen]

The options for save() API seem to be:
  1. refresh_fields=[...] for save() - if set, then issue RETURNING for 
fields mentioned in the refresh_fields API.
  2. expressions: my_model.updated_at = TransactionNow(); my_model.save(), 
where TransactionNow is flagged to refresh the value from DB post-save.
  3. field flags like done in the PR.

Field flags are nice for things you always want to load from database (say, 
updated_at).

If the use cases we target are "generate value in db on insert" and 
"generate value in db on insert or update", and only those, then I think we 
can go with just a flag that tells Django to always skip the model's value 
on save(), and to reload from DB always. 

We are going to override the assigned value on save(), but I think it has 
to be this way. Consider:
  a_object.save()  # a_object.updated_at is set to '2016-02-03' as it is 
database delegated
  a_object.save()  # a_object.updated_at has a value, so we should error if 
we don't ignore the value
forcing users to set the value to None pre-save breaks the whole field flag 
feature. We could go directly with expressions in that case. We could also 
use the value in the update and require the DB trigger to ignore the value.

We don't have a separate flag for doing the refresh on insert only, but 
that isn't IMO such a big problem. We are going to add a minor overhead to 
updating by refreshing a field value from the DB when it hasn't changed, 
but making the API simpler is worth this IMO.

So, my opinion is that for the field flags API we need just one flag. A 
name like db_generated comes to mind. This gives a nice way to support the 
most common use case - that of calculating an inserted_at/updated_at field 
directly in DB.

After we have the field flag API, I think we should aim for expressions 
support. For expressions we should have enough flags for allowing full 
control over save() behavior. So, if the db_generated field flag isn't 
enough for some use case, you can go low-level and use expressions 
directly. We could perhaps even allow db_generated value to be an 
expression - if so, that expression would be used on save() and 
bulk_create().

For the update_fields change, I think we can do that completely separately. 
The same goes for changing the way how deferred fields are implemented.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dedc791d-232d-4c93-9716-687963ca5272%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remove null assingment check for non-nullable fields?

[Anssi Kääriäinen]

On Wednesday, February 3, 2016 at 1:26:26 AM UTC+2, Tim Graham wrote:
>
> There's a proposal to remove this behavior:
>
>
> >>> obj.fk = None
> ValueError('Cannot assign None: "Obj.fk" does not allow null values.)
>

I don't see this check as necessary - if you try to save such an object you 
will get an error. If you don't, then there is no harm done.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e11ba330-bdcb-4083-b790-dbf57b4d38aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Review Request] Added support for database delegated fields (like AutoField)

[Anssi Kääriäinen]

On Monday, February 1, 2016 at 10:33:26 AM UTC+2, Aymeric Augustin wrote:
>
> > On 31 janv. 2016, at 22:55, Shai Berger  > wrote: 
> > 
> > Your message seems to be confusing the queryset API with the 
> model-instance 
> > API. 
>
> Oops :-( 
>
> Anyway, it seems that we agree on: 
>
> - controlling this behavior in the query rather than in the field 
> definition — this avoids the awkward “ignore what the field says” query 
> parameter 
> - trying not to provide separate APIs for insert and update behavior 
> - improving the save(update_fields=…) API to support inserts as well as 
> updates 
>

For the .save() method we could go with a new argument fields=[...]. The 
way it works is that only the mentioned fields are inserted or updated. If 
you want the current behavior of update_fields, set force_update=True.

The main reason why update_fields patch didn't ever try to target insert 
statements was that the feature didn't make much sense. We didn't have 
database defaults and we didn't have any other sort of expressions support 
for inserts. There was very limited use cases for skipping fields in 
insert. I'm not sure we have much of an use case now either (notably one 
can write DbDefaultExpression that compiles to sql, params = "default", [] 
already). What is the main use case for this feature?

I've been toying around with an idea that we add a non-data descriptor (one 
implementing just __get__) for all fields of an model, and use this for 
deferred loading. Currently we use dynamically created subclasses of the 
model which is a hack. The way this would work is that if an instance's 
__dict__ doesn't have any value for given field, then we interpret that 
field as deferred. This would also give a natural way to refresh a given 
field from DB. You would set a field "deferred" manually by "del 
instance.field", and after that instance.field would reload that field. The 
descriptor wouldn't cost much, as non-data descriptors don't add any 
measurable overhead when the attribute is set on the instance's __dict__. I 
think the feature would be backwards compatible (though without trying it 
is hard to be sure).

This feature would also allow skipping fields on insert: instance = 
MyModel(); del instance.some_field; instance.save() -> some_field is 
skipped as it is deferred; instance.some_field -> some_field is refreshed 
from DB. This would close the gap in the APIs.

I would like to see support for expressions that can tell Django the field 
should be skipped in update/insert and if the field should be refreshed 
from database after update/insert. This would give a nice low-level API for 
manipulating the behavior of save() and bulk_insert(). I don't see a need 
to mix this with the current pull request. The biggest problem for such 
expressions is that they don't make any sense when using a model. For 
example TransactionNow() isn't a nice value for last_update field in any 
other context than saving. Not sure this is a big enough problem to worry, 
but something to consider anyways.

Refreshable expressions would give a natural way for skipping fields on 
insert. Also, things such as database defaults would be easy to use with 
the ORM:

class DbDefault(Expression):
refresh_on_insert = True
refresh_on_update = True

def as_sql(self, compiler, connection):
return "default", []

my_instance = MyInstance(uuid=DbDefault())
print(my_instance.uuid) -> "DbDefault()"
my_instance.save()
print(my_instance.uuid) -> "f81d4fae-7dec-11d0-a765-00a0c91e6bf6"  (or 
whatever the DB happens to generate)

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5310c9fc-d26a-469b-9a9c-dcdd437195b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making QuerySets composable

[Anssi Kääriäinen]

We have room for some improvement here. It should be possible to add
subqueries to the current query. This would be especially useful for
adding aggregates.

For aggregates we need this feature because the current API doesn't
work for anything but simple cases. For example
Book.objects.annotate(authors_cnt=Count('authors'),
suppliers_cnt=Count('suppliers')) will currently produce two joins,
one to authors and one to suppliers. Thus, if a book has three authors
and two suppliers, it will have both counts as six. Solving this
problem (taking in account .filter() calls, too) is extremely complex.

If the user could instead tell the ORM to add the counts in
subqueries, the ORM would have much easier job of generating a correct
query.

For the "add single aggregate case" it might be sufficient to add a
strategy flag to aggregates. So,
Book.objects.annotate(authors_cnt=Count('authors',
strategy=models.SUBQUERY)) is a possible API.

If you want multiple aggregates in a single subquery we likely need
something more explicit. Maybe something like
Book.objects.add_related(
authors_subq=models.AggregateSubquery(
Author.objects.annotate(cnt=Count('pk'), avg_age=Avg('age')))
This would add authors_subq__cnt and authors_subq__avg_age
automatically to the outer query as annotations.

The add_related() method would come handy in other cases, too.
Consider joining a translation for currently active language:
Book.objects.add_related(
current_translation=models.ModelRelation('translations',
only=Q(lang=get_current_language()))

You could also attach raw SQL in the same way:
Room.objects.add_related(
current_reservation=models.RawSQLRelation(
"select * from reservations where start_date <=
now()::date and end_date >= now()::date",
on_condition=Q(pk=F('current_reservation__room_id')),
model=Reservation)
))

We could later on add support for more esoteric concepts like lateral
joins or materializing the subquery into a temporary table.

I believe the current API for aggregates and multivalued relations has
come to a dead end. Simpler queries work well enough with the current
API. For more complex queries the API isn't easy to understand, and it
is extremely hard to translate complex ORM queries to SQL. Consider
Book.objects.filter(
Q(authors__height__gte=180), ~Q(authors__age__gte=50)
).annotate(author_cnt=Count('authors'))

This query is *extremely* hard to translate to SQL. Just the first
filter case is something nasty, the Q-object doesn't mean "keep all
rows that has author's age less than 50 and height greater than 180".
It means "a book must have at least one author taller than 180. *All*
of such authors for the book must have age less than 50 or the age
must be null". Still, we need to generate a count over such authors
for the book. And this example doesn't even reveal the full complexity
of the problem.

In summary, I vote +1 for the ability to compose querysets. The first
issue to tackle is to come up with a good extensible API for this.

 - Anssi

On Fri, Jan 22, 2016 at 6:44 PM, charettes  wrote:
> Hi Patryk,
>
>> Currently we have Prefetch objects that kind of solve this problem for M2M
>> relations but perhaps it would be nice to also be able to use QuerySets in
>> select_related() or even in filter(). I don't think Prefetch objects are
>> best
>> suited for that and I personally find having to instantiate them
>> explicitly
>> kind of ugly.
>
> From what I understand you're not a fan of Prefetch objects but I think most
> of your problems can be solved by using them and custom managers as the
> formers
> also support foreign keys[1].
>
> class AuthorDisplayManager(models.Manager):
> def get_queryset(self):
> queryset = super().get_queryset()
> return queryset.filter(
> visible=True,
> ).annotate(
> num_books=Count('books'),
> num_awards=Count('awards'),
> ).prefetch_related(
> 'foo',
> )
>
> class Author(models.Model):
> ...
> for_display = AuthorDisplayManager()
>
>
> class BookDisplayManager(models.Manager):
> def get_queryset(self):
> queryset = super().get_queryset()
> return queryset.filter(
> visible=True,
> ).prefetch_related(
> Prefetch(
> 'author',
> Author.for_display.all(),
> )
> )
>
>
> class Book(models.Model):
> ...
> for_display = BookDisplayManager()
>
> Simon
>
> [1] https://code.djangoproject.com/ticket/17003
>
> Le vendredi 22 janvier 2016 05:25:24 UTC-5, Patryk Zawadzki a écrit :
>>
>> Hi,
>>
>> Currently the way QuerySets work makes it hard to reuse code. Consider an
>> example where I have template tags to render an author and a book (that in
>> turn displays author). I don't want my template code to execute any
>> additional queries per row displayed so I am forced to prefetch ev

Re: Make ManyToMany fields support "to_field".

[Anssi Kääriäinen]

On Tue, Jan 12, 2016 at 11:22 PM, Shai Berger  wrote:
> Personally, I'd like it better if add() could still be used with an explicit
> through model, provided that instances of the through model could be created
> using only the two FK values

+1

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1G24qNwu1d2%2BFoBS%3DfyKw%3Dv%2Bv09kdani%2BmA_BrsqNrRwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: group-by promotion

[Anssi Kääriäinen]

On Saturday, January 9, 2016 at 7:01:38 AM UTC+2, Curtis Maloney wrote:
>
> Hey all, 
>
> I'm trying to write a PR for adding Postgres9.4's "FILTER" syntax for 
> aggregates, but I've hit a bug I can't figure how to fix [and nobody's 
> awake on IRC :)] 
>
> https://github.com/django/django/pull/5956 
>
> Basically, for the query: 
>
>  qset = FilterAggParentModel.objects.annotate( 
>  zero_count=FilterAgg(Count('filteraggtestmodel'), 
> Q(filteraggtestmodel__integer_field=1)), 
>  child_count=Count('filteraggtestmodel'), 
>  ) 
>
> It generates the SQL: 
>
> SELECT "postgres_tests_filteraggparentmodel"."id", 
>  COUNT("postgres_tests_filteraggtestmodel"."id") FILTER(WHERE 
> "postgres_tests_filteraggtestmodel"."integer_field" = 0) AS "zero_count", 
>  COUNT("postgres_tests_filteraggtestmodel"."id") AS "child_count" 
> FROM "postgres_tests_filteraggparentmodel" 
> LEFT OUTER JOIN "postgres_tests_filteraggtestmodel" ON 
> ("postgres_tests_filteraggparentmodel"."id" = 
> "postgres_tests_filteraggtestmodel"."parent_id") 
> GROUP BY "postgres_tests_filteraggparentmodel"."id", 
> "postgres_tests_filteraggtestmodel"."integer_field" 
>
>
> The problem being that 
> "postgres_tests_filteraggtestmodel"."integer_field" is included in the 
> GROUP BY clause... 
>
> Does anyone who understands this part of the ORM have a suggestion for 
> how to fix this? 
>

If you don't return anything from get_group_by_cols() no part of the 
expression should be added to the GROUP BY clause. I wonder why the 
integer_field is added to the group by, a bug somewhere in compiler maybe?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/980bd68a-f75c-4d64-a18f-58420bc45859%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: PostGres 9.5 Upsert

[Anssi Kääriäinen]

On Sun, Jan 10, 2016 at 5:22 PM, Sean Brant  wrote:

> I've always considered this a bug waiting to happen. I have seen many errors
> with the get operation failing because it returned more then one value.
> Usually you don't notice the error until you hit production. I always
> suggest the lookup use fields that have unique indexes.
>
> Changing that would be backwards incompatible so maybe it's a docs issue.

If there are no valid use cases for using get_or_create() without an
unique index, then we could consider doing a backwards incompatible
change here. I believe there are cases where you want to do a
get_or_create() without  unique index (though I can't come up with one
right now).

We could always check if there is a backing unique index and use
get_or_create only in those cases.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1EbwD7hJLFOjsiUeMxofov9zwRbX_Zr%2BZArqdy%2Bj_fT_g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: PostGres 9.5 Upsert

[Anssi Kääriäinen]

On Sun, Jan 10, 2016 at 11:10 AM, Florian Apolloner
 wrote:
> On Sunday, January 10, 2016 at 9:09:37 AM UTC+1, Anssi Kääriäinen wrote:
>>
>> The save() operation matches the semantics of upsert exactly - maybe we
>> could use upsert there?
>
> Not sure .save() would be a good candidate for that. In the best case,
> .save() should execute one INSERT or UPDATE query without extra junk for
> "error" handling. Ie if you set a pk and want to update then pass
> force_update into it. I know that we already try UPDATE followed by INSERT
> in save() but I am wondering how often that is actually used by people.

Yes, it is likely that save() wouldn't benefit much from using upsert.
Upsert is likely a bit slower than plain update, and we end up doing
an update in almost all cases anyways. It could be an useful option
for some use cases, but it likely isn't a good default.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1H4d%3DqEm--XrZuf310VXhGP_zt3q6pjQ0i-tV4quTYuBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: PostGres 9.5 Upsert

[Anssi Kääriäinen]

If I recall correctly, MySQL doesn't offer a way to specify on which index
you want to do the conflict resolution. This leads to problems - the upsert
might affect the wrong row if there are multiple unique indexes on the
table.

PostgreSQL's version of upsert has a problem, too. It doesn't offer a
direct way to know if the result of the upsert was an insert or update, but
Django needs that knowledge, at least for save(). Maybe there is a way (the
oid return value seems promising).

For get_or_create and update_or_create the problem is that the user is free
to offer any condition to be used for matching, but PostgreSQL limits the
upsert matching to columns in unique index. So, we can use upsert only for
unique index cases.

The save() operation matches the semantics of upsert exactly - maybe we
could use upsert there?

 - Anssi

On Sunday, January 10, 2016, Cristiano Coelho 
wrote:

> I agree! Also, does this already happen for the MySQL backend? MySQL has
> the insert on conflict update, that could work the same way.
> However, if I'm not wrong, the docs states that the above methods have a
> race condition (obvious since right now it does two operations), but if the
> code would actually use native database operations, the race conditions
> might be gone for those cases, so that should probably be documented as
> well.
>
> El viernes, 8 de enero de 2016, 21:13:26 (UTC-3), bliy...@rentlytics.com
>  escribió:
>>
>> Hey Guys,
>>
>> Postgres 9.5 has added the functionality for UPSERT aka update or
>> insert.  Any interest in aligning UPSERT on the db layer with the
>> get_or_create or update_or_create functionality in django?  Sounds like my
>> company would be interested in doing the work if the PR will get the
>> traction.
>>
>> -Ben
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ae38ba8e-3e79-47fb-92b9-dd305176c58e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1Gub-uL3AdgFSxQkdUNNe_Q47%3D7O%3DZ4qsW3Exwjhkd4ZA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: delegating our static file serving

[Anssi Kääriäinen]

In my opinion one of the most important goals for Django is to make
deploying tiny projects securely and easily a trivial matter. We
aren't there right now.

If you have a site where traffic is less than one hit per minute, you
shouldn't care about performance that much.

With Django, it is pretty easy to write a decent "one-off" application
in an hour. Unfortunately, proper (that is, secure and automated)
deployment takes a lot longer than that.

Maybe part of the problem is that core developers tend to work on
larger sites, where you likely want a customized deployment anyways,
and working on deployment scripts doesn't take such a large portion of
the total time.

Anything that improves the development experience for tiny sites gets
a big +1 from me.

 - Anssi

On Thu, Dec 31, 2015 at 1:17 PM, Jannis Leidel  wrote:
>
>> On 29 Dec 2015, at 01:36, Tim Graham  wrote:
>>
>> I'd like to work together with Dave to develop a proof of concept that 
>> integrates whitenoise into Django. I spent about an hour looking through 
>> whitenoise and our own static file serving code, and I think integrating 
>> whitenoise will yield a simpler user experience with about the same amount 
>> of code as have now.
>>
>> Essentially, we'd recommend adding something like this to existing wsgi.py 
>> files (it would be in the default startproject template)
>>
>> from whitenoise.django import DjangoWhiteNoise
>> application = DjangoWhiteNoise(application)
>> application.add_files(settings.MEDIA_ROOT, prefix=settings.MEDIA_URL)
>>
>> which would have the benefit of working out of the box in production too, I 
>> think. Of course, you could disable that based on settings.DEBUG or some 
>> other toggle.
>>
>> We could then deprecate:
>>
>> * django/contrib/staticfiles/views.py
>> * django/contrib/staticfiles/management/commands/runserver.py
>> * django/contrib/staticfiles/handlers.py
>> * django/views/static.py
>>
>> Any objections to doing further investigation in this area?
>
> None, this sounds like a great combination and a logical continuation of what 
> I did back then with staticfiles.
>
> The fact that staticfiles had all this custom logic to handle file serving in 
> staticfiles was a necessary evil to stay backward compatible and lower the 
> risk of staticfiles becoming a failure -- it wasn’t clear at all if it’d 
> match the workflow for most of Django’s user base (and it still hasn’t 
> completely I think). Whitenoise is agnostic enough about where the served 
> files come from so this fits nicely.
>
>> On Saturday, June 20, 2015 at 8:09:11 AM UTC-4, David Evans wrote:
>> On Friday, 5 December 2014 19:14:29 UTC, Carl Meyer wrote:
>> On 12/04/2014 10:33 PM, Collin Anderson wrote:
>> > Hi All,
>> >
>> > I'm pretty interested in getting secure and _somewhat_ efficient static
>> > file serving in Django.
>> >
>> > Quick history:
>> > 2005 - Jacob commits #428: a "static pages" view.  Note that this view
>> > should only be used for testing!"
>> > 2010 - Jannis adds staticfiles. Serving via django is considered "grossly
>> > inefficient and probably insecure".
>> > 2011 - Graham Dumpleton adds wsgi.file_wrapper to Gunicorn.
>> > 2012 - Aymeric adds StreamingHttpResponse and now files are read in chunks
>> > rather than reading the entire file into memory. (No longer grossly
>> > inefficient IMHO.)
>> >
>> > I propose:
>> > - Deprecate the "show_indexes" parameter of static.serve() (unless people
>> > actually use it).
>> > - Have people report security issues to secu...@djangoproject.com (like
>> > always)
>> > - Audit the code and possibly add more security checks and tests.
>> > - add wsgi.file_wrapper support to responses (5-line proof of concept:
>> > https://github.com/django/django/pull/3650 )
>> > - support serving static files in production, but still recommend
>> > nginx/apache or a cdn for performance.
>> > - make serving static files in production an opt-in, but put the view in
>> > project_template/project_name/urls.py
>> >
>> > I think it's a huge win for low-traffic sites or sites in the "just trying
>> > to deploy and get something live" phase. You can always optimize later by
>> > serving via nginx or cdn.
>> > We already have the views, api, and logic around for finding and serving
>> > the correct files.
>> > We can be just as efficient and secure as static/dj-static without needing
>> > to make people install and configure wsgi middleware to the application.
>> > We could have staticfiles classes implement more complicated features like
>> > giving cache recommendations, and serving pre-gzipped files.
>> >
>> > Is this a good idea? I realize it's not totally thought through. I'm fine
>> > with waiting until 1.9 if needed.
>>
>> I also think this is a good plan. It certainly makes sense to look at
>> "static" and "whitenoise" for ideas and compare their code to ours to
>> see where we could be more efficient or secure, but it's much less churn
>> for Django users if we simply improve our existing code rathe

Re: FK constraints are not checked at the end of nested atomic blocks

[Anssi Kääriäinen]

On Tuesday, December 22, 2015 at 1:20:23 AM UTC+2, Shai Berger wrote:
>
> > We could add a documented API to check wanted models for foreign key 
> > constraint violations (.check_constraints(Model1, Model2) would be a 
> > likely API). Then lets just document that depending on the database 
> > this check can be slow. On some databases we can get away with SET 
> > CONSTRAINTS IMMEDIATE 
> > [http://www.postgresql.org/docs/9.1/static/sql-set-constraints.html], 
> > which should be fast. 
>
> Then why don't we implement check_constraints() as 
> "SET CONSTRAINTS IMMEDIATE; SET CONSTRAINTS DEFERRED" on PG? That is 
> documented to only check outstanding changes in the transaction. 
>
> I am not sure about the performance implications on Oracle, but the Oracle 
> docs suggest "Making constraints immediate at the end of a transaction is 
> a 
> way of checking whether COMMIT can succeed."[1] 
>
> As far as I know, neither MySql nor Sqlite support deferrable constraints. 
>

Hmmh, if we only need to check deferred constraints, this can be made to 
work. I don't believe SET CONSTRAINTS is expensive at all, it does the same 
job the DB needs to do at commit time in any case.

Using SET CONSTRAINTS ALL IMMEDIATE; SET CONSTRAINTS ALL DEFERRED; is a bit 
problematic, as any constraint that was DEFERRABLE INITIALLY IMMEDIATE 
would then be set to deferred mode after the commands. This isn't a big 
problem in practice, unless we at some point allow creating DEFERRABLE 
INITIALLY IMMEDIATE constraints. We could also explicitly name all the 
constraints we want to defer, but I'm not sure if we actually reliably know 
the names of the constraints.

So, we could add a flag "check_deferred_constraints" to atomic(), and use 
the above SET CONSTRAINTS commands to achieve the wanted results. We will 
likely need to document that DEFERRABLE INITIALLY IMMEDIATE constraints 
will be set to deferred mode.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ecb654fc-42cc-4310-a849-a320e42bf0b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: FK constraints are not checked at the end of nested atomic blocks

[Anssi Kääriäinen]

The only generic way to check the constraints is to run through all
tables and their constraints. The constraint check itself is a check
that a matching row in a foreign table exists for each row of the
checked table. This means that the check could take hours on large
databases. In addition the check isn't safe against concurrency
issues. I'm afraid that if we have a flag for this for atomic(), then
users are going to think it is a good idea to run the check just in
case.

We could add a documented API to check wanted models for foreign key
constraint violations (.check_constraints(Model1, Model2) would be a
likely API). Then lets just document that depending on the database
this check can be slow. On some databases we can get away with SET
CONSTRAINTS IMMEDIATE
[http://www.postgresql.org/docs/9.1/static/sql-set-constraints.html],
which should be fast.

I do think DEFERRABLE INITIALLY IMMEDIATE constraints would make a lot
of sense for what Django is doing with the constraints, that is we
want to keep the constraints in check except in special cases.
Unfortunately I can't see an easy way to switch constraints to
INITIALLY IMMEDIATE mode due to backwards compat and migrations
issues.

 - Anssi


On Mon, Dec 21, 2015 at 4:41 PM, Shai Berger  wrote:
> While expensive operations may be unsuitable for general consumption, this
> kind of validation may be useful in special cases and offline jobs, notably
> in "data" migrations. So I think we should consider supporting them via an
> argument to atomic().
>
> Be that as it may, so long as we do not plan to change global behavior, we
> should fix the documentation. The current text and example code strongly
> imply that FKs are checked at the end of atomic blocks, even though they
> manage to avoid saying it explicitly.
>
> Shai.
>
> On 21 בדצמבר 2015 14:17:53 GMT+02:00, Aymeric Augustin
>  wrote:
>>
>> 2015-12-21 13:12 GMT+01:00 Anssi Kääriäinen :
>>>
>>> The check_constraints operation is extremely expensive, and not suitable
>>> for use at the end of savepoints.
>>
>>
>> Ah. Too bad!
>>
>> --
>> Aymeric.
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/F0A7E950-24FE-457C-877D-905EA3E9AA65%40platonix.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1FENw4gKKYhN2PS2ZFxisUAt_kzDV9wKD%2B9fxmQo9h3cg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: FK constraints are not checked at the end of nested atomic blocks

[Anssi Kääriäinen]

The check_constraints operation is extremely expensive, and not suitable
for use at the end of savepoints.

The reason why we use deferred constraints by default is mostly historical.
We do need deferred constraints for fixture loading, and applications might
rely on current implementation, but we could perhaps get away with
deferrable initially immediate constraints if the user chooses those.
Allowing users to choose constraint style requires relatively large amount
of work, so nobody has tackled that problem yet.

 - Anssi

On Monday, December 21, 2015, Артём Клименко  wrote:

> I opened this ticket https://code.djangoproject.com/ticket/25955
>
> The last proposal for the improvement
> https://code.djangoproject.com/ticket/25955#comment:5
> add parameter in OPTIONS
>
> DATABASES = {
> 'default': {
> 'ENGINE': 'django.db.backends.postgresql_psycopg2',
> # ...
> 'OPTIONS': {
> 'savepoint_check_constraints': True
> }
> },
> }
>
>
> https://code.djangoproject.com/attachment/ticket/25955/django_master_v2.patch
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/f1d474af-1191-4110-b664-2561d9477631%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GjzZWEQxvZAGZuaSsjzVYhV8eNVLAgY8eqa5Chmw9NEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Anssi Kääriäinen]

On Friday, December 18, 2015, Andrew Godwin  wrote:

>
>
> On Fri, Dec 18, 2015 at 3:44 PM, Mark Lavin  > wrote:
>
>> > You seem to be assuming I'm here to foist a brand new middle layer on
>> everyone; I'm not. I'm here to make one that fits neatly into Django, that
>> I think most people will want to turn on, and that provides a lot of value
>> in exchange for a slight round-trip performance hit - my goal is sub-5ms,
>> and preferably sub-3. If it starts being 10/20/30 milliseconds of cost,
>> then we'll have to change our approach until it's acceptable.
>>
>> Yes that's how I read this plan and that's why I think it needs some
>> clarity. I didn't mean for this to turn into a long discussion about
>> performance. This was meant to be a discussion about the transition plan.
>> To go back to my original message, I see no gain for existing WSGI
>> applications to have this on by default, even using the in-memory
>> channel, when they upgrade to 1.10 (or whenever this lands). The current
>> plan reads as though it will.
>>
>
> I agree - that was my original intention and how the current version of
> channels works, but it's no longer my plan, and I should update the
> integration plan to be more specific and discuss things like introducing
> different HttpRequest subclasses other than WSGIRequest.
>
>
My concern (and critisism) was about running everything through channels.

Back to the original question about release schedule. Is the 1.8 and 1.9
external package going to require changes to Django core? If so, are the
changes going to be substantial?

For 1.10, my vote goes to time based releases, that is, we don't decide at
this point what must be in 1.10.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1G-bC56UR-SKQfWujSxb%3DLe4WXedYj3bGRasfFkDWLn0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels integration plan, first draft

[Anssi Kääriäinen]

I have a gut feeling this isn't going to work that well. The reasons
include:
  - Backwards compatibility: how is a large site going to upgrade from 1.9
to 1.10?
  - Complexity of setup.
  - Error conditions: for example, what happens when an interface server
sends a request to worker, and then dies (that is, the response channel has
zero listeners). Similarly for chunked messages.
  - Does the architecture really scale enough? The channel backend is going
to be a bottleneck, it needs the ability to handle a huge amount of data
and a huge amount of individual messages. In particular, the request
channel is going to be contested. We know classic http scales, but is the
same true for interface server architecture?
  - Performance. Each request and response needs two additional network
roundtrips. One to save to the channel server, one to fetch from the
channel server. If the messages are large, this adds a lot of latency.
  - Untested architecture: does any big site use this kind of architecture
for all http handling?

A realistic test for this is to push a scalable amount of scalable sized
requests through the stack. The stack should recover even if you shut down
parts of the network, any single interface server, channel backend server
or worker server. Of course, when a server or a part of the network
recovers, the stack would need to recover from that. Compare the
performance, simplicity of setup and ability to recover from error
conditions to a setup with only classic Django http servers.

I'm sorry if this feels negative. But, you are planning to change the very
core of what a Django server is, and I feel we need to know with certainty
that the new architecture really works. And not only that, it needs to be
at least as good as classic http handling for existing users.

 - Anssi

On Thursday, December 17, 2015, Andrew Godwin  wrote:

> Yes, that is the idea. While it obviously adds overhead (a millisecond or
> two in my first tests), it also adds natural load balancing between workers
> and then lets us have the same architecture for websockets and normal HTTP.
>
> (The interface server does do all the HTTP parsing, so what gets sent over
> is slightly less verbose than normal HTTP and needs less work to use, but
> it's not a big saving)
>
> Andrew
>
> On Thu, Dec 17, 2015 at 9:01 PM, Anssi Kääriäinen  > wrote:
>
>> Is the idea a large site using classic request-response architecture
>> would get the requests at interface servers, these would then push the HTTP
>> requests through channels to worker processes, which process the message
>> and push the response through the channel backend back to the interface
>> server and from there back to the client?
>>
>>  - Anssi
>>
>> On Thursday, December 17, 2015, Andrew Godwin > > wrote:
>>
>>> To address the points so far:
>>>
>>>  - I'm not yet sure whether "traditional" WSGI mode would actually run
>>> through the in memory backend or just be plugged in directly to the
>>> existing code path; it really depends on how much code would need to be
>>> moved around in either case. I'm pretty keen on keeping a raw-WSGI path
>>> around for performance/compatability reasons, and so we can hard fail if
>>> you try *any* channels use (right now the failure mode for trying to use
>>> channels with the wsgi emulation is silent failure)
>>>
>>> - Streaming HTTP responses are already in the channels spec as chunked
>>> messages; you just keep sending response-style messages with a flag saying
>>> "there's more".
>>>
>>> - File uploads are more difficult, due to the nature of the worker model
>>> (you can't guarantee all the messages will go to the same worker). My
>>> current plan here is to revise the message spec to allow infinite size
>>> messages and make the channel backend handle chunking in the best way
>>> (write to shared disk, use lots of keys, etc), but if there are other
>>> suggestions I'm open. This would also let people return large http
>>> responses without having to worry about size limits.
>>>
>>> - Alternative serialisation formats will be looked into; it's up to the
>>> channel backend what to use, I just chose JSON as our previous research
>>> into this at work showed that it was actually the fastest overall due to
>>> the fact it has a pure C implementation, but that's a year or two old.
>>> Whatever is chosen needs large support and forwards compatability, however.
>>> The message format is deliberately specified as JSON-capable structures
>>> (dicts, lists, strings) as it's assumed any serialisation format 

Re: Channels integration plan, first draft

[Anssi Kääriäinen]

Is the idea a large site using classic request-response architecture would
get the requests at interface servers, these would then push the HTTP
requests through channels to worker processes, which process the message
and push the response through the channel backend back to the interface
server and from there back to the client?

 - Anssi

On Thursday, December 17, 2015, Andrew Godwin  wrote:

> To address the points so far:
>
>  - I'm not yet sure whether "traditional" WSGI mode would actually run
> through the in memory backend or just be plugged in directly to the
> existing code path; it really depends on how much code would need to be
> moved around in either case. I'm pretty keen on keeping a raw-WSGI path
> around for performance/compatability reasons, and so we can hard fail if
> you try *any* channels use (right now the failure mode for trying to use
> channels with the wsgi emulation is silent failure)
>
> - Streaming HTTP responses are already in the channels spec as chunked
> messages; you just keep sending response-style messages with a flag saying
> "there's more".
>
> - File uploads are more difficult, due to the nature of the worker model
> (you can't guarantee all the messages will go to the same worker). My
> current plan here is to revise the message spec to allow infinite size
> messages and make the channel backend handle chunking in the best way
> (write to shared disk, use lots of keys, etc), but if there are other
> suggestions I'm open. This would also let people return large http
> responses without having to worry about size limits.
>
> - Alternative serialisation formats will be looked into; it's up to the
> channel backend what to use, I just chose JSON as our previous research
> into this at work showed that it was actually the fastest overall due to
> the fact it has a pure C implementation, but that's a year or two old.
> Whatever is chosen needs large support and forwards compatability, however.
> The message format is deliberately specified as JSON-capable structures
> (dicts, lists, strings) as it's assumed any serialisation format can handle
> this, and so it can be portable across backends.
>
> - I thought SCRIPT_NAME was basically unused by anyone these days, but
> hey, happy to be proved wrong. Do we have any usage numbers on it to know
> if we'd need it for a new standalone server to implement? It's really not
> hard to add it into the request format, just thought it was one of those
> CGI remnants we might finally be able to kill.
>
> Andrew
>
> On Thu, Dec 17, 2015 at 6:32 PM, Anssi Kääriäinen  > wrote:
>
>> On Thursday, December 17, 2015, Carl Meyer > > wrote:
>>
>>> Hi Andrew,
>>
>>
>>> - I share Mark's concern about the performance (latency, specifically)
>>> implications for projects that want to keep deploying old-style, given
>>> all the new serialization that would now be in the request path. I think
>>> some further discussion of this, with real benchmark numbers to refer
>>> to, is a prerequisite to considering Channels as a candidate for Django
>>> 1.10. To take a parallel from Python, Guido has always said that he
>>> won't consider removing the GIL unless it can be done without penalizing
>>> single-threaded code. If you think a different approach makes sense here
>>> (that is, that it's OK to penalize the simple cases in order to
>>> facilitate the less-simple ones), can you explain your reasons for that
>>> position?
>>>
>>
>> We would also need some form of streamed messages for streamed http
>> responses.
>>
>> Is it possible to handle old-style http the way it has always been
>> handled?
>>
>>  - Anssi
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com
>> 
>> .
>> To post to this group, send email to django-developers@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CALMtK1Gz%3DaYMLyFW2da2C6Wo_-c_V2T_4p6K9eh0vwrKB91dKw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/django-developers/CALMtK1Gz%3DaYMLyFW2da2C6Wo_-c_V2T_4p6K9eh0vwrKB91dKw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/

Re: Channels integration plan, first draft

[Anssi Kääriäinen]

On Thursday, December 17, 2015, Carl Meyer  wrote:

> Hi Andrew,


> - I share Mark's concern about the performance (latency, specifically)
> implications for projects that want to keep deploying old-style, given
> all the new serialization that would now be in the request path. I think
> some further discussion of this, with real benchmark numbers to refer
> to, is a prerequisite to considering Channels as a candidate for Django
> 1.10. To take a parallel from Python, Guido has always said that he
> won't consider removing the GIL unless it can be done without penalizing
> single-threaded code. If you think a different approach makes sense here
> (that is, that it's OK to penalize the simple cases in order to
> facilitate the less-simple ones), can you explain your reasons for that
> position?
>

We would also need some form of streamed messages for streamed http
responses.

Is it possible to handle old-style http the way it has always been handled?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1Gz%3DaYMLyFW2da2C6Wo_-c_V2T_4p6K9eh0vwrKB91dKw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Annotation failure (Combining multiple aggregations)

[Anssi Kääriäinen]

On Tuesday, December 8, 2015 at 9:19:52 AM UTC+2, Anssi Kääriäinen wrote:
>
> On Friday, December 4, 2015 at 5:47:43 PM UTC+2, Paulo Maciel wrote:
>>
>> Combining multiple aggregations with annotate() will yield the wrong 
>> results <https://code.djangoproject.com/ticket/10060>, as multiple 
>> tables are cross joined. Due to the use of LEFT OUTER JOIN, duplicate 
>> records will be generated if some of the joined tables contain more records 
>> than the others
>>
>
> This has nothing to do with cross joins, usage of LEFT JOINS or having 
> more records in one table than the others. The reason is that we should use 
> subqueries where we currently use JOINs.
>
> As a complex example, (one) correct SQL query for 
> Book.objects.annotate(chapter_cnt=Count('chapter'), 
> author_cnt=Count('author')) is:
>
>   SELECT subq1.chapter_cnt, subq2.author_cnt, book_id
>FROM book
>LEFT JOIN (SELECT count(*) as chapter_cnt FROM chapter GROUP BY 
> book_id) as subq1 ON subq1.book_id = book.book_id
>LEFT JOIN (SELECT count(*) as author_cnt FROM author GROUP BY book_id) 
> as subq2 ON subq2.book_id = book.book_id
>GROUP BY book.book_id
>
> This is still an easy case for the ORM. To add a bit more complexity, 
> consider 
> Book.objects.filter(Q(chapter__pages__lt=100)|Q(author__age__gt=50)).annotate(chapter_cnt=Count('chapter'),
>  
> author_cnt=Count('author')). I *think* we need a query like:
>
>   SELECT subq1.chapter_cnt, subq2.author_cnt, book_id
>FROM book
>LEFT JOIN (SELECT count(*) as chapter_cnt FROM chapter WHERE 
> chapter.pages < 100 GROUP BY book_id) as subq1 ON subq1.book_id = 
> book.book_id
>LEFT JOIN (SELECT count(*) as author_cnt FROM author WHERE author.age > 
> 50 GROUP BY book_id) as subq2 ON subq2.book_id = book.book_id
>   WHERE subq1.book_id IS NOT NULL OR subq2.book_id IS NOT NULL
>GROUP BY book.book_id
>
> There are even more complex cases. For example you might need a subquery 
> inside the generated subqueries in some cases.
>
> I have some ideas of how to fix this (though in cases like this it is hard 
> to know if the ideas actually work before trying). I believe fixing this 
> takes at least a couple of months of development time. At least my 
> development budget seems to be around one day a month currently.
>
> An alternate approach is to error out when the ORM detects a complex 
> enough query, and add explicit ways for users to add subqueries, subselects 
> and stuff like that to ORM queries. This might make more sense than trying 
> to generate complex queries automatically, as there are multiple ways to 
> write the above query, and the different forms perform vastly differently 
> depending on the data in the tables and the used database backend. Still, 
> we can have it both ways - automatically generate a correct query, but also 
> offer a way to explicitly add subqueries to ORM queries.
>

I started writing some notes about the changes needed to handle 
automatically any .filter().annotate() combo. The notes are of very bad 
quality, but I'll publish them here even if that is the case. The point is 
that there is too much work, and the end result would be something where 
users don't understand why they get the queries they are getting. Still, 
the code would be complex enough to be extremely fragile.

I strongly believe the approach must be that we offer users explicit ways 
to modify the query (ability to join subqueries will get us a long way 
here), and that Django simply errors out when it sees query constructs it 
can't handle. We can later on maybe automate *some* of the error cases, but 
doing it for all cases seems extremely hard.

The notes assume we implement filter() for multivalued joins by doing 
subqueries instead of using joins. This is in practice needed for the 
approach to work at all.

When we do qs = Organization.objects.filter(subunits__size__gte=20)
  - add a new subquery condition to the query's where. Note that we added a 
subquery
condition for "subunits".
  - query at this point is:
  select * from ... where pk in (select parent_unit_id from 
organization where size >= 20)

Next, when we do qs = qs.annotate(sum=Sum('subunits__size'))
  - We find the existing where condition (with a subquery) for "subunits"
  - Remove the subquery condition from the where condition, decrease the 
refcount for leading joins to that subquery by 1
  - Add the sum to the subquery
  - Add the subquery as join to the query (*now* we can do the filter as a 
join. The
group by in the subquery enforces at maximum a single row per row in 
the outer query.).
The join type should be INNER, except in cases where the subquery 
condition w

Re: Decoupling the ORM

[Anssi Kääriäinen]

On Tuesday, December 15, 2015 at 5:43:55 PM UTC+2, Samuel Bishop wrote:
>
> Having worked through the code of several Django nosql/alternative 
> database backend libraries, forks, etc... 
>
> I've noticed that that one of the biggest challenges they run into, is 
> 'conforming' to many of the things Django expects these lowest layers to do.
>
> I opened this ticket https://code.djangoproject.com/ticket/25265 to begin 
> getting feedback on an initial idea for how to 'fix' the problem.
> Since then I've had further time to ponder the problem and while it still 
> seems to me that the best mechanism is to draw a line between the 'upper' 
> and 'lower' layers of Django, 
> I'm no longer 100% sure the correct place to enable this is the queryset 
> via an additional method, because I've realized that this is not just an 
> opportunity to get NoSQL databases into Django, but also an opportunity to 
> finally provide support for alternative Python ORMs, such as SQLAlchemy. 
>
> I've been digging around the code for this so I dont mind writing up the 
> code for this, but there is the big question of 'where to decouple' things. 
> Initial feedback in the thread 
> https://code.djangoproject.com/ticket/25265#comment:4 
> 
>  
> has raised the suggestion that moving one layer further up may be the right 
> place to go. It would be very helpful for me to get extra input from Django 
> developers familiar with the QuerySet and Query, before I start writing, so 
> I would love to hear feedback on the idea.
>

Assume the goal is perfect admin integration with a MongoDB backend. The 
approach can be either:
1) Use Django's standard models, create a QuerySet compatible 
MongoDBQuerySet.
2) Use completely different models, which respond to the APIs needed by 
Admin. This includes implementing a QuerySet compatible MongoDBQuerySet.

There is a lot more work to 2), but the benefit is that you get to use 
models actually meant to be used with a non-relational backend. For 
example, Django's User, Permission and Group models are implemented in a 
way that makes sense for a relational backend. If you use relational schema 
on non-relational database you are going to face big problems if you try to 
run the site with any non-trivial amount of data. For this reason I believe 
2) to be the right approach.

But, to get there, a QuerySet compatible MongoDBQuerySet is needed anyways. 
Here the choices are those mentioned in 
https://code.djangoproject.com/ticket/25265#comment:4 
.
 
That is, you can go with Django's QuerySet and Query, and just implement a 
MongoDBCompiler. Or, you can use QuerySet with MongoDBQuery class. Or, 
finally, you can implement MongoDBQuerySet directly from scratch.

If you implement Compiler or Query, you are targeting internal APIs which 
we *will* change in the future, maybe even in dramatic ways. If you target 
QuerySet, you are targeting a public API that doesn't change often. And, 
even if it changes, you will get a nice deprecation period for the changes.

It might seem a lot of work to start from QuerySet, but for a 
non-relational backend there isn't actually *that* much work involved. Most 
of Django's Query and Compiler classes deal with joins, SQL's NULL 
peculiarities or SQL's way of doing aggregations. All of these are 
non-issues for non-relational backends.

So, I think you should start with implementing a custom QuerySet for your 
wanted backend. You can also try to make it work with all Django models, 
but that approach is very likely to fail. For starters, Django's models use 
an autoincrementing integer field for primary key, whereas most (if not 
all) nonrelational databases use something different. Another interesting 
case is ManyToManyFields, which assumes a relational data model.

It is very tempting to go with an approach where you just implement a 
custom Compiler class for your nonrelational backend. This would, in 
theory, allow users to run any existing Django application on 
non-relational database by just using a non-relational backend. The problem 
with this approach is that it doesn't work well enough in practice, and the 
maintenance overhead in the long run is huge.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/

Re: Annotation failure (Combining multiple aggregations)

[Anssi Kääriäinen]

On Tuesday, December 8, 2015 at 5:28:08 PM UTC+2, Michael wrote:
>
> On Tuesday, December 8, 2015 at 1:26:52 AM UTC-6, Anssi Kääriäinen wrote:
>>
>>
>> Come to think of it, we already have a workaround:
>>Book.objects.annotate(author_cnt=RawSQL("(select count(*) from author 
>> where author.book_id = book.book_id)").
>>
>
> Yes, you can do a subquery to get the Sum or Count but then you cannot 
> "filter": `qs.filter(author_cnt__gt=0)` won't work.
>
> I think it would require the correct "GROUP BY" and "HAVING" clause to be 
> able to filter, correct?
>

This one actually does work with filter(). As the count is inside a 
subselect, it can be used directly in the WHERE clause without GROUP BY or 
HAVING. Using a direct count without a subselect wouldn't work.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f7516a98-5be5-49ba-8e40-1a49e95b9537%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Annotation failure (Combining multiple aggregations)

[Anssi Kääriäinen]

On Tuesday, December 8, 2015 at 2:52:07 PM UTC+2, Josh Smeaton wrote:
>
> Yeah I'm wary that a lot of the problems with extra() will be replicated 
> by users switching to RawSQL() instead. It won't handle realiasing or group 
> by correctly.
>

The group by case we can fix by creating a separate RawAggregate class, or 
by adding aggregate flag to RawSQL. The hardcoding of column and table 
names, and the realisasing problem can be solved with refsql style 
annotations: qs = Author.objects.annotate( casewhen=RefSQL("case when {
{name}} = %s then height else weight end", ('Anssi',), 
  output_field=models.IntegerField()) ).order_by('name')(from 
https://github.com/akaariai/django-refsql/blob/master/testproject/testproject/tests.py).

The idea is quite simply that instead of using hardcoded table.col 
references in the code, you use normal Django lookups, and refsql converts 
those to Col expressions.

But, whenever possible we should provide real expressions.

  - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5c5d7dd2-16fa-4495-8d3d-e6395b01681c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Annotation failure (Combining multiple aggregations)

[Anssi Kääriäinen]

On Tuesday, December 8, 2015 at 3:16:20 AM UTC+2, Michael wrote:
>
> I have the same problem.
>
> I tried to use `extra` but then I can't filter so it's not the best.
>
> Anybody has more info or a workaround?
>

Come to think of it, we already have a workaround:
   Book.objects.annotate(author_cnt=RawSQL("(select count(*) from author 
where author.book_id = book.book_id)").
The fact that you can do subselects with models.RawSQL is something that a 
lot of users seem to miss. There are a couple of reasons why one shouldn't 
use RawSQL too much, like the way it hard codes column names and table 
aliases.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a2a73387-3f90-4d21-bff8-f709694dd370%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Annotation failure (Combining multiple aggregations)

[Anssi Kääriäinen]

On Friday, December 4, 2015 at 5:47:43 PM UTC+2, Paulo Maciel wrote:
>
> Combining multiple aggregations with annotate() will yield the wrong 
> results , as multiple tables 
> are cross joined. Due to the use of LEFT OUTER JOIN, duplicate records 
> will be generated if some of the joined tables contain more records than 
> the others
>

This has nothing to do with cross joins, usage of LEFT JOINS or having more 
records in one table than the others. The reason is that we should use 
subqueries where we currently use JOINs.

As a complex example, (one) correct SQL query for 
Book.objects.annotate(chapter_cnt=Count('chapter'), 
author_cnt=Count('author')) is:

  SELECT subq1.chapter_cnt, subq2.author_cnt, book_id
   FROM book
   LEFT JOIN (SELECT count(*) as chapter_cnt FROM chapter GROUP BY book_id) 
as subq1 ON subq1.book_id = book.book_id
   LEFT JOIN (SELECT count(*) as author_cnt FROM author GROUP BY book_id) 
as subq2 ON subq2.book_id = book.book_id
   GROUP BY book.book_id

This is still an easy case for the ORM. To add a bit more complexity, 
consider 
Book.objects.filter(Q(chapter__pages__lt=100)|Q(author__age__gt=50)).annotate(chapter_cnt=Count('chapter'),
 
author_cnt=Count('author')). I *think* we need a query like:

  SELECT subq1.chapter_cnt, subq2.author_cnt, book_id
   FROM book
   LEFT JOIN (SELECT count(*) as chapter_cnt FROM chapter WHERE 
chapter.pages < 100 GROUP BY book_id) as subq1 ON subq1.book_id = 
book.book_id
   LEFT JOIN (SELECT count(*) as author_cnt FROM author WHERE author.age > 
50 GROUP BY book_id) as subq2 ON subq2.book_id = book.book_id
  WHERE subq1.book_id IS NOT NULL OR subq2.book_id IS NOT NULL
   GROUP BY book.book_id

There are even more complex cases. For example you might need a subquery 
inside the generated subqueries in some cases.

I have some ideas of how to fix this (though in cases like this it is hard 
to know if the ideas actually work before trying). I believe fixing this 
takes at least a couple of months of development time. At least my 
development budget seems to be around one day a month currently.

An alternate approach is to error out when the ORM detects a complex enough 
query, and add explicit ways for users to add subqueries, subselects and 
stuff like that to ORM queries. This might make more sense than trying to 
generate complex queries automatically, as there are multiple ways to write 
the above query, and the different forms perform vastly differently 
depending on the data in the tables and the used database backend. Still, 
we can have it both ways - automatically generate a correct query, but also 
offer a way to explicitly add subqueries to ORM queries.

 - Anssi

>
>
> The Count 
> 
>  aggregate 
> has a distinct parameter that may help:
>
> q = Book.objects.annotate(Count('authors', distinct=True), Count('chapters', 
> distinct=True))
>
> Why not "distinct=True" to use in Sum?
>
> q = Book.objects.annotate(Sum('val_a', distinct=True), Sum('val_b', 
> distinct=True))
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e32741dc-097e-4fba-b6ed-e831dc545c9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Backporting ticket 25548 into 1.9.x

[Anssi Kääriäinen]

On Mon, Dec 7, 2015 at 3:20 PM, Tim Graham  wrote:
> I've done the backport. I think the only reason it wasn't done initially is
> that there's no indication on the ticket that the issue was a regression.

For some reason we don't mention backporting of regression fixes in
https://docs.djangoproject.com/en/1.9/internals/release-process/.
Should we add an explicit bullet point for that?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1H4a1U6%2Brmms_1Wdosgz99BN70RLn8n0U1qcf0Nc%2BMv6Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Non-atomic migrations in Django?

[Anssi Kääriäinen]

I don't see any problem with optional non-transactional migrations. So, +1
for the idea and API. I haven't looked at the implementation, so no
comments about that.

 - Anssi

On Tuesday, November 24, 2015, Ludwig Hähne  wrote:

> Hi all,
>
> I'd like to get your feedback on supporting non-atomic migrations in
> Django.
>
> Database transactions are always wrapped in transactions (on DBs that
> support transactional DDL like Postgres). Generally this is a good idea.
>
> However, one can't do batched updates in data migrations which is
> essential for performing changes on big tables on a live DB [1]. It's also
> not possible to create indexes concurrently on Postgres from inside a
> transaction.
>
> Therefore, I'd like to have support for non-atomic migrations in Django
> because it's pretty messy to work around not having proper support for that
> [2].
>
> Here's a proof-of-concept implementation [3] of exempting specific
> migrations from being wrapped in a transaction by setting `atomic = False`
> on the migration:
>
>
> https://github.com/django/django/compare/master...Ableton:non-atomic-migrations
>
> Do you agree that non-atomic migrations should be supported by Django?
>
> Is setting an `atomic` property on a migration a good API for that?
>
> If there's a chance this will be merged, I'd add documentation,
> incorporate your feedback, and open a ticket or PR.
>
> Thanks,
> Ludwig
>
> [1] http://pankrat.github.io/2015/django-migrations-without-downtimes/
> [2]
> http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration
> [3]
> https://github.com/django/django/compare/master...Ableton:non-atomic-migrations
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GDj0_wheW5dHEdp1ZLYcXa-6OpxfHgcnHqWtMcT9YG0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: __ne, #5763

[Anssi Kääriäinen]

On Saturday, November 21, 2015, Carsten Fuchs  wrote:

> Hi Anssi,
>
> Am 2015-11-21 um 12:50 schrieb Anssi Kääriäinen:
>
>> In summary, the imaginary query of comment 14
>>
>>  Blog.objects.filter(entry__tag__name='django',
>> entry__author_count__ne=2)
>>
>>
>> This isn't a real query. There isn't a field author_count, the query
>> needs an annotation somewhere. So, I don't think this argument is
>> convincing without the annotation (note that the place of the annotation
>> matters). In addition, providing example data and the expected results
>> would be great, too.
>>
>
> Well, yes, this is a fictional example, but if you replace author_count by
> some valid field that doesn't require an annotation, e.g. author__name, the
> (imaginary) query should be valid (for the purpose of demonstrating the
> problem)?


The author_count name suggested this was an aggregation. If this is just a
regular field, then things are a bit simpler. Note that negated Q-object +
filter and exclude() queries are the same thing.

>
> The key issue is really this, quoted from the linked Django documentation:
>
> "To handle both of these situations, Django has a consistent way of
>> processing
>> filter() calls. Everything inside a single filter() call is applied
>> simultaneously to filter out items matching all those requirements.
>> Successive
>> filter() calls further restrict the set of objects, but for multi-valued
>> relations, they apply to any object linked to the primary model, not
>> necessarily those objects that were selected by an earlier filter() call."
>>
>
> That is, sometimes we *have* to put several filters into a single filter()
> call to obtain the desired set. If such a situation requires a negation,
> exclude() cannot help, because "[...], they apply to *any* object linked to
> the primary model, not necessarily those objects that were selected by an
> earlier filter() call".
>
>
There is a fix for exactly this issue in pr
https://github.com/django/django/pull/4385. After the pr, you could just
use .filter(Q(entry__tags__name='django')&~Q(entry__author_count=2)).

The discussion seems to miss a real definition of what exactly the ne
>> lookup should do. There are two ways to implement ne, one is as a
>> complement of exact, another is as the != operator. In SQL the first one
>> is "col != val OR col IS NULL", the latter one is just "col != val".
>>
>
> Thanks for pointing this out, I wasn't aware of this (in this context)
> before. It seems to be another facet in the overall problem, but
> independent from the above, isn't it? (In my normal, "non-negated" queries,
> where required I account for NULLs explicitly all the time...)


Yeah. We should correct negated filtering to work as documented when
combined with other conditions.

So, to fix the issue in comment 14 of the ticket, the above mentioned PR is
the right fix. If you want the SQL != operator, then we need a new lookup.

 - Anssi


> Best regards,
> Carsten
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/56506174.4050609%40cafu.de
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1G3N-8zcXajSFnwuc3_THi1v5h%3DHxbivbGEV7dN_zxVjw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: __ne, #5763

[Anssi Kääriäinen]

On Saturday, November 21, 2015, Carsten Fuchs  wrote:

> Hi Marten,
>
> Am 2015-11-21 um 11:53 schrieb Marten Kenbeek:
>
>> The 'con' side argument is that it would create in inconsistency in the
>> API, since we don't have any other negated lookups either. If we can get
>> the same behaviour by fixing the current API, Django should not
>> introduce an unnecessary consistency.
>>
>
> I think that this is where the communication problem is:
>
> The "con" side seems to see a problem in the current API regarding
> exclude() (personally, about what it is I only have a vague idea from older
> linked discussions: NOT(...) vs. "proper" negation) and argues that fixing
> the problem solves the issue and avoids inconsistency in the API brought by
> the introduction of __ne.
>
> The "pro" side sees a problem in functionality when __ne is lacking, and
> cannot see how the "con" side's arguments would help. The problem in
> functionality is detailed in
> https://code.djangoproject.com/ticket/5763#comment:14, which essentially
> quotes (possibly an older version of)
> https://docs.djangoproject.com/en/1.8/topics/db/queries/#spanning-multi-valued-relationships
> and shows a related example where exclude() cannot replace __ne. As we/I
> cannot see how fixing/changing exclude() would help in this situation, we
> keep asking about __ne.  ;-)
>
> In summary, the imaginary query of comment 14
>
> Blog.objects.filter(entry__tag__name='django',
> entry__author_count__ne=2)


This isn't a real query. There isn't a field author_count, the query needs
an annotation somewhere. So, I don't think this argument is convincing
without the annotation (note that the place of the annotation matters). In
addition, providing example data and the expected results would be great,
too.

The discussion seems to miss a real definition of what exactly the ne
lookup should do. There are two ways to implement ne, one is as a
complement of exact, another is as the != operator. In SQL the first one is
"col != val OR col IS NULL", the latter one is just "col != val".

The complement is already achieved with .exclude(col__exact=val), so we
don't need a duplicate API for that. But we don't have any way of doing the
!= operator alone for nullable fields, and I don't see a fundamental reason
why we couldn't add it. The biggest problem is that the new __ne lookup
isn't actually a complement of __exact, and that could be confusing. But
this is the case for lt and gte, too.

The underlying problem here is SQL's three-valued boolean logic.
Unsuprisingly, this feature of SQL causes a lot of confusion.

- Anssi


> seems quasi not be possible to implement in today's Django (but probably
> with your suggestion below).
>
> Anyway, with the new lookup API, it has become trivial for any project
>> to implement the __ne lookup. It is the first example in the how-to
>> guide:
>> https://docs.djangoproject.com/en/dev/howto/custom-lookups/#a-simple-lookup-example
>> .
>>
>
> Thanks, I'll definitively check and try this out, I didn't know it before!
>
> Best regards,
> Carsten
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/565054EA.40508%40cafu.de
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1HgO7dXgKH8QoKrCcRzUa-t7%2Be6WPdPUOk4hSn_eYNb6Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Question] Many-To-Many query where only pk is returned

[Anssi Kääriäinen]

We din't currently do join trimming from the beginning of the query's
tables, only from the end of the tables.

Fixing this issue only doesn't seem worth it, as the current join trimmig
code can't be adapted to handle trimming from the beginning of the query's
tables. But we might be able to trim any non-necessary table from the query
in the compiler stage. If we were able to do that, then there might be a
point in implementing that. I know we sometimes include non-necessary
tables in the query for example in multitable inheritance scenarios.

 - Anssi

On Thursday, November 19, 2015, Cristiano Coelho 
wrote:

> You are right. I believe an optimization like this would probably help
> just a few people, as only fetching data from the intermediary table is a
> rare thing. But if it is an easy change, which improves performance, why
> not? However I think the change is quite complicated.
>
> El jueves, 19 de noviembre de 2015, 11:51:51 (UTC-3), charettes escribió:
>>
>> Hi Cristiano,
>>
>> If I get it correctly you'd like m2m querying to start with the
>> intermediary (FROM) table and JOIN the referenced one only if more fields
>> than the primary key are selected.
>>
>> class Book(models.Model):
>> name = models.CharField(max_length=100)
>>
>> class Author(models.Model):
>> books = models.ManyToMany(Book)
>>
>> author = Author.objects.get(pk=1)
>> author.books.values_list('pk')
>>
>> Would result in the following query:
>> SELECT book_id FROM author_books WHERE author_id = 1;
>>
>> Instead of:
>> SELECT id FROM book JOIN author_books ON (book.id =
>> author_books.book_id) WHERE author_id = 1;
>>
>> I think this is a sensible optimization but I wonder about its
>> feasibility. It looks like the `pk` reference would require some special
>> handling to reference `book_id` since it's not actually a primary key on
>> the intermediate table.
>>
>> Simon
>>
>> Le mercredi 18 novembre 2015 19:41:22 UTC-5, Cristiano Coelho a écrit :
>>>
>>> Hello there,
>>>
>>> Lets say I have these two models (sorry about the spanish names!) (
>>> Django 1.8.6 and MySQL backend )
>>>
>>> class Especialidad(models.Model):
>>> nombre = models.CharField(max_length=250, blank=False, unique=True)
>>>
>>>
>>>
>>> class Usuario(AbstractBaseUser):
>>> permisosEspecialidad = models.ManyToManyField("Especialidad", 
>>> blank=True)
>>>
>>> Let u be some Usuario instance, and the following query:
>>>
>>> u.permisosEspecialidad.all().values_list('pk',flat=True)
>>>
>>> The actual printed query is:
>>>
>>> SELECT `samiBackend_especialidad`.`id`
>>> FROM `samiBackend_especialidad`
>>> INNER JOIN `samiBackend_usuario_permisosEspecialidad` ON ( 
>>> `samiBackend_especialidad`.`id` = 
>>> `samiBackend_usuario_permisosEspecialidad`.`especialidad_id` )
>>> WHERE `samiBackend_usuario_permisosEspecialidad`.`usuario_id` = 8
>>>
>>> As my understanding, since I'm only selecting the id field which is already 
>>> present in the intermediary table (and is also a FK), the actual join is 
>>> redundant, as I have all the info I need in this case.
>>>
>>> So the query could work like this
>>>
>>> SELECT `samiBackend_usuario_permisosEspecialidad`.`especialidad_id`
>>> FROM  `samiBackend_usuario_permisosEspecialidad`
>>> WHERE `samiBackend_usuario_permisosEspecialidad`.`usuario_id` = 8
>>>
>>>
>>> I guess this works this way because this particular case might be hard to 
>>> detect or won't be compatible with any additional query building, however, 
>>> for ForeignKey relations, this optimization is already done (If you select 
>>> the primary key from the selected model only, it wont add a join)
>>>
>>> What would be the complications to implement this? Would it worth the 
>>> effort?
>>>
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/f96c5d2f-3655-461f-8534-de8622cd284d%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://grou

Re: [Feature Request] Performant values_list query

[Anssi Kääriäinen]

I don't think anybody has anything against making values() or
values_list() queries faster. The question is purely about
implementation. If you can offer something that is faster than the
current code without introducing backwards incompatibilities or making
the code a lot more complex, then such a change has a good possibility
to be merged.

You can likely find some conditions when you can fast-path the
results_iter() handling for values_list() querysets. It is certain
that you can't do this in all cases.

If you want to proceed with the implementation, the best way forward
is to just run the full Django test suite against your changes. It
will point out a couple of cases where you'll see why certain things
are done (from_db_value() for example).

Of course, you should work against Django's master branch.

 - Anssi

On Mon, Nov 16, 2015 at 3:02 PM, Cristiano Coelho
 wrote:
> I would like to add, that minor change on a critical query meant an increase 
> from 37 to 47 requests per second on a benchmark test with apache, that's a 
> 22% improvement that will be bigger on larger results (this was just with 700 
> rows result set), compared to using some pagination with a limit of 100 that 
> raises the requests per second value to almost 200, there's a clear 
> bottleneck in there.
>
> Looking at the code of values list queryset, maybe is it possible to replace 
> the call of results_iter to directly use execute_sql(MULTI) avoiding all the 
> unnecessary ifs and loops that results_iter does that is mainly useful for 
> other query set objects, but not for values list. I did a test with the above 
> and values list queries were performing as good as using directly a django 
> cursor, but not as good as using as_sql() and then the cursor, but very close.
>
> What would be the issues of adding such and optimization to values list? I'm 
> even thinking of manually monkey patching it so all my values list queries 
> are improved in my project, since I use values list most of the time due to 
> performance reasons since building objects on large results is extremely 
> slow, blame on python sadly.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/7729d524-1449-48a4-92b2-72b866c19e7c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1EBv1e%2BDnoWk5qRyKKbwHhaHPD%3D%2BtaTt-HCgptuTEhNBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Testing Django using Docker

[Anssi Kääriäinen]

There would be huge benefit if we could somewhat easily test against
different database and python versions. Right now, if you are faced with
"doesn't work on MySQL 5.7" bug, it is a bit complicated to verify such
issues if you don't happen to have the version installed.

- Anssi
On Nov 9, 2015 12:22, "Enrique Paredes"  wrote:

> There's one good basic example of docker-compose in pydanny's cookiecutter
> django package, if you need it.
>
> —
> Sent from Mailbox 
>
>
> On Mon, Nov 9, 2015 at 9:04 AM, Jani Tiainen  wrote:
>
>> Hi,
>>
>> I think you should document this in Wiki at least.
>>
>> Few additional questions notes:
>>
>> What's the Docker overhead? (starup/shutdown time)
>>
>> Could waiting for db container be more robust? Now it's just "random
>> number".
>>
>> Using docker-compose would be nice to avoid problems with container
>> startup times.
>>
>>
>> On 07.11.2015 14:46, Andreas Madsack wrote:
>>
>> Hello,
>>
>> I did a first version of a Dockerfile for testing using Docker with a
>> Postgresql Docker container.
>> see https://github.com/mfa/django-runtests-docker
>>
>> The readme contains a walkthrough for a fresh digitialocean box (you need
>> 1GB RAM! 512MB and a swap file also works).
>> On a linux host you can run the testsuite with "sh run.sh".
>>
>> Should this way of running the tests be integrated in the docs?
>> And should the Dockerfile and the runscript be in the django tests folder?
>>
>> Regards,
>> Andreas
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> 
>> https://groups.google.com/d/msgid/django-developers/5f4750dd-36c6-4d80-b222-6d203ada2974%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/56405370.8060100%40gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1447064529975.9bd091b8%40Nodemailer
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1EFjAEcaz-YkepgbzuXk4RNaHiupGUOBv8A1K0RidCNhA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django ORM query syntax enhancement

[Anssi Kääriäinen]

I did similar work in https://github.com/django/django/pull/4801. I
aimed for 1.9, but had to defer the work due to problems with
.exclude() filtering.

The problem is that if you query with
.exclude(LessThan(F('friends__age'), 30)), you'll get different
results than with .exclude(friends__age__lt=30). The latter does a
subquery while the expression version isn't capable of that.
Unfortunately, fixing this is a bit nasty. I don't see any other way
than some solution like https://github.com/django/django/pull/4385 -
that is we prewalk the expression tree and decide which parts need to
be pushed in to a subquery. This should work really nice with
expressions, too.

I haven't taken a closer look on your approach, so I can't say which
one we should use as basis for future work once we have solved the
subquery problem.

 - Anssi

On Mon, Oct 19, 2015 at 2:09 PM, Josh Smeaton  wrote:
> I think you forgot the PR link Alexey. I'd like to have a look at the
> changes you've made. I'm not sure about the extensions to F() just yet, I'm
> hoping they aren't actually needed for your patch to work, but will need to
> review to be sure.
>
> Cheers
>
>
> On Monday, 19 October 2015 21:49:37 UTC+11, Alexey Zankevich wrote:
>>
>> Hi all,
>>
>> Here is a raw pull request, allowing lookup instances passing to filter
>> method
>> There are two new test cases working:
>>
>> 1. Passing lookup with F object:
>> Article.objects.filter(GreaterThan(F('id'), Value(5)))
>> 2. And passing lookup with transforms:
>> Article.objects.filter(Contains(Upper(Lower(Upper(F('headline', 'ARTICLE
>> 5'))
>>
>> All the existing tests are passing successfully, by the way (at least for
>> SQLite backend).
>>
>> Several comments:
>>
>> F expressions became real expressions (Expression subclass).
>> Expression got a new method - get_source_f_object, which walks through
>> children classes and find if it has F object passed. In that case
>> output_field-related code should be postponed, until compiler resolves it
>> (which cannot be done during lookup init time).
>> some code related to query.build_filter method (responsible to
>> lookups/transform parsing, and calculating joins), extracted into separate
>> helper classes - QueryKeywordLookupHelper and QueryObjectLookupHelper. They
>> have common code base, but some of methods work in a different way.
>>
>> Except review, it would be really great if someone help with writing test
>> cases (and I'll get them working in case of any issues). Also, there is a
>> workaround field=None -> isnull=True for oracle database, can anybody check
>> if it is working correctly?
>>
>> Regards,
>> Alexey
>>
>>
>> On Thursday, October 1, 2015 at 11:38:39 AM UTC+3, Anssi Kääriäinen wrote:
>>>
>>> Lets concentrate on making expressions in filter calls and lookups as
>>> expressions reality. When we have done that we can try out different
>>> syntax approaches in 3rd party apps. Finally, after we have field
>>> tested the approaches, we can merge something in to Django.
>>>
>>>  - Anssi
>>>
>>> On Thu, Oct 1, 2015 at 10:49 AM, Alexey Zankevich
>>>  wrote:
>>> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for
>>> >> core.
>>> >
>>> > When Django supports expression lookups, it will be possible to build
>>> > syntax
>>> > sugar third-party library on top of it.
>>> > For example, it will be possible to implement dotted syntax for F
>>> > objects:
>>> >
>>> > UserModel.objects.filter(Decode(F.user.name, 'utf-8') == u'Бармаглот')
>>> >
>>> > alternatively it can be even combined with your syntax
>>> >
>>> > UserModel.objects.filter(F.user.name.decode('utf-8') == u'Бармаглот')
>>> >
>>> > However, the second solution will be more complicated from technical
>>> > point
>>> > of view (as will require lazy object, collecting all the func or
>>> > transform
>>> > calls), also it will require to be really careful with transform names
>>> > not
>>> > to conflict with possible field names.
>>> > For example, it's hardly possible to have "decode" field in the model,
>>> > but
>>> > easily - "date".
>>> >
>>> >
>>> > On Thursday, October 1, 2015 at 10:18

Re: Django ORM query syntax enhancement

[Anssi Kääriäinen]

Lets concentrate on making expressions in filter calls and lookups as
expressions reality. When we have done that we can try out different
syntax approaches in 3rd party apps. Finally, after we have field
tested the approaches, we can merge something in to Django.

 - Anssi

On Thu, Oct 1, 2015 at 10:49 AM, Alexey Zankevich
 wrote:
>> I think the `F('name').decode('utf-8')` syntax is a good candidate for
>> core.
>
> When Django supports expression lookups, it will be possible to build syntax
> sugar third-party library on top of it.
> For example, it will be possible to implement dotted syntax for F objects:
>
> UserModel.objects.filter(Decode(F.user.name, 'utf-8') == u'Бармаглот')
>
> alternatively it can be even combined with your syntax
>
> UserModel.objects.filter(F.user.name.decode('utf-8') == u'Бармаглот')
>
> However, the second solution will be more complicated from technical point
> of view (as will require lazy object, collecting all the func or transform
> calls), also it will require to be really careful with transform names not
> to conflict with possible field names.
> For example, it's hardly possible to have "decode" field in the model, but
> easily - "date".
>
>
> On Thursday, October 1, 2015 at 10:18:40 AM UTC+3, Loïc Bistuer wrote:
>>
>> > On Oct 1, 2015, at 13:38, Anssi Kääriäinen  wrote:
>> >
>> > +1 to not requiring all transforms to handle __underscore__ syntax.
>>
>> +1
>>
>> > I think what we want to do is allow users choose which syntax they
>> > prefer. The idea is that Django will support both JSONExtract('data',
>> > path=['owner', 'other_pets', 0, 'name']) and
>> > data__owner__other_pets__0__name out of the box.
>> >
>> > We will also make it possible to support callable transforms. For
>> > example:
>> >filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))
>> > is equivalent to
>> >filter(F('name').decode('utf-8') == Value(u'Бармаглот'))
>> >
>> > The callable transforms syntax will not be a part of Django, but it
>> > will be possible to create an extension for this (it is actually
>> > surprisingly easy to do once we have support for expressions in
>> > filter).
>>
>> I'm pretty convinced we need a better API / sugar as part of core in the
>> long run.
>>
>> The `filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))`
>> syntax is not pythonic and very hard to decipher, and we've reached the
>> limits of our historical double underscore syntax (doesn't support multiple
>> arguments, limited to ascii, etc.).
>>
>> I think the `F('name').decode('utf-8')` syntax is a good candidate for
>> core:
>>
>> - It's a small extension to the existing F objects, so it's easy to grasp
>> for existing Django users.
>> - It's just a thin sugar on top of the canonical API.
>> - All the required machinery being already in place (lookup & transform
>> registration) the change to Django is minimal.
>>
>> > - Anssi
>> >
>> > On Thu, Oct 1, 2015 at 4:00 AM, Josh Smeaton 
>> > wrote:
>> >> No, not all Lookups or Transforms are required to handle __underscore__
>> >> syntax. The entire point of supporting object based lookups is to
>> >> handle
>> >> cases that get more complex than a single argument transform or a left
>> >> and
>> >> right hand side lookup.
>> >>
>> >> In particular, I think your Decode(utf8) example is a good one. It
>> >> shows
>> >> that you could maybe shoehorn multiple arg transforms into the
>> >> __underscore__ api, but it wouldn't be entirely obvious how to do so.
>> >> You'd
>> >> probably need to register partial transformations for each encoding you
>> >> wanted to support. The contrib.postgres module has (from memory)
>> >> KeyTransform classes that do a similar thing.
>> >>
>> >> Cheers
>> >>
>> >>
>> >> On Thursday, 1 October 2015 02:13:41 UTC+10, Alexey Zankevich wrote:
>> >>>
>> >>> I'll try to turn lookups into expression (will use master branch).
>> >>> I also have a question. Meanwhile the old query syntax with
>> >>> underscores is
>> >>> pretty go

Re: Django ORM query syntax enhancement

[Anssi Kääriäinen]

+1 to not requiring all transforms to handle __underscore__ syntax.

I think what we want to do is allow users choose which syntax they
prefer. The idea is that Django will support both JSONExtract('data',
path=['owner', 'other_pets', 0, 'name']) and
data__owner__other_pets__0__name out of the box.

We will also make it possible to support callable transforms. For example:
filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))
is equivalent to
filter(F('name').decode('utf-8') == Value(u'Бармаглот'))

The callable transforms syntax will not be a part of Django, but it
will be possible to create an extension for this (it is actually
surprisingly easy to do once we have support for expressions in
filter).

 - Anssi

On Thu, Oct 1, 2015 at 4:00 AM, Josh Smeaton  wrote:
> No, not all Lookups or Transforms are required to handle __underscore__
> syntax. The entire point of supporting object based lookups is to handle
> cases that get more complex than a single argument transform or a left and
> right hand side lookup.
>
> In particular, I think your Decode(utf8) example is a good one. It shows
> that you could maybe shoehorn multiple arg transforms into the
> __underscore__ api, but it wouldn't be entirely obvious how to do so. You'd
> probably need to register partial transformations for each encoding you
> wanted to support. The contrib.postgres module has (from memory)
> KeyTransform classes that do a similar thing.
>
> Cheers
>
>
> On Thursday, 1 October 2015 02:13:41 UTC+10, Alexey Zankevich wrote:
>>
>> I'll try to turn lookups into expression (will use master branch).
>> I also have a question. Meanwhile the old query syntax with underscores is
>> pretty good for simple queries (like Model.objects.filter(name='Bob'), it
>> gets really ugly for parametrized calls, a new JSONField is a good example
>> of that:
>>
>> >>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
>> []
>>
>> It will get even more messy if we want to pass a string as a second param
>> of the func.
>>
>> ex.:
>>
>> 1. filter(Decode(F('name'), 'utf-8'), Value(u'Бармаглот'))   # <- neat
>> 2. filter(name__decode__utf8=u'Бармаглот')  # <- ?? ambiguous and not nice
>> at all
>>
>> So question - is it implied all the funcs, transforms and lookups to have
>> underscore-based equivalent? It can affect the final implementation pretty
>> much. In my opinion underscore-based equivalent should not be really
>> required for funcs (the problem doesn't seem to affect transforms as they
>> will not accept multiple params according to another thread).
>>
>> Thanks,
>> Alexey
>>
>>
>> On Wednesday, September 30, 2015 at 9:19:51 AM UTC+3, Anssi Kääriäinen
>> wrote:
>>>
>>> I don't think we need split-to-subq support for Lookups before we make
>>> them expressions. Lookups as expressions are usable outside .filter(),
>>> and we need the split-to-subq support only in .filter(expression).
>>>
>>>  - Anssi
>>>
>>> On Wed, Sep 30, 2015 at 8:46 AM, Josh Smeaton 
>>> wrote:
>>> > I'm mixing my versions, sorry to those following along. 1.9 has just
>>> > reached
>>> > alpha. Lookups as Expressions should be doable for 1.10 which master is
>>> > currently tracking.
>>> >
>>> > Cheers
>>> >
>>> >
>>> > On Wednesday, 30 September 2015 15:31:24 UTC+10, Josh Smeaton wrote:
>>> >>
>>> >> The alpha for 1.10 has already been cut, and I'm not sure that the
>>> >> kinds
>>> >> of changes needed here are appropriate to add now that the alpha is
>>> >> out. One
>>> >> could *maybe* make the argument that changing Lookup to an Expression
>>> >> now
>>> >> rather than later is the right move considering Transforms just
>>> >> underwent
>>> >> the same change for 1.10. Personally though, I don't think I have the
>>> >> time
>>> >> right now to do this change. I would support you if you were able, but
>>> >> we'd
>>> >> still be at the mercy of the technical board (I assume) for getting
>>> >> this
>>> >> change in for 1.10.
>>> >>
>>> >> Do you think Lookup as Expressions requires the subquery/exclude fix
>>> >> you
>>> &

Re: Django ORM query syntax enhancement

[Anssi Kääriäinen]

I don't think we need split-to-subq support for Lookups before we make
them expressions. Lookups as expressions are usable outside .filter(),
and we need the split-to-subq support only in .filter(expression).

 - Anssi

On Wed, Sep 30, 2015 at 8:46 AM, Josh Smeaton  wrote:
> I'm mixing my versions, sorry to those following along. 1.9 has just reached
> alpha. Lookups as Expressions should be doable for 1.10 which master is
> currently tracking.
>
> Cheers
>
>
> On Wednesday, 30 September 2015 15:31:24 UTC+10, Josh Smeaton wrote:
>>
>> The alpha for 1.10 has already been cut, and I'm not sure that the kinds
>> of changes needed here are appropriate to add now that the alpha is out. One
>> could *maybe* make the argument that changing Lookup to an Expression now
>> rather than later is the right move considering Transforms just underwent
>> the same change for 1.10. Personally though, I don't think I have the time
>> right now to do this change. I would support you if you were able, but we'd
>> still be at the mercy of the technical board (I assume) for getting this
>> change in for 1.10.
>>
>> Do you think Lookup as Expressions requires the subquery/exclude fix you
>> mention above? I would think not -- not until we were ready to document and
>> support .filter(Lookup(F(), Value()). If it wasn't a requirement, it'd make
>> the Lookup->Expression work much easier. It wouldn't even need to be
>> documented (other than the release notes), as it'd just be an implementation
>> change.
>>
>> Cheers,
>>
>> On Wednesday, 30 September 2015 14:33:14 UTC+10, Anssi Kääriäinen wrote:
>>>
>>> On the core ORM side we need to make
>>> .exclude(LessThan(F('friends__age'), 30)) do a subquery.  This way
>>> .exclude(friends__age__lt=30) does the same thing as the expression
>>> version. This isn't that easy to do. If we just use
>>> resolve_expression, then the friends relation will generate a join,
>>> and then as second step do a negated filter on the joined value.
>>> Instead we want to detect that the LessThan expression needs to be
>>> pushed in to a subquery.
>>>
>>> So, we need to solve:
>>>
>>>  A) A way to ask an expression if it is referencing a multijoin
>>> (possible approach is to just have a method
>>> "refs_multi_valued_relation(query)")
>>>  B) When the ORM sees an expression that is reffing a multijoin in an
>>> exclude filter, then we need to push the expression in to a subquery.
>>>
>>> A) requires some new work. This shouldn't be that hard to implement,
>>> we just recursively ask subexpressions if they reference a multijoin.
>>>
>>> Something like https://github.com/django/django/pull/4385 will make B)
>>> much easier to implement.
>>>
>>> I've been working on making Q-objects responsible for resolving
>>> themselves. See https://github.com/django/django/pull/4801. This
>>> should solve 3).
>>>
>>> We don't seem to be missing any major parts. I (or some volunteer)
>>> just need to finish the PRs, and then we should be really close to
>>> full support for expressions in filter.
>>>
>>> Josh: do you think we could get Lookup as expressions in to 1.10
>>> instead of 1.11?
>>>
>>>  - Anssi
>>>
>>> On Wed, Sep 30, 2015 at 3:46 AM, Josh Smeaton 
>>> wrote:
>>> > 1. Lookups should become Expressions, just as Transforms have become
>>> > Expressions. This will let us process Lookup arguments as Expressions
>>> > all
>>> > the way the way through. I think this should be a major goal for
>>> > version
>>> > 1.11.
>>> >
>>> > 2. Chaining transforms is now possible since they are just Func
>>> > expressions.
>>> > Func(Func(Func('field_name'))) is no issue.
>>> >
>>> > 3. Sounds like an OK idea, but I haven't looked into the details enough
>>> > to
>>> > really comment. I do think we should create the correct form as early
>>> > as
>>> > possible (parsing into a chain of Lookup/Transform expressions) so we
>>> > don't
>>> > have to do parsing in multiple places. The entry points to .filter()
>>> > and
>>> > .exclude(), or their direct counterparts in sql.query sound ideal.
>>> > Anssi has
>>> > mentioned elsewhere that WhereNode's should only contain fully

Re: Django ORM query syntax enhancement

[Anssi Kääriäinen]

On the core ORM side we need to make
.exclude(LessThan(F('friends__age'), 30)) do a subquery.  This way
.exclude(friends__age__lt=30) does the same thing as the expression
version. This isn't that easy to do. If we just use
resolve_expression, then the friends relation will generate a join,
and then as second step do a negated filter on the joined value.
Instead we want to detect that the LessThan expression needs to be
pushed in to a subquery.

So, we need to solve:

 A) A way to ask an expression if it is referencing a multijoin
(possible approach is to just have a method
"refs_multi_valued_relation(query)")
 B) When the ORM sees an expression that is reffing a multijoin in an
exclude filter, then we need to push the expression in to a subquery.

A) requires some new work. This shouldn't be that hard to implement,
we just recursively ask subexpressions if they reference a multijoin.

Something like https://github.com/django/django/pull/4385 will make B)
much easier to implement.

I've been working on making Q-objects responsible for resolving
themselves. See https://github.com/django/django/pull/4801. This
should solve 3).

We don't seem to be missing any major parts. I (or some volunteer)
just need to finish the PRs, and then we should be really close to
full support for expressions in filter.

Josh: do you think we could get Lookup as expressions in to 1.10
instead of 1.11?

 - Anssi

On Wed, Sep 30, 2015 at 3:46 AM, Josh Smeaton  wrote:
> 1. Lookups should become Expressions, just as Transforms have become
> Expressions. This will let us process Lookup arguments as Expressions all
> the way the way through. I think this should be a major goal for version
> 1.11.
>
> 2. Chaining transforms is now possible since they are just Func expressions.
> Func(Func(Func('field_name'))) is no issue.
>
> 3. Sounds like an OK idea, but I haven't looked into the details enough to
> really comment. I do think we should create the correct form as early as
> possible (parsing into a chain of Lookup/Transform expressions) so we don't
> have to do parsing in multiple places. The entry points to .filter() and
> .exclude(), or their direct counterparts in sql.query sound ideal. Anssi has
> mentioned elsewhere that WhereNode's should only contain fully resolved
> expressions, so resolving will need to be done directly after parsing (or
> during).
>
> Part 1 above can be started now if you have the time or interest. We can
> nail down the particulars of part 3 while we're solving part 1. Part 1 may
> drive some of part 3.
>
> Cheers
>
>
> On Wednesday, 30 September 2015 04:49:54 UTC+10, Alexey Zankevich wrote:
>>
>> Here is a list of issues to solve to support explicit transforms and
>> lookups by filter (and exclude) methods.
>>
>> 1. Make Lookup.__init__ signature to support initialization with F objects
>> or string path (e.g. GreaterThan(F('user__id'), 10) or
>> GreaterThan('user__id', 10)), not sure it's possible to use simultaneously
>> with the current approach with lhs, rhs initialization (even with moving it
>> to a separate class method, e.g Lookup.build(lhs, rhs)), so I assume
>> creating so-called util classes which will delegate SQL-related
>> functionality to existing Lookup classes.
>>
>> 2. Chain transforms by passing them as argument:
>>
>> Lower(Unaccent(F('user__name)))
>>
>> 3. Decide if Q objects shall support explicit lookups/transforms as
>> argument as well - it's a kind of logical step, as without Q objects it will
>> not be possible to perform complicated conditions (AND, OR, NOT).
>> In that case lookup/transform parsing should be moved from QuerySet object
>> to Q object - filter will take already parsed lookup tree.
>> Example:
>>
>> Q(user__name__lower__unaccent__icontains='Bob') will internally parse it
>> and build next structure:
>>
>> Q(Icontains(Lower(Unaccent(F('user__name', 'Bob')
>>
>>
>> On Sunday, August 16, 2015 at 4:18:26 PM UTC+3, Alexey Zankevich wrote:
>>>
>>> Hi all,
>>>
>>> This topic is related to the current ORM query syntax with underscores.
>>> There are lots of arguing related to it, anyway it has pros and cons.
>>>
>>> Let's take a concrete example of querying a model:
>>>
>>> >>>
>>> >>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>>>
>>>
>>> Pros:
>>>
>>> 1. The syntax is easy to understand
>>> 2. Can be extended with custom transforms and lookups
>>>
>>> However, there are several cons:
>>>
>>> 1. Long strings is hard to read, especially if we have fields with
>>> underscores.
>>> It's really easy to make a mistake by missing one:
>>>
>>> >>>
>>> >>> GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)
>>>
>>> Not easy to catch missing underscore between user and profile, is it?
>>> Even
>>> though, it's not easy to say whether it should be "user_profile"
>>> attribute or
>>> user.profile foreign key.
>>>
>>> 2. Query strings can't be reused, thus the approach violates DRY
>>> principle.
>>> For example, we need to or

Re: SQL ALchemy support on django

[Anssi Kääriäinen]

The meta API used by admin and model forms should be completely public
and stable. The problem is that some methods of the meta API return
field instances, and the API for the fields (especially for related
fields) isn't public nor stable. You'll need to somehow return field
instance that act like Django's.

 - Anssi

On Thu, Sep 17, 2015 at 7:00 AM, Asif Saifuddin  wrote:
> Hi,
>
> With "support" I meant having a official djangoish way to use sqlalchemy on
> django. As I have to use model meta API to add support for any non django
> backend to orm my question is, is django meta API fully formalized? If not
> how much work is needed to formalize them?(as denial did a lots of work then
> their should be proper guideline or enough insight to start digging? )
>
> Or we are ok with the existing API's?
>
> Regards
>
> Asif
>
> On Thu, Sep 17, 2015 at 5:20 AM, Russell Keith-Magee
>  wrote:
>>
>> Hi Asif,
>>
>> It depends entirely what you mean by "support".
>>
>> Django is just Python, so there's absolutely no reason you can't write
>> a Django site that uses Django for the URLs, views and forms, but
>> SQLAlchemy for the data access.
>>
>> Out of the box, you won't be able to use ModelForms or the Admin.
>> However, with the new stable Meta API that was added in 1.8, you
>> should be able to write a layer for SQLAlchemy that makes a SQLAlchemy
>> table definition "quack" like a Django model, allowing you to wrap.
>>
>> Daniel Parathion did this as a proof of concept as part of his GSoC
>> project using the Gmail API - he was able to expose a Gmail inbox as a
>> Django model, browse his inbox in Admin, and use a ModelForm to
>> compose an email from within the admin.
>>
>> I gave a presentation at the recent DjangoCon US about this topic; the
>> slides are available here:
>>
>> https://speakerdeck.com/freakboy3742/i-never-meta-model-i-didnt-like
>>
>> Video should be available in the next week or so. Short version: it
>> won't be *trivial*, but it's certainly *possible*, although you're in
>> undocumented territory, so you'll have to navigate the sharp edges
>> yourself.
>>
>> Yours,
>> Russ Magee %-)
>>
>>
>>
>>
>> On Wed, Sep 16, 2015 at 4:35 PM, Asif Saifuddin  wrote:
>> > Hi,
>> >
>> >
>> > How much difficult it will be to add SQLalchemy support to django? what
>> > are
>> > the technical difficulties? And which are the ways to add sqlalchemy
>> > support
>> > to django?
>> >
>> > As SQLalchemy is separated in core and orm will it be wise to use
>> > sqlalchemy
>> > core as the backend for django orm? Or  through the meta api? any
>> > suggestion
>> > or direction to the solutions?
>> >
>> > The available 3rd party packages seems to be incomplete.
>> >
>> >
>> > Regards
>> >
>> > Asif
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Django developers (Contributions to Django itself)" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to django-developers+unsubscr...@googlegroups.com.
>> > To post to this group, send email to django-developers@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/django-developers.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/django-developers/33078771-6802-44f9-9182-1eb2f7104e23%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/django-developers/puVGMV7GlNE/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAJxq849Ccd%2Bvz9uYf45UtcjnMXToRChoO-MyTz5b3YLFFEp6Ww%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAKAqTgrgsSz6OEWQXt5LzLVOCybsLyV1yZJV%3DrPBbVHu6D9xKA%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django it

Re: CHECK Constraints and migrations

[Anssi Kääriäinen]

I recall another idea that would work very well with model level
Q-object based checks: making lookups and transforms work on Python
side, too. Then, when you supply a check  "ends_after_starts", we
could both add it as a database level constraint, and also validate
the constraint on Python side when altering data.

I think it is a really good idea to add more control to Fields over
what migrations they generate, but unfortunately I don't have time nor
expertise to work on this.

 - Anssi

On Fri, Aug 28, 2015 at 7:30 AM, Andrew Godwin  wrote:
> No, fields can't really specify any of the SQL used to make/update/delete
> them (partially as the SQL is heavily dependent on the database backend, and
> Django traditionally has the fields as the database-agnostic bit and the
> backends as the specific bit). All they can do is supply a db_type, which
> has to fit in the space a type would go for ADD COLUMN, etc. (that's how
> contrib.postgres has its fields work, for example).
>
> I know that Marc Tamlyn was looking into this for some of his PostgreSQL
> stuff, and other people have floated some ideas around letting fields have
> more control over migration stuff too - they may be able to chime in when we
> get to a sensible time in Europe on this.
>
> I'm definitely far from the only person who can work on fixing up the
> migration stuff; in fact, I haven't done much to it at all recently and it's
> been all other people, so if you think you want to take a stab at it, you're
> more than welcome, and I'm more than happy to review patches against
> migrations.
>
> Andrew
>
> On Thu, Aug 27, 2015 at 9:25 PM, Gavin Wahl  wrote:
>>
>> That's too bad. I don't think it's really worth it unless the experience
>> works seamlessly, because if I'm going to write a migration manually I might
>> as well just use a RunSQL. I did have another slightly hacky idea that might
>> work. Can custom Fields specify the operations used to create and drop them?
>> It might then be possible to write a pseudo-field called 'Check':
>>
>> class FooModel(models.Model):
>> # ...
>> check_ends_after_start = Check(Q(end_date__isnull=True) |
>> Q(end_date__gt=F('start_date')))
>>
>> That instead of adding a column called ends_after_start, it would add a
>> constraint.
>>
>> Do you think it's possible for anyone besides you to work on making the
>> migration framework more extensible? I'm interested in contributing in this
>> area.
>>
>>
>> On Thursday, August 27, 2015 at 8:07:55 PM UTC-6, Andrew Godwin wrote:
>>>
>>> Hi Gavin,
>>>
>>> Hooking into the migration autodetector is currently not really possible
>>> - it's quite inextensible, which is something I'd love to get ironed out in
>>> the long term but isn't really possible right now to modify it from
>>> third-party code.
>>>
>>> You can, however, supply custom migration Operation classes that people
>>> could put into their migrations manually, or potentially have some other way
>>> of outputting those migration files with operations in them (a lot of the
>>> code to write them out is re-useable). Hopefully someone else has a better
>>> idea!
>>>
>>> Andrew
>>>
>>> On Thu, Aug 27, 2015 at 6:27 PM, Gavin Wahl  wrote:

 I'm interested in writing a third-party app that allows users to
 declaratively write check constraints and have them automatically be added
 and removed by migrations. I'm envisioning being able to attach a list of Q
 objects to your model, something like:


 class Meta:
 checks = [
 Q(end_date__isnull=True) | Q(end_date__gt=F('start_date'))
 ]

 Now that we have migrations, formalized Meta, and custom lookups, is it
 possible for a third-party app to manage this seamlessly?

 I'm having trouble figuring out where it could hook in to migrations to
 detect changes to the check constraints, and automatically generate the
 migrations with operations to update the constraints. What code I should be
 reading to figure out how to do it?

 --
 You received this message because you are subscribed to the Google
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at http://groups.google.com/group/django-developers.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/django-developers/CACPudh1gRB_Jov_bH-maDGr4EBRugFuV3Y%2BBzYM9w0yyki7zkg%40mail.gmail.com.
 For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+un

Re: Django SQL templates

[Anssi Kääriäinen]

The django-refsql package implements a limited subset of this. You can
reference to columns in an existing ORM query using {{field}} syntax.
For example:
User.objects.annotate(ref=RefSQL("case when {{best_friend__age}} >
{{age}} then 'older' else 'younger' end"))
This would also set up a join to User's best friend model.

See https://github.com/akaariai/django-refsql

I don't like the template idea too much. While it might work well for
some use cases, it seems that ORM + raw SQL is enough for now. For
those use cases where you want to use this style, it should be
possible to implement this in a third party project. If it turns out
we are missing some public APIs in Django, we can consider adding some
more hooks.

 - Anssi

On Thu, Aug 27, 2015 at 10:43 PM, Alexey Zankevich
 wrote:
> Hi all,
>
> After reading future of QuerySet.extra() discussion, I was thinking about
> the current custom ways to tweak SQL queries in Django. Firstly, it was the
> current .extra method, which everyone likely wants to get rid of. Secondly,
> performing raw SQL queries from queryset manager or even directly from
> cursor.
> It's really a great approach to perform raw SQL and still receive objects of
> the given model at the end, I think. However, the main disadvantage of the
> current implementation - it's required to type in real table/field names,
> which destroys model's abstraction level (when you never reference
> tables/columns by their real names).
> For such cases it's great to have DSL to build SQL queries, but it's a big
> question how low- or high-level language should be. Anyway, it ended up with
> the idea to have SQL templates, similar to current Django HTML templates. I
> even implemented a sample as proof of the concept:
>
> https://github.com/Nepherhotep/django-orm-sugar/blob/exec_sql_template/tests/sql_template_test_app/app/tests.py
> https://github.com/Nepherhotep/django-orm-sugar/blob/exec_sql_template/tests/sql_template_test_app/app/templates/select_user_profile.sql
>
> How it works - there is a template "select_user_profile.sql" in a standard
> templates folder:
>
> {% autoescape off %}
> SELECT * FROM {{ model }} WHERE {{ model.name }} = {{ search_term }}
> {% endautoescape %}
>
> I had to put autoescape tags to prevent quotes in model name converted into
> HTML entities.
> A function below should be called somewhere in views:
>
> raw_queryset = exec_sql_template(UserProfile, "select_user_profile.sql",
> {'search_term': 'Bob'})
>
> This function automatically creates a variable "model" (which is actually a
> special model wrapper, which prints table or column names) and passes it
> into template context. To prevent SQL injection, passed params converted
> into %(variable_name)s representation, and escaped automatically by driver.
> Thus the solution above doesn't work with SQLite (as it doesn't support
> named variables).
> Also I wanted to automatically provide aliases for all existing models in
> form . (not implemented yet):
>
> SELECT *
> FROM
> {{ model }}
> LEFT OUTER JOIN
> {{ myapp.ModelName }}
> ON
> {{ myapp.ModelName.profile }} = {{ model.pk }}
>
> Almost everything can work out of the box using standard Django templates,
> except quotes converted into entities, unless they don't disabled
> explicitly. Also, not sure if all the databases support 'tablename'.'name'
> syntax to safely define table/column, otherwise wrapper classes should be
> engine dependent.
>
> I don't suggest a concrete implementation, just wanted to get responses of
> the concept.
>
> Regards,
> Alexey
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/da716f91-e5ab-466b-8d3e-475cd3738c12%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GcFYiRC%2BM6Zk9bDtaCp4Yk%3DnOvb_6%3DHOBgEeuCGS-hiQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Keeping apps without migrations?

[Anssi Kääriäinen]

Could we allow applications to have fake migration files? I'm thinking
of something like operations.AllModelsUpdated(). After this operation
is ran, Django will use the present models for that app. You could
place this as your last migration, and run direct SQL in previous
migrations (or use some completely different mechanism) . Direct SQL
without model state changes will be fast. The last migration would
supposedly be fast, as it only needs to construct up to date images of
the current models.

The problem is that I'm not at all sure how this will mix with
migrations from other applications, so this proposal is perhaps
technically impossible to do. But if this is feasible, then it should
be possible to mix migrations and manual database changes pretty much
seamlessly. This one would work much better if we had project level
migrations - implementing a migration operation that tells Django that
all of the project's models are up to date is almost trivial.

I believe a fake migration would work pretty well for the use case
where you want to use manual database control for your project, but
use 3rd party applications with migrations.

Another idea I've been thinking is that maybe we could have snapshots
for migrations. The reason I believe this would work is that building
the model state is expensive, but in many cases it is completely
redundant to build the model state from scratch.

The snapshots would essentially do the following operation: after
migrations X, Y and Z are applied the model state will be as presented
in this migration. The snapshots would only contain model state, not
any operations. When you run manage.py migrate, the app state builder
will be able to skip to latest snapshot that has all dependencies
already applied, and then continue building model state from there.
When performance begins to degrade, you can create a new snapshot. The
snapshots would probably have to be project level to work properly.
This would seriously speed up migrations, except for the case where
you run the migrations against fresh database.

 - Anssi


On Fri, Aug 28, 2015 at 3:18 AM, Andrew Godwin  wrote:
> I think I agree with Markus that we do not yet have a good enough
> performance story to make migrations mandatory. If you want to ditch them
> and do your own database management, we should let you - which we could do
> with some kind of MIGRATIONS_MODULES opt-out that basically sets everything
> to pseudo-unmanaged, but I see no reason not to have the syncdb-like
> solution keep working as it at least runs off of SchemaEditor now.
>
> Andrew
>
> On Thu, Aug 27, 2015 at 5:14 PM, Markus Holtermann
>  wrote:
>>
>> The general benefit of *not* having migrations is the incredible better
>> performance to setup the tables. Especially for larger projects this is
>> something we should at least keep in mind (see
>> https://code.djangoproject.com/ticket/24743). I spent a fair bit of time on
>> the issue and have not yet found a satisfying solution.
>>
>> Supporting apps without migrations would essentially continue to support
>> the old `syncdb` behavior, though not as a standalone command, but as a flag
>> for `manage.py migrate`. It takes the models from those apps and calls
>> `schema_editor.create_model()`, taking the deferred SQL and running it after
>> all models of apps w/o migrations have been created (current behavior [in
>> tests] I think). This would continue to raise issues if models in apps w/o
>> migrations refer to models in apps w/ migrations (missing constraints), but
>> that is a limitation we had since 1.7 (silent failure; 1.8 fails loud).
>>
>> While migrations should provide *the* way to maintain your database I see
>> the points that have been made in other recent discussions. Although Django
>> doesn't force you to use migrations there are some things (e.g. not create
>> the `django_migrations` table) we probably should not do unless you actually
>> use migrations.
>>
>> We might want to consider looking at the `MIGRATION_MODULES` settings
>> variable and allowing to set values to `None` in order to disable migrations
>> for that app, but that's a different story.
>>
>> /Markus
>>
>>
>> On Friday, August 28, 2015 at 4:26:19 AM UTC+10, Tim Graham wrote:
>>>
>>> It would be great if we could decide this by next Friday (September 4) or
>>> so, so that we have some time to do the necessary implementations and
>>> testing before alpha. (For example, running manage.py test without
>>> migrations currently gives "no such table ...". It would be nice to give a
>>> friendlier message if we require migrations for running tests
>>> (https://github.com/django/django/pull/5100 does this to some extent, but
>>> could be improved I think).
>>>
>>> I know at least Markus said he would like to present an argument for
>>> keeping support for apps without migrations.
>>>
>>> On Thursday, August 13, 2015 at 1:56:05 PM UTC-4, Tim Graham wrote:

 It's a one line change in the code to keep the status quo fro

Re: The hypothetical models.MultiField: is there a rationale to avoid it ? Or could it make it into future releases ?

[Anssi Kääriäinen]

I've been thinking how we could move forward with composite fields without 
requiring too much changes in one go (large changes tend to never happen).

It seems the first step forward would be to introduce a composite field 
with the following properties:

class Foo(models.Model):
x = models.IntegerField()
y = models.IntegerField()
point = models.CompositeField((x, y))

foo.point would always return a named tuplet. You could customize the 
return value to a full fledged object by implementing a custom subclass of 
CompositeField, but Django will not have any opinion on what should happen 
when you do:
foo.x = 1
foo.y = 2
foo.point.x = 2
foo.point.x == foo.x ???

this was one of the main concerns in the DEP.

In addition, on query side you should be able to use 
.filter(point__x__lte=...), and you should be able to select 
.values('point'). We also need migrations support, and likely changes in a 
lot of other places in Django.

The draft DEPs for this feature shouldn't be used as definite resource when 
implementing this feature.

 - Anssi

On Thursday, August 20, 2015 at 6:49:11 PM UTC+3, Aron Podrigal wrote:
>
> Have a look at [1] it is a composite field implementation.
>
> [1] 
> https://groups.google.com/forum/m/#!msg/django-developers/MZUcOE6-7GY/sZkBaHvC8wgJ
> [2] 
> https://github.com/django/deps/blob/master/draft/0191-composite-fields.rst
> [3] 
> https://github.com/django/deps/blob/master/draft/0192-standalone-composite-fields.rst
> On Aug 20, 2015 10:31 AM, > wrote:
>
>>
>>
>> Le mardi 18 août 2015 01:36:28 UTC+2, Tim Graham a écrit :
>>>
>>> I think the general idea is captured in ticket #5929 -- Allow Fields to 
>>> use multiple db columns (complex datatypes). Is that the gist of your 
>>> proposal?
>>>
>>
>> Thank you for this link! It seems to discuss the same end result as what 
>> I tried to present in my first message: the ability to have a single 
>> models.Field managing an arbitrary number of DB columns under the hood.
>>
>> The proposed approach is perhaps a bit different: if I understood the 
>> ticked correctly, it proposes to change the base Field class to make it 
>> possible, when deriving from it, to manage one or several DB columns. My 
>> first idea was more to mimic the composite pattern implementation already 
>> in use with forms.MultiValueField:
>> * The models.Field *leaf* classes would still manage a single DB column.
>> * Introduce a models.MultiField class, which is a container of 
>> models.Field classes (be it leaf classes or other MultiField classes). This 
>> container would address the multiple columns indirectly, through the 
>> interface of the composing fields. And, to the eyes of the rest of the 
>> code, it would behave as a normal field, notably offering the to_python() 
>> feature, hiding the composition in its implementation details.
>>
>> I did not take time yet to try and assemble a prototype of this idea; In 
>> fact, I first wanted to confirm if such approach has not already been 
>> rejected in the past, before investing work in it ;) 
>>
>> Does it sound like a feasible/interesting idea ? Or is there a good 
>> reason not to do it / too many obvious technical complications that I did 
>> not foresee ?
>>
>> Thank you for reading,
>>   Ad
>>
>>
>>> https://code.djangoproject.com/ticket/5929
>>>
>>> On Monday, August 17, 2015 at 5:11:01 AM UTC-4, boito...@gmail.com 
>>> wrote:

 Hi,

   While implementing  our collection management system based on Django, 
 we are always excited by the extensibility of the framework.
   Most recently, we were exposed to the *forms.MultiValueField* and* 
 widgets.MultiWidget*, that seem to offer composition capacities to 
 users of the *form* and *widget* layers. Yet, we did not find any 
 equivalent in the *model* layer, which seemed a bit surprising knowing 
 that those 3 layers can work hand-in-hand very easily

   Is there a rationale to prevent implementation of such a 
 models.MultiField class ? It could be a wrapper around the composite 
 pattern in the *model* layer, allowing users to easily define custom 
 models.Field that would leverage existing *models.Field* classes, by 
 assembling them for specific purposes (while maximizing reuse).

 

 This question was also raised in Stack Overflow here: 
 http://stackoverflow.com/q/32014748/1027706. Below is a summary of the 
 question's example motivating such feature request:

 Imagine we want to store partial date in the DB (i.e., a date that is 
 either complete , or just month+year, or just year). We could model it in 
 the models layer using a *models.DateField* + a *models.CharField* 
 (this last field storing whether the date is complete, or month+year, or 
 just year).

 Now, if we move to the forms layer, let's say we want a custom 
 validation step that when a date is partial, the "unused" part of the 
>>

Re: Django ORM query syntax enhancement

[Anssi Kääriäinen]

7;Bob'
>> Equal(Unaccent('user__name'), 'Bob')
>>
>> >>> Equal(P.user.name, 'Bob') == 'Bob'
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> Exception: Lookups comparison is not allowed
>>
>> >>> Contains(P.user.name, 'Bo')  # lookup
>> >>> Date(P.user.last_login_date, datetime.now().date)  # transform
>>
>>
>> Questions to discuss and possible pitfalls:
>>
>> 1. Handling descended ordering in order_by
>>
>> >>> path = P.user.last_login_date
>> >>> -path
>> '-user__last_login_date'
>>
>> or even
>>
>> >>> -p
>> NegP('user__last_login_date')
>>
>> Is it possible path subtraction is required in future by any reason? In
>> that
>> case the given approach will be inconvenient. Or may be better to just let
>> users to format desc ordering themselves?
>>
>> >>> '-{}'.format(path)
>>
>> 2. Don't think it's a big deal, but users might do next thing:
>>
>> >>> path = P.user.last_login_date
>> >>> GameSession.objects.filter(path=today)
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> FieldError: Cannot resolve keyword 'path' into field.
>>
>> At least need to mention it in documentation
>>
>> 3. Classes vs factory functions:
>>
>> >>> Equal(P.user.name, 'Bob')
>> Equal('user__name', 'Bob')
>>
>> vs
>>
>> >>> equal(P.user.name, 'Bob')
>> Equal('user__name', 'Bob')
>>
>>
>> If you are fine with described syntax I can start with DEP.
>>
>> Regards,
>> Alexey
>>
>> On Monday, August 17, 2015 at 12:26:45 PM UTC+3, Anssi Kääriäinen wrote:
>>>
>>> I like this idea, too.
>>>
>>> The work I and Loic have been doing has been about transforms that can
>>> accept arguments. I don't see why the two styles couldn't be mixed
>>> directly.
>>>
>>> .filter(Q.user.profile.last_login.date(timezone='Europe/Helsinki') <
>>> '2015-01-01')
>>> .filter(Q.user.last_name.collate('fi').unaccent() == 'Kaariainen')
>>>
>>> OTOH if you want to use expressions directly, that should be OK, too
>>> (if you don't mind the non-optimal readability):
>>> .filter(Exact(unaccent(collate(Q.user.last_name, 'fi'))), 'Kaariainen'))
>>>
>>> All in all, +1 to working towards these goals. I'm not yet convinced
>>> we need all this in Django core directly. Maybe we should first try
>>> these ideas in 3rd party apps, and then pick a proven solution into
>>> Django core?
>>>
>>>  - Anssi
>>>
>>> On Mon, Aug 17, 2015 at 3:33 AM, Josh Smeaton 
>>> wrote:
>>> > Hi Alexey,
>>> >
>>> > I find this approach really interesting, and I don't mind the API
>>> > you've
>>> > created at all. Even if this doesn't make it into Django, I think
>>> > there's
>>> > enough utility in your library that other people would want to use it.
>>> > I'm
>>> > not going to comment on specifics just yet like method names, because I
>>> > figure that if we can agree on an approach, the bike-shedding can come
>>> > later.
>>> >
>>> > So first things first. I think we can all, or at least mostly, agree
>>> > that
>>> > having an alternative syntax for building filters is a good idea.
>>> > String
>>> > based lookups are cool, and won't go away, but sometimes you want to
>>> > have
>>> > greater control over what kind of filter you want to build without
>>> > having to
>>> > register global lookups and transforms. I see similarities with
>>> > prefetch_related. You can provide a string based field lookup, or use a
>>> > Prefetch() object to give users more control over the behaviour of the
>>> > prefetch.
>>> >
>>> > I've been working on a patch
>>> > (https://github.com/django/django/pull/5090) to
>>> > unify Transforms and Expressions (Func specifically). The work behind
>>> > that
>>> > can be (and should be) extended to include Lookups -- making them fully

Re: Django ORM query syntax enhancement

[Anssi Kääriäinen]

I like this idea, too.

The work I and Loic have been doing has been about transforms that can
accept arguments. I don't see why the two styles couldn't be mixed
directly.

.filter(Q.user.profile.last_login.date(timezone='Europe/Helsinki') <
'2015-01-01')
.filter(Q.user.last_name.collate('fi').unaccent() == 'Kaariainen')

OTOH if you want to use expressions directly, that should be OK, too
(if you don't mind the non-optimal readability):
.filter(Exact(unaccent(collate(Q.user.last_name, 'fi'))), 'Kaariainen'))

All in all, +1 to working towards these goals. I'm not yet convinced
we need all this in Django core directly. Maybe we should first try
these ideas in 3rd party apps, and then pick a proven solution into
Django core?

 - Anssi

On Mon, Aug 17, 2015 at 3:33 AM, Josh Smeaton  wrote:
> Hi Alexey,
>
> I find this approach really interesting, and I don't mind the API you've
> created at all. Even if this doesn't make it into Django, I think there's
> enough utility in your library that other people would want to use it. I'm
> not going to comment on specifics just yet like method names, because I
> figure that if we can agree on an approach, the bike-shedding can come
> later.
>
> So first things first. I think we can all, or at least mostly, agree that
> having an alternative syntax for building filters is a good idea. String
> based lookups are cool, and won't go away, but sometimes you want to have
> greater control over what kind of filter you want to build without having to
> register global lookups and transforms. I see similarities with
> prefetch_related. You can provide a string based field lookup, or use a
> Prefetch() object to give users more control over the behaviour of the
> prefetch.
>
> I've been working on a patch (https://github.com/django/django/pull/5090) to
> unify Transforms and Expressions (Func specifically). The work behind that
> can be (and should be) extended to include Lookups -- making them fully
> fledged expression objects. That work will need to happen regardless. Once
> this patch is finished and lands, then we should be able to extend .filter()
> and .exclude() to allow expressions to be composed as filters:
>
> GameSession.objects.filter(GreaterThan('user__profile__last_login_date',
> yesterday))
> # or, if last_login was a datetime, and we wanted to compare the date part
> only
> GameSession.objects.filter(Equal(Date('user__profile__last_login'),
> datetime.now().date))
> # or if we wanted to implement __gt__ and __eq__ etc:
> GameSession.objects.filter(Date('user__profile__last_login') ==
> datetime.now().date))
>
> Loic and Anssi have also been working on alternative syntaxes
> (https://github.com/django/django/pull/4953):
>
> GameSession.objects.filter(E('user__profile__last_login').date() ==
> datetime.now().date)
>
>
> Article.objects.filter(E('headline').collate('fi') == 'Article1')
>
> Both of these approaches still suffer from "the string problem" that you're
> trying to address, but minimises the final component by extracting the
> lookup and transform components as objects instead. So I think your idea
> here could nicely coexist:
>
> GameSession.objects.filter(Equal(Date(Q.user.profile.last_login),
> datetime.now().date))
> GameSession.objects.filter(E(Q.user.profile.last_login).date() ==
> datetime.now().date)
>
> Or, even building this into F expressions rather than Q expressions:
>
> GameSession.objects.filter(Equal(Date(F.user.profile.last_login),
> datetime.now().date))
> GameSession.objects.filter(E(F.user.profile.last_login).date() ==
> datetime.now().date)
>
> I think it may look better with F objects, considering they are Field
> references, and since the Lookups (Equal/GTE) parts accept F expressions
> anyway. I'm not too concerned about this particular detail though.
>
> A DEP is probably the right way to go here but I wonder if we should expand
> the scope to include alternative filter syntax as a whole
> (expressions/callable transforms) as well as the dot field reference
> notation you've proposed above. Then we can consider how all these things
> might work together, and clearly document why we've gone one way and not
> another. Obviously, alternatives can exist outside of core where the API is
> available.
>
> I'll be happy to work as the shepherd if needed. But I'd also like some
> input from Loic and Anssi especially, as well as others in the core team
> with some interest in the ORM.
>
> Regards,
>
>
> On Sunday, 16 August 2015 23:18:26 UTC+10, Alexey Zankevich wrote:
>>
>> Hi all,
>>
>> This topic is related to the current ORM query syntax with underscores.
>> There are lots of arguing related to it, anyway it has pros and cons.
>>
>> Let's take a concrete example of querying a model:
>>
>> >>>
>> >>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>>
>>
>> Pros:
>>
>> 1. The syntax is easy to understand
>> 2. Can be extended with custom transforms and lookups
>>
>> However, there are several cons:
>>
>> 1. Lon

Re: future of QuerySet.extra()?

[Anssi Kääriäinen]

On Tuesday, August 4, 2015, Shai Berger  wrote:
>
> The classic database aggregation examples involve aggregation over time:
> Sum
> of sales per quarter (which is, itself, a function over date); average
> temparature per month; etc. All these require group-by clauses which name
> (expressions over) columns which are not FKs.
>
>
The following should work:

 
qs.annotate(quarter=Quarter('sold_date')).values('quarter').annotate(sum=Sum('amount'))

I recall some changes to how the group by expression is generated for
expressions. If I recall correctly the group by should have the expression
itself now, not the base columns of the expression.

There are many similar examples involving "enum columns" -- columns with a
> limited set of choices, where the choices are not instances of another
> model.


I'm not sure of this one.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1FqTkdmnXNYvjZ8U_QF9%2BWzrMCb1DDFQU5WB4SiKPX06g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: future of QuerySet.extra()?

[Anssi Kääriäinen]

On Tue, Aug 4, 2015 at 8:06 AM, Tai Lee  wrote:
> I think that while `extra()` gets a bad rap, it is extremely useful when
> creating generic pluggable apps, and I for one would miss it if it were to
> be removed.

The idea is that expressions should do everything that extra() can do.
There might be cases where this is not yet true, but I hope we will
get there in a couple of releases.

The extra() method should not  be better than expressions for reusable
apps. Actually the contrary, with expressions you can generate
different SQL for different databases.

> In a comment on #7231 a few years ago I wrote:
>
>> I can't reply to the discussion linked on Google Groups anymore, but
>> wanted to add my comment that I have found many uses for .extra() and also
>> hit the limitation of being unable to add left outer joins to a query
>> generated by the ORM. I would not like to see it disappear in favour of raw
>> SQL.
>>
>>
>> My current use case is for joining reverse generic relations on queries
>> against a model that does not have a GenericRelation field back to the
>> generic model (because I don't know which models it will be used with -- I
>> can't add models to 3rd party code).
>>
>>
>> To require the whole query in raw SQL in order to join generic relations
>> would mitigate the pluggability of the generic model in question.
>
>
> While `raw()` may be allow you to get model instances from any custom SQL,
> it will often be prohibitively complicated or impossible to provide generic
> queryset methods that extend 3rd party apps. The RawSQL expression example
> given would appear to execute a subquery for each row of the result set. It
> wouldn't seem to help with the example above, unless I'm missing something?

What we want is something like
qs.add_relation(new_rel=Relation(SomeModel, on_clause=some_expression,
allow_nulls=True))

The some_expression variable generates sql for the ON clause, and
allow_nulls=True means the join will be LEFT join. Typically
some_expression could be something like Q(id=F('new_rel__other_id')) &
Q(new_rel__lang='fi').

The effect of this would be that the qs has a new relation called
new_rel to SomeModel, and that relation could be used in other query
constructs (for example .filter(new_rel__somef__lte=10)).

Making this work is high priority to me.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GQ1oAgeSCgNvNu%3D1ucKEuduDOttpwCUCxSd-Ct8auzcg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: future of QuerySet.extra()?

[Anssi Kääriäinen]

On Mon, Aug 3, 2015 at 2:07 PM, Tim Graham  wrote:
> Is there an example we could give in the docs to point readers in that
> direction?

There doesn't seem to be direct documentation about RawSQL expression.
The RawSQL is once mentioned in the documentionation, but it
definitely needs a bit more in the database functions documentation.

The usage of RawSQL is really simple:
qs.annotate(val=RawSQL("(select col from sometable where othercol
= %s)", (someparam,)))

This is equivalent to qs.extra(select={'val': "select col from
sometable where othercol = %s"}, select_params=(someparam,))

The main benefit of using RawSQL is that you can set output_field if
needed. The main downside is that if you refer to some table alias of
the qs in the raw sql, then it is possible that Django changes that
alias (for example, when the qs is used as a subquery in yet another
query).

Combining raw SQL with alias relabeling support is possible, too with
the following library: https://github.com/akaariai/django-refsql

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1G0xtEd7v-s60X-%3D2Ppwn7xS0NEmD_cWh8_0G2RT063mg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: default values on database level

[Anssi Kääriäinen]

On Wednesday, July 29, 2015 at 8:05:10 AM UTC+3, Aron Podrigal wrote:
>
> I would like to propose making Fields.default to rely on the database 
> generated defaults where possible. This could be implemented by having 
> some  constants like fields.CURRENT_TIMESTAMP, etc.  The tricky part here 
> is to detect database backend specific capabilities and have a python 
> equivalent fallback.
>
> Any thoughts?
>

A couple of questions about this:
  - What should happen when you instantiate a new model with DB default, 
and show that model to the user in a form. Is the field shown to the user, 
and if so, what value should it have?
  - How to implement actual fetching of the values after save() - 
PostgreSQL supports RETURNING, but this is not the case for all databases.
  - When is the default applied? On first save of the model, or on model 
init time?
  - Do we want to allow extending this to defaults that are applied on 
every save (automatic database backed modified timestamps for example)?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f1a7cc68-de1a-49f8-bdf8-ebfaade59955%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: future of QuerySet.extra()?

[Anssi Kääriäinen]

+1

 - Anssi

On Friday, July 31, 2015 at 11:00:01 PM UTC+3, Tim Graham wrote:
>
> I had in mind a documentation note like this:
>
> Use this method as a last resort
>
>
> This is an old API that we aim to deprecate at some point in the future. 
> Use it only if you cannot express your query using other queryset methods. 
> If you do need to use it, please file a ticket with your use case so that 
> we can enhance the QuerySet API to allow removing extra(). We are no 
> longer improving or fixing bugs for this method.
>
> On Friday, July 31, 2015 at 2:07:34 PM UTC-4, Collin Anderson wrote:
>>
>> I wonder if there's a way in the docs we can deprecate it as in "we don't 
>> recommend you use it", but not actually schedule it for removal.
>>
>> On Friday, July 31, 2015 at 2:01:20 PM UTC-4, Marc Tamlyn wrote:
>>>
>>> I don't know about unmaintained, but I think there's a consensus that 
>>> .extra() has a horrible API and we should do away with it eventually. That 
>>> said I think there are still enough things that can't be done without it at 
>>> present. A lot fewer now we have expressions, but still some.
>>>
>>> I'd be happy to put a moratorium on improving it, but we can't deprecate 
>>> it yet.
>>>
>>> On 31 July 2015 at 18:58, Tim Graham  wrote:
>>>
 In light of the new expressions API, the idea of deprecating 
 QuerySet.extra() has been informally discussed in IRC and elsewhere. I 
 wonder if there is consensus to mark extra() as "unmaintained" and to 
 suggest filing feature requests for functionality that can be performed 
 through extra() but not through other existing QuerySet methods? There are 
 at least several tickets (examples below) of edge cases that don't work 
 with extra(). It seems like a waste of time to leave these tickets as 
 accepted and to triage new issues with extra() if they won't be fixed.

 https://code.djangoproject.com/ticket/24142
 https://code.djangoproject.com/ticket/19434
 https://code.djangoproject.com/ticket/12890

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at http://groups.google.com/group/django-developers.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/django-developers/6e1be326-3b17-49ca-accf-03eec5ad41ef%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3d1f957c-318b-4a28-8158-442f2e6cbb55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: future of QuerySet.extra()?

[Anssi Kääriäinen]

You can annotate raw SQL with expressions. I think the only case that can't 
be done with expressions is addition of extra tables to the query. I am 
certain we will get a solution to this too in future releases.

 - Anssi

On Friday, July 31, 2015 at 9:01:45 PM UTC+3, Andres Osinski wrote:
>
> Would the expressions API be able to define fields that are aggregations 
> of complex foreign relationships? I keep using it when I need fields that 
> are the result of weird aggregations over multiple JOINs with some unusual 
> stuff like CASE fields and subqueries.
>
> On Fri, Jul 31, 2015 at 2:58 PM, Tim Graham  > wrote:
>
>> In light of the new expressions API, the idea of deprecating 
>> QuerySet.extra() has been informally discussed in IRC and elsewhere. I 
>> wonder if there is consensus to mark extra() as "unmaintained" and to 
>> suggest filing feature requests for functionality that can be performed 
>> through extra() but not through other existing QuerySet methods? There are 
>> at least several tickets (examples below) of edge cases that don't work 
>> with extra(). It seems like a waste of time to leave these tickets as 
>> accepted and to triage new issues with extra() if they won't be fixed.
>>
>> https://code.djangoproject.com/ticket/24142
>> https://code.djangoproject.com/ticket/19434
>> https://code.djangoproject.com/ticket/12890
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/6e1be326-3b17-49ca-accf-03eec5ad41ef%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Andrés Osinski
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1c66bb8d-ee9b-4215-aec8-1e63c844%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Why Django is making migrations for external apps?

[Anssi Kääriäinen]

To me it seems the choice of doing migrations for app/model should be done
by the author of the app.

How about model/app level setting migrations_mode, where the choices are
auto, manual and never. Auto is the defaul and behaves like current Django.
Manual means that you'll have to mention the app explicitly for
makemigrations. This would be a good default for Django's apps and for 3rd
party apps. Finally never disables migrations for the app entirely, both
creation and applying of migrations.

 - Anssi

On Sunday, July 19, 2015, Shai Berger  wrote:

> On Friday 17 July 2015 19:48:30 Carl Meyer wrote:
> > On 07/17/2015 10:38 AM, Marcin Nowak wrote:
> > > Sounds enough good. One important thing for me is that I have many of
> > > external apps and few in project managed by me. Reconfiguring most of
> > > apps just to tell Django "leave them alone" sounds like an overhead.
> > > That's why I would like to tell Django "here are my apps, manage them
> > > please".
> >
> > I think we need to clarify what this proposed feature does, exactly.
> > James' AppConfig proposal (with the parallel to unmanaged models) sounds
> > to me like a "migrations should ignore this app entirely" flag (which
> > would impact both `makemigrations` and `migrate`). Such a flag wouldn't
> > be useful for any reusable app that has models (and thus migrations).
> > I'm not opposed to this, though (much like unmanaged models themselves)
> > I don't think it'll be frequently used. It's basically a way to opt out
> > of migrations entirely.
> >
>
> Agreed.
>
> > It seems like what Marcin wants is more of a "don't ever create new
> > migrations for this app, but go ahead and use its existing ones
> > normally" flag (thus something that affects only `makemigrations`, not
> > `migrate`).
>
> That's what I'd want too.
>
> > Personally I don't really see much use case for this feature
> > except as a workaround for reusable apps that generate spurious new
> > migrations based on settings changes
>
> I've also seen it with apps that use "EmailField" without specifying
> explicit
> length (which I'd expect to be the common way to use it...)
>
> > (e.g. the dynamic-choices case).
> > Normally an app shouldn't ever generate a new migration from
> > `makemigrations` unless you've modified its models code. So I'd be more
> > in favor of addressing the actual problems causing spurious new
> migrations.
> >
>
> That's "the best getting in the way of the good". Solving the problem at
> the
> source is best, and making reusable apps more robust in this sense is a
> great
> cause, but it shouldn't stop us from putting workarounds in the hands of
> users.
>
> > Also, FWIW, the latter is already fairly easy to implement yourself by
> > overriding `makemigrations` in your project and giving it a default list
> > of app names if none are provided.
> >
>
> I don't think that's a good answer for users (and it can't even be
> implemented
> as a reusable app without adding a setting).
>
> Shai.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1HfFTEEmm7m1UDVhcTX2xiGfjRpN105yqq0h6FY2zHXpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

ForeignObject defaults to virtual_only=False?

[Anssi Kääriäinen]

If I recall correctly the real significance of virtual_only is that fields
added with this flag are copied to child models. Generic relations need
this as they need to know the model they are used on. Normal fields are not
copied, instead they are fetched directly from the parent model when _meta
is queried.

So, it might be possible to rename virtual_only to something like
copy_to_childs, and then virtuality of a field would be determined only on
it having a column.

Even better would be if fields had a get_migration_operations(from_state,
to_state) method. Then migrations wouldn't have to know anything else than
to call that method. This would also be very useful method for 3rd party
fields.

 - Anssi

On Wednesday, July 15, 2015, Tim Graham > wrote:

> ForeignObject defaults to virtual_only=False [1]. This seems
> counterintuitive as my understanding is that ForeignObject fields are
> virtual (i.e. they don't directly map to a database column).
>
> I am testing Andrew's patch to allow disabling migrations in tests [2] and
> ran into a problem in Django's test suite because of this. For models with
> ForeignObject fields [3], these fields appear in model._meta.local_fields
> and so are added to migrations which raises an exception because
> field.column is None. I tried changing ForeignObject to default to
> virtual_only=True (and kept ForeignKey virtual_only=False), but there were
> non-trivial test failures, so I wanted to check my understanding before
> going down a wormhole.
>
> My experimental branch is https://github.com/django/django/pull/4997
>
> [1]
> https://github.com/django/django/blob/b356dc4e07915521db1e768d6357e3d982877a6e/django/db/models/fields/related.py#L1830-L1832
> [2] https://github.com/django/django/pull/4986
> [3]
> https://github.com/django/django/blob/b356dc4e07915521db1e768d6357e3d982877a6e/tests/foreign_object/models.py#L60-L68
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/e88f0124-ba52-4b21-8825-ae73824f1040%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1HYiBjRHw6%2Br3ibmBNQH%3DGx3wp48LhUA5akt1msdP2bhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: 1.9 release planning

[Anssi Kääriäinen]

+1 to not changing already planned version numbers.

 - Anssi

On Tue, Jun 23, 2015 at 1:24 PM, Aymeric Augustin
 wrote:
> I'm against making changes to the version numbers we've already planned for
> and also against 1.10, 1.11 etc. version numbers.
>
> Such numbers can easily break version checks that don't expect this case.
> There's lots of code in the wild with version checks, some of which will
> probably behave incorrectly.
>
> Besides, honestly, 1.10 is just ugly :-)
>
> --
> Aymeric.
>
>
>
> 2015-06-23 9:43 GMT+02:00 Marc Tamlyn :
>>
>> +1 to 1.11
>>
>> It was an arbitrary decision not to use 2.0 in the first place because we
>> were not going to do special version numbers. Now Y.0 is a special version
>> (dropping backwards compat after the LTS) it makes more sense.
>>
>> Marc
>>
>> On 22 Jun 2015 19:28, "Tim Graham"  wrote:
>>>
>>> Done that in https://github.com/django/django/pull/4908
>>>
>>> On Monday, June 22, 2015 at 1:35:19 PM UTC-4, Loïc Bistuer wrote:

 We can just leave RemovedInDjango20Warning as an alias (not a subclass)
 to PendingDeprecationWarning in 1.8. As long as we don't refer to it in the
 rest of the codebase it isn't ambiguous.

 --
 Loïc

 > On Jun 23, 2015, at 00:21, Collin Anderson  wrote:
 >
 > People import the warning in order to silence them, right?
 >
 > On Monday, June 22, 2015 at 1:19:51 PM UTC-4, Markus Holtermann wrote:
 > +1 -- Going with 1.8, 1.9, 1.10, 1.11 (LTS), 2.0 sounds like a solid
 > plan. I don't think any of the (Pending)DeprecationWarnings are much
 > of
 > a public API. I've never seen them in the wild.
 >
 > /Markus
 >
 > On Mon, Jun 22, 2015 at 11:20:52AM -0400, Michael Manfre wrote:
 > >+1. I really don't like the idea of 2.x being odd.
 > >
 > >On Mon, Jun 22, 2015 at 10:33 AM, Loïc Bistuer 
 > >wrote:
 > >
 > >> Just when we thought we had a winner... I'd like to make a final
 > >> proposal.
 > >>
 > >> Instead of delaying adoption of SemVer to 3.0 we could do so in 2.0
 > >> by
 > >> introducing the 1.10 and 1.11LTS releases.
 > >>
 > >> The upside is that the new policy applies right away and we avoid
 > >> the
 > >> oddball 2.0 and 2.1 releases.
 > >>
 > >> It's much less invasive than the previous idea of renaming 1.9 to
 > >> 2.0, but
 > >> it still requires renaming PendingDeprecationWarnings in 1.8, and
 > >> both
 > >> warnings in 1.9.
 > >>
 > >> What do you think?
 > >>
 > >> --
 > >> Loïc
 > >>
 > >> > On Jun 17, 2015, at 08:47, Josh Smeaton 
 > >> > wrote:
 > >> >
 > >> > I'm also +1 on the proposal as it stands, and neutral on when the
 > >> > semver
 > >> versioning should begin.
 > >> >
 > >> > On Wednesday, 17 June 2015 05:03:47 UTC+10, Michael Manfre wrote:
 > >> > I'm +1 on the Google doc proposal and like Markus, I support
 > >> > relabeling
 > >> 1.9 to 2.0 to line the versions up with the new paradigm without
 > >> the X.1
 > >> LTS oddball.
 > >> >
 > >> > Regards,
 > >> > Michael Manfre
 > >> >
 > >> > On Tue, Jun 16, 2015 at 12:34 PM, Collin Anderson
 > >> > 
 > >> wrote:
 > >> > I also like the gdoc as it is. (1.8 LTS, 1.9, 2.0, 2.1 LTS, 3.0,
 > >> > 3.1,
 > >> 3.2 LTS, 4.0, etc.) LTS is the final of a major version number, and
 > >> we
 > >> sacrifice a little bit of strict semver, but it give some nice
 > >> meaning to
 > >> the version numbers.
 > >> >
 > >> >
 > >> > On Tuesday, June 16, 2015 at 12:22:44 PM UTC-4, Carl Meyer wrote:
 > >> > Thanks Loïc,
 > >> >
 > >> > On 06/16/2015 01:15 AM, Loïc Bistuer wrote:
 > >> > > I've attempted to summarize the history of this thread. Note
 > >> > > that I
 > >> > > marked as +1 any generally positive feedback towards a given
 > >> > > proposal, please correct if you feel misrepresented.
 > >> > >
 > >> > [snip]
 > >> > >
 > >> > > # Third iteration:
 > >> > >
 > >> > > 5/ Switching to Semantic Versioning
 > >> > >
 > >> > > Donald mentioned SemVer on IRC a few days ago. Since then
 > >> > > various
 > >> > > proposal were made to reconcile it with our release policy. So
 > >> > > far
 > >> > > Collin, Carl, Loïc, Tim, and Josh have expressed positive
 > >> > > feedback to
 > >> > > various proposals in that direction but I don't think we have
 > >> > > yet
 > >> > > reached consensus on a specific one. Tim updated the Google doc
 > >> > > to
 > >> > > reflect my latest proposal, so including me that's 2 formal +1
 > >> > > for
 > >> > > it, but I'd say we should wait for at least a couple more votes
 > >> > > before taking it to the technical board.
 > >> > >
 > >> > > Refs: - http://semver.org/ - Carl's analysis
 > >> > >
 > >>
 > >> htt

Re: 1.9 release planning

[Anssi Kääriäinen]

+1 to Collin's release schedule.

This schedule should make it extremely easy to support "develop using
latest release, maintain using latest LTS". With the above schedule if
you started with 1.8 you are already on LTS. If you start with 1.9,
you should have an easy upgrade path all the way till 2.1 which is
LTS. Also upgrading from 1.8 to 2.1 should be easy enough if you
didn't use any deprecated features when running on 1.8.

 - Anssi

On Thu, Jun 11, 2015 at 12:06 AM, Tim Graham  wrote:
> Yep, I think Collin's schedule is it. I'm happy with that option and 3rd
> party apps shouldn't need to add any compatibility shims to support 2
> releases -- they just need to ensure they aren't using any deprecated APIs.
>
>
> On Wednesday, June 10, 2015 at 3:48:39 PM UTC-4, Collin Anderson wrote:
>>
>> Hi Tim,
>>
>> What I mean is we could still make it easy to support both LTS releases,
>> even if we drop features deprecated in 1.8 before the next LTS according to
>> the normal release schedule. Right? Because apps wouldn't need to use those
>> deprecated features to support both 1.8 and 2.1. We could drop them in 2.0
>> like normal?
>>
>> I'm trying to lessen the increased maintenance burden of Loic's proposal
>> while still making it possible to easily support both LTS releases.
>>
>> > For "maintenance costs" I am not worried about supported deprecated APIs
>> > in old releases, only how long they stay around in master as that could be 
>> > a
>> > barrier to innovation.
>> Right, so the cost would be an extra 8 months before removing features
>> deprecated in 1.9 from master.
>>
>> Thanks,
>> Collin
>>
>> On Wednesday, June 10, 2015 at 2:09:13 PM UTC-4, Tim Graham wrote:
>>>
>>> Collin, I'm not following your reasoning about why dropping features
>>> deprecated in one LTS (e.g. 1.8) in the next LTS (e.g. 2.1; I think you made
>>> a typo in your timeline putting it next to 2.0?) will make it possible to
>>> easily support both LTS releases? That's the purpose of Loic's proposal I
>>> believe.
>>>
>>> For "maintenance costs" I am not worried about supported deprecated APIs
>>> in old releases, only how long they stay around in master as that could be a
>>> barrier to innovation.
>>>
>>> On Wednesday, June 10, 2015 at 1:54:53 PM UTC-4, Collin Anderson wrote:

 On Friday, May 8, 2015 at 12:12:37 PM UTC-4, Carl Meyer wrote:
>
> And there is a significant added maintenance cost to my proposal
> compared to yours. Dropping deprecated APIs in the release after an LTS
> means we still have to support those APIs for three more years
> (possibly
> for four or five years after they were first deprecated). Dropping them
> _in_ the LTS release shortens that window drastically.


 If we release every 8 months, that means we normally support deprecated
 features for 2 years. If we maintained LTS compatibility, then, yes, that
 would mean "supporting" the APIs for more than 5 years after deprecation.
 But to be clear, most of that time it's only security support for those
 APIs, and the APIs are long gone from master. Right?

 Thanks,
 Collin


 On Wednesday, June 10, 2015 at 1:37:30 PM UTC-4, Collin Anderson wrote:
>
> Hi All,
>
> Here are some thoughts in reply to some of the comments in the
> django-compat thread. I don't stick to the LTSs myself, but I've helped
> maintain apps that have 1.4 compatibility.
>
> On Tuesday, June 9, 2015 at 7:05:45 PM UTC-4, Loïc Bistuer wrote:
>>
>> 1.8 (LTS): No features dropped.
>> 1.9: Dropped features deprecated in 1.4, 1.5, 1.6, 1.7
>> 2.0: No features dropped.
>> 2.1 (LTS): No features dropped.
>> 2.2: Dropped features deprecated in 1.8, 1.9, 2.0
>
>
> I'd propose something slightly different, that's very close to our
> current deprecation timeline:
> 1.8 (LTS): No features dropped.
> 1.9: Dropped features deprecated in 1.5, 1.6, 1.7
> 2.0: Dropped features deprecated in 1.8
> 2.1 (LTS): No features dropped.
> 2.2: Dropped features deprecated in 1.9, 2.0
> 2.3: Dropped features deprecated in 2.1
>
> Seems to me features deprecated in an LTS are fair game to disappear in
> the next LTS. This allows us to remove "dotted paths in reverse() and 
> url()"
> because that's deprecated in 1.8.
>
> If we can guarantee compatibility between LTSs, I think that would be a
> huge win, at the cost of extending the removal of some deprecated features
> by one release (8 months).
>
> Collin
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.

Re: URL dispatcher API

[Anssi Kääriäinen]

One important thing to keep in mind is reverse() performance.

While it isn't important that the prototype is fast, making sure the API
doesn't limit performance for the most important operations must be checked.

At least my experience is that URL resolving performance isn't that
important, but having fast reverse() is really important. The reason is
that single request typically has just one resolve operation, and thus the
amount of time used for resolving tends to end up in the noise. But large
list/table views can have up to thousands of reverse() operations, when
there are multiple urls per row.

I believe we can have clean, fast and extensible URL resolving system, so
I'm not too worried about this. Still, lets make sure performance won't be
a problem.

 - Anssi

On Monday, May 4, 2015, Marten Kenbeek  wrote:

> Hi all,
>
> I'd like to discuss the API I'm proposing for the new url dispatcher I'm
> working on. I'll try to explain the API and some of the rationale behind it.
>
> There is a working proof-of-concept at
> https://github.com/knbk/django/tree/url_dispatcher. Currently, all the
> names are chosen as not to conflict with the old implementation. A short
> description can be found at
> https://gist.github.com/knbk/96999abaab4ad4e5f8f9, which also contains a
> link to an example url configuration.
>
> My main goal was to create a simple, extensible API. In the old API, about
> 90% of the work was done in the `resolve()`, `_reverse_with_prefix()` and
> `populate()` of the RegexURLResolver. This made for a tightly coupled
> design that was almost impossible to extend.
>
> My starting point was the RegexURLResolver and RegexURLPattern. Both have
> a regex constraint that can match the current path and extract arguments.
> The former can then pass the remainder of the path to its list of resolvers
> and patterns; the latter can return a ResolverMatch containing the callback
> specified by that pattern. I've kept this distinction, with the `Resolver`
> and `View` classes. The change of name from `Pattern`  to `View` is because
> the `View` object has a bit more logic that determines how the view is
> called. It is a thin, callable wrapper that can optionally decorate the
> actual view function with a set of decorators passed down from parent
> resolvers, and when overwritten, can do some more processing before or
> after the view function is called.
>
> The hard-coded dependence on a regex pattern has been abstracted to a
> Constraint object. Each Resolver and View has a (set of) Constraint
> object(s), that can match the current url and extract arguments. A
> RegexPattern that simply mimics the old behaviour will be available, but
> other implementations, such as a Constraint that matches the request's host
> or method, are easily provided. A Constraint can also reverse a set of
> arguments to a partial url. That means that the full set of constraints
> used to match an url to a view, together with a suitable set of arguments,
> can be reversed to the url itself.
>
> The main strength of a Constraint is that it can contain very specific
> logic about its arguments. For example, a Constraint may know that it
> resolves to a Model's primary key. If it then receives a Model instance of
> that particular type, it will know how to reverse that model instance to a
> valid string-based partial url, so that it can later be resolved to match
> the same object. It could also e.g. infer the regex pattern from the
> field's type.
>
> There's one final piece to the puzzle: the URL object. This is the state
> of the url in the process of resolving or reversing an url. It's a two-way
> street: when resolving, it starts out as a full path, and the Constraints
> chip away at the path, while the set of constraints and extracted argument
> grows. When reversing, it starts out as a set of constraints and arguments,
> and reconstructs the partial urls from those constraints and arguments
> until a full url path is reconstructed. It shifts some of the logic from
> the Resolver to the URL, so that it is easier to extend the Resolver. It is
> also a simple container that allows any Constraint access to the full
> request. Last but not least, it allows to dynamically build an url against
> the current request. This is useful if e.g. a constraint matches a
> different subdomain than the current request, so that a link automatically
> points to the right domain.
>
> I'm looking forwards to your feedback.
>
> Thanks,
> Marten
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/m

Re: An easier path to upgrade from one LTS release to another

[Anssi Kääriäinen]

On Thu, May 7, 2015 at 11:34 AM, Marc Tamlyn  wrote:
> I'm not sure that would be a wise move - for people who don't keep up with
> deprecation warnings but otherwise move one version at a time it would make
> upgrading from an LTS to the following release 3 times harder than normal,
> encouraging stagnation.

The other side of this coin is that you get painless upgrades to the
latest LTS from the latest stable version. So, with a bit of
exaggeration, one could say that our current model encourages
stagnation to non-LTS versions.


 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1EC%2B9nakK2MjW%2B5mV3Nyt10XbQi2vTw6_WCfDwQ2GQLJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: An easier path to upgrade from one LTS release to another

[Anssi Kääriäinen]

One approach worth consideration is dropping deprecated features only
after next LTS. This would mean that any non-deprecated feature that
is available in current LTS will also be available (but possibly
deprecated) in the next LTS. This should make a development model
where code is developed using latest stable release, and supported
using latest LTS a lot easier.

I don't belileve we can offer an official LTS-to-LTS upgrade guide
(maintaining one is simply too much work). You can of course try to do
that, but you'll likely get a better result by upgrading one point
release at a time.

 - Anssi

On Thu, May 7, 2015 at 11:04 AM, Abdullah Esmail
 wrote:
> Hello,
> I apologize if this has been discussed already. I searched the topics, but
> didn't find anything.
> First, I'd like to thank all of you for the truly amazing framework. I've
> been using django since 1.0 and it made my life much more easier and
> enjoyable and _stable_.
>
> The reason why I love django is that stability is a high priority,
> backward-compatibility is well maintained and clear, crystal-clear and
> extra-detailed documentation, the "batteries-included" aspect, based on
> python, and the steady and stable evolution of the framework. Everything is
> well put together in an almost-perfect package. Thank you.
>
> To my main issue, have you explored the possibility of creating a direct
> path from one LTS release to the next?
> I think the "official" way of doing this now is to go through all releases
> in-between (1.4 -> 1.5 -> 1.6 -> 1.7 -> 1.8).
> It would be really great if there was a direct LTS-to-LTS path (1.4 -> 1.8).
>
> I'm not sure if the new system check framework can play a role here to make
> this easier than before.
> This might add a whole new layer of complexity and extra work, but I believe
> it's worth it.
> If it makes any difference, I don't mind helping with this and actually
> bringing this feature to life.
> I'm not that familiar with the internals of django, but I'm willing to learn
> and get my hands dirty.
>
> I guess I should ask, is there a good reason why there is no direct upgrade
> path between LTS releases?
> Do you guys think it's not worth the extra effort and the current way is an
> acceptable trade-off?
>
> Thank you,
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/96e73e82-93fe-4ef7-92c4-ba67abb35b37%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1E-F8i4_%2BYLH6KRzWMvuJEmr8geDDU9mUY9MFRudhLw0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: New function to sort words with accents.

[Anssi Kääriäinen]

There is a ticket for collations support. See
https://code.djangoproject.com/ticket/21181. I think a Collate
expression would be very welcome addition.

On Wed, May 6, 2015 at 8:52 AM, Aymeric Augustin
 wrote:
> If you're using PosgreSQL, django.contrib.postgres provides such a function.
>
> --
> Aymeric.
>
> Le 6 mai 2015 à 02:52, Paulo Maciel  a écrit :
>
> I am returning the query Something.objects.all().order_by(Lower('name'))
> but the order is not respecting names that start with accent, causing these
> results appear at the end of ordering. We need a function to ignore accents
> and order correctly names with accents, as well as function Lower treats the
> results.
>
> Currently:
>
> arroz
> baleia
> dado
> limão
> xerox
> água
> ótimo
>
> As it should be:
>
> água
> arroz
> baleia
> dado
> limão
> ótimo
> xerox
>
>
> We need a function like it to fix the problem:
> Person.objects.all().order_by(Unaccent('name'))
>
> This is a priority issue because it affects thousands of Brazilians and
> other nationalities who use Portuguese.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/27620089-a89e-4e13-9576-637e37395754%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/F4794B20-EA76-4A41-A4CD-8D2DA473E82F%40polytechnique.org.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1EdrJDDy35NdD%2BRghU4_k8VsJ45CQfXOr3CTT1pjCVv%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Refactor year, month, day lookups?

[Anssi Kääriäinen]

Maybe .values(month='pub_date__month') could work? Otherwise the month
would be the full lookup string.

On Wednesday, March 25, 2015, Alexander Hill  wrote:

> In the dicts produced by a values query like that, what key would the
> result be found under? The full lookup string i.e. 'pub_date__month'?
>
> On Thu, Mar 26, 2015 at 2:36 AM, Anssi Kääriäinen  > wrote:
>
>> There remains a bit of work to do to make transforms usable everywhere in
>> the ORM. The example should work with just
>> .values('pub_date__month').annotate(...)
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/django-developers/WYWrQkBJ2hs/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to
>> django-developers+unsubscr...@googlegroups.com
>> 
>> .
>> To post to this group, send email to django-developers@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/1b62a33e-f0b6-4153-bbc3-2e48ebdb7ac7%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CA%2BKBOKx%2BOYZu%3D9OSKRFMgzgHvQV%2B54yrhKC86jF%2BT9%2B8Mi2e%2Bw%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CA%2BKBOKx%2BOYZu%3D9OSKRFMgzgHvQV%2B54yrhKC86jF%2BT9%2B8Mi2e%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1H4U3RGJCYkUMGh-krFfXe8e97xM0vvMOyiziV%3D%2ButtVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Refactor year, month, day lookups?

[Anssi Kääriäinen]

There remains a bit of work to do to make transforms usable everywhere in the 
ORM. The example should work with just .values('pub_date__month').annotate(...)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1b62a33e-f0b6-4153-bbc3-2e48ebdb7ac7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Overlap between Func and Transform

[Anssi Kääriäinen]

Transforms are just single argument expressions available using lookup strings. 
The biggest complication is that Transforms have slightly different API to 
expressions.

I don't see a reason why transforms couldn't be a special case of expression. 
There are some API incompabilities, but they should be solvable.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/09fc315c-c446-425d-b7a5-266030b996f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Refactor year, month, day lookups?

[Anssi Kääriäinen]

+1 for implementing these. They weren't part of the original patch because the 
patch was a large one already without these.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4e98e7d1-f4d5-4c5d-af11-cb787edb407a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.