Hey Anssi, Just pushed test_lookup_exclude and test_lookup_exclude2, which passed successfully. Can you please review if they cover exactly described case?
Thanks, Alexey On Monday, October 19, 2015 at 2:28:42 PM UTC+3, Anssi Kääriäinen wrote: > > I did similar work in https://github.com/django/django/pull/4801. I > aimed for 1.9, but had to defer the work due to problems with > .exclude() filtering. > > The problem is that if you query with > .exclude(LessThan(F('friends__age'), 30)), you'll get different > results than with .exclude(friends__age__lt=30). The latter does a > subquery while the expression version isn't capable of that. > Unfortunately, fixing this is a bit nasty. I don't see any other way > than some solution like https://github.com/django/django/pull/4385 - > that is we prewalk the expression tree and decide which parts need to > be pushed in to a subquery. This should work really nice with > expressions, too. > > I haven't taken a closer look on your approach, so I can't say which > one we should use as basis for future work once we have solved the > subquery problem. > > - Anssi > > On Mon, Oct 19, 2015 at 2:09 PM, Josh Smeaton <[email protected] > <javascript:>> wrote: > > I think you forgot the PR link Alexey. I'd like to have a look at the > > changes you've made. I'm not sure about the extensions to F() just yet, > I'm > > hoping they aren't actually needed for your patch to work, but will need > to > > review to be sure. > > > > Cheers > > > > > > On Monday, 19 October 2015 21:49:37 UTC+11, Alexey Zankevich wrote: > >> > >> Hi all, > >> > >> Here is a raw pull request, allowing lookup instances passing to filter > >> method > >> There are two new test cases working: > >> > >> 1. Passing lookup with F object: > >> Article.objects.filter(GreaterThan(F('id'), Value(5))) > >> 2. And passing lookup with transforms: > >> Article.objects.filter(Contains(Upper(Lower(Upper(F('headline')))), > 'ARTICLE > >> 5')) > >> > >> All the existing tests are passing successfully, by the way (at least > for > >> SQLite backend). > >> > >> Several comments: > >> > >> F expressions became real expressions (Expression subclass). > >> Expression got a new method - get_source_f_object, which walks through > >> children classes and find if it has F object passed. In that case > >> output_field-related code should be postponed, until compiler resolves > it > >> (which cannot be done during lookup init time). > >> some code related to query.build_filter method (responsible to > >> lookups/transform parsing, and calculating joins), extracted into > separate > >> helper classes - QueryKeywordLookupHelper and QueryObjectLookupHelper. > They > >> have common code base, but some of methods work in a different way. > >> > >> Except review, it would be really great if someone help with writing > test > >> cases (and I'll get them working in case of any issues). Also, there is > a > >> workaround field=None -> isnull=True for oracle database, can anybody > check > >> if it is working correctly? > >> > >> Regards, > >> Alexey > >> > >> > >> On Thursday, October 1, 2015 at 11:38:39 AM UTC+3, Anssi Kääriäinen > wrote: > >>> > >>> Lets concentrate on making expressions in filter calls and lookups as > >>> expressions reality. When we have done that we can try out different > >>> syntax approaches in 3rd party apps. Finally, after we have field > >>> tested the approaches, we can merge something in to Django. > >>> > >>> - Anssi > >>> > >>> On Thu, Oct 1, 2015 at 10:49 AM, Alexey Zankevich > >>> <[email protected]> wrote: > >>> >> I think the `F('name').decode('utf-8')` syntax is a good candidate > for > >>> >> core. > >>> > > >>> > When Django supports expression lookups, it will be possible to > build > >>> > syntax > >>> > sugar third-party library on top of it. > >>> > For example, it will be possible to implement dotted syntax for F > >>> > objects: > >>> > > >>> > UserModel.objects.filter(Decode(F.user.name, 'utf-8') == > u'Бармаглот') > >>> > > >>> > alternatively it can be even combined with your syntax > >>> > > >>> > UserModel.objects.filter(F.user.name.decode('utf-8') == > u'Бармаглот') > >>> > > >>> > However, the second solution will be more complicated from technical > >>> > point > >>> > of view (as will require lazy object, collecting all the func or > >>> > transform > >>> > calls), also it will require to be really careful with transform > names > >>> > not > >>> > to conflict with possible field names. > >>> > For example, it's hardly possible to have "decode" field in the > model, > >>> > but > >>> > easily - "date". > >>> > > >>> > > >>> > On Thursday, October 1, 2015 at 10:18:40 AM UTC+3, Loïc Bistuer > wrote: > >>> >> > >>> >> > On Oct 1, 2015, at 13:38, Anssi Kääriäinen <[email protected]> > >>> >> > wrote: > >>> >> > > >>> >> > +1 to not requiring all transforms to handle __underscore__ > syntax. > >>> >> > >>> >> +1 > >>> >> > >>> >> > I think what we want to do is allow users choose which syntax > they > >>> >> > prefer. The idea is that Django will support both > >>> >> > JSONExtract('data', > >>> >> > path=['owner', 'other_pets', 0, 'name']) and > >>> >> > data__owner__other_pets__0__name out of the box. > >>> >> > > >>> >> > We will also make it possible to support callable transforms. For > >>> >> > example: > >>> >> > filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот'))) > >>> >> > is equivalent to > >>> >> > filter(F('name').decode('utf-8') == Value(u'Бармаглот')) > >>> >> > > >>> >> > The callable transforms syntax will not be a part of Django, but > it > >>> >> > will be possible to create an extension for this (it is actually > >>> >> > surprisingly easy to do once we have support for expressions in > >>> >> > filter). > >>> >> > >>> >> I'm pretty convinced we need a better API / sugar as part of core > in > >>> >> the > >>> >> long run. > >>> >> > >>> >> The `filter(Exact(Decode(F('name'), 'utf-8'), > Value(u'Бармаглот')))` > >>> >> syntax is not pythonic and very hard to decipher, and we've reached > >>> >> the > >>> >> limits of our historical double underscore syntax (doesn't support > >>> >> multiple > >>> >> arguments, limited to ascii, etc.). > >>> >> > >>> >> I think the `F('name').decode('utf-8')` syntax is a good candidate > for > >>> >> core: > >>> >> > >>> >> - It's a small extension to the existing F objects, so it's easy to > >>> >> grasp > >>> >> for existing Django users. > >>> >> - It's just a thin sugar on top of the canonical API. > >>> >> - All the required machinery being already in place (lookup & > >>> >> transform > >>> >> registration) the change to Django is minimal. > >>> >> > >>> >> > - Anssi > >>> >> > > >>> >> > On Thu, Oct 1, 2015 at 4:00 AM, Josh Smeaton <[email protected]> > > >>> >> > wrote: > >>> >> >> No, not all Lookups or Transforms are required to handle > >>> >> >> __underscore__ > >>> >> >> syntax. The entire point of supporting object based lookups is > to > >>> >> >> handle > >>> >> >> cases that get more complex than a single argument transform or > a > >>> >> >> left > >>> >> >> and > >>> >> >> right hand side lookup. > >>> >> >> > >>> >> >> In particular, I think your Decode(utf8) example is a good one. > It > >>> >> >> shows > >>> >> >> that you could maybe shoehorn multiple arg transforms into the > >>> >> >> __underscore__ api, but it wouldn't be entirely obvious how to > do > >>> >> >> so. > >>> >> >> You'd > >>> >> >> probably need to register partial transformations for each > encoding > >>> >> >> you > >>> >> >> wanted to support. The contrib.postgres module has (from memory) > >>> >> >> KeyTransform classes that do a similar thing. > >>> >> >> > >>> >> >> Cheers > >>> >> >> > >>> >> >> > >>> >> >> On Thursday, 1 October 2015 02:13:41 UTC+10, Alexey Zankevich > >>> >> >> wrote: > >>> >> >>> > >>> >> >>> I'll try to turn lookups into expression (will use master > branch). > >>> >> >>> I also have a question. Meanwhile the old query syntax with > >>> >> >>> underscores is > >>> >> >>> pretty good for simple queries (like > >>> >> >>> Model.objects.filter(name='Bob'), > >>> >> >>> it > >>> >> >>> gets really ugly for parametrized calls, a new JSONField is a > good > >>> >> >>> example > >>> >> >>> of that: > >>> >> >>> > >>> >> >>>>>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy') > >>> >> >>> [<Dog: Rufus>] > >>> >> >>> > >>> >> >>> It will get even more messy if we want to pass a string as a > >>> >> >>> second > >>> >> >>> param > >>> >> >>> of the func. > >>> >> >>> > >>> >> >>> ex.: > >>> >> >>> > >>> >> >>> 1. filter(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')) # > <- > >>> >> >>> neat > >>> >> >>> 2. filter(name__decode__utf8=u'Бармаглот') # <- ?? ambiguous > and > >>> >> >>> not > >>> >> >>> nice > >>> >> >>> at all > >>> >> >>> > >>> >> >>> So question - is it implied all the funcs, transforms and > lookups > >>> >> >>> to > >>> >> >>> have > >>> >> >>> underscore-based equivalent? It can affect the final > >>> >> >>> implementation > >>> >> >>> pretty > >>> >> >>> much. In my opinion underscore-based equivalent should not be > >>> >> >>> really > >>> >> >>> required for funcs (the problem doesn't seem to affect > transforms > >>> >> >>> as > >>> >> >>> they > >>> >> >>> will not accept multiple params according to another thread). > >>> >> >>> > >>> >> >>> Thanks, > >>> >> >>> Alexey > >>> >> >>> > >>> >> >>> > >>> >> >>> On Wednesday, September 30, 2015 at 9:19:51 AM UTC+3, Anssi > >>> >> >>> Kääriäinen > >>> >> >>> wrote: > >>> >> >>>> > >>> >> >>>> I don't think we need split-to-subq support for Lookups before > we > >>> >> >>>> make > >>> >> >>>> them expressions. Lookups as expressions are usable outside > >>> >> >>>> .filter(), > >>> >> >>>> and we need the split-to-subq support only in > >>> >> >>>> .filter(expression). > >>> >> >>>> > >>> >> >>>> - Anssi > >>> >> >>>> > >>> >> >>>> On Wed, Sep 30, 2015 at 8:46 AM, Josh Smeaton > >>> >> >>>> <[email protected]> > >>> >> >>>> wrote: > >>> >> >>>>> I'm mixing my versions, sorry to those following along. 1.9 > has > >>> >> >>>>> just > >>> >> >>>>> reached > >>> >> >>>>> alpha. Lookups as Expressions should be doable for 1.10 which > >>> >> >>>>> master > >>> >> >>>>> is > >>> >> >>>>> currently tracking. > >>> >> >>>>> > >>> >> >>>>> Cheers > >>> >> >>>>> > >>> >> >>>>> > >>> >> >>>>> On Wednesday, 30 September 2015 15:31:24 UTC+10, Josh Smeaton > >>> >> >>>>> wrote: > >>> >> >>>>>> > >>> >> >>>>>> The alpha for 1.10 has already been cut, and I'm not sure > that > >>> >> >>>>>> the > >>> >> >>>>>> kinds > >>> >> >>>>>> of changes needed here are appropriate to add now that the > >>> >> >>>>>> alpha is > >>> >> >>>>>> out. One > >>> >> >>>>>> could *maybe* make the argument that changing Lookup to an > >>> >> >>>>>> Expression > >>> >> >>>>>> now > >>> >> >>>>>> rather than later is the right move considering Transforms > just > >>> >> >>>>>> underwent > >>> >> >>>>>> the same change for 1.10. Personally though, I don't think I > >>> >> >>>>>> have > >>> >> >>>>>> the > >>> >> >>>>>> time > >>> >> >>>>>> right now to do this change. I would support you if you were > >>> >> >>>>>> able, > >>> >> >>>>>> but > >>> >> >>>>>> we'd > >>> >> >>>>>> still be at the mercy of the technical board (I assume) for > >>> >> >>>>>> getting > >>> >> >>>>>> this > >>> >> >>>>>> change in for 1.10. > >>> >> >>>>>> > >>> >> >>>>>> Do you think Lookup as Expressions requires the > >>> >> >>>>>> subquery/exclude > >>> >> >>>>>> fix > >>> >> >>>>>> you > >>> >> >>>>>> mention above? I would think not -- not until we were ready > to > >>> >> >>>>>> document and > >>> >> >>>>>> support .filter(Lookup(F(), Value()). If it wasn't a > >>> >> >>>>>> requirement, > >>> >> >>>>>> it'd > >>> >> >>>>>> make > >>> >> >>>>>> the Lookup->Expression work much easier. It wouldn't even > need > >>> >> >>>>>> to > >>> >> >>>>>> be > >>> >> >>>>>> documented (other than the release notes), as it'd just be > an > >>> >> >>>>>> implementation > >>> >> >>>>>> change. > >>> >> >>>>>> > >>> >> >>>>>> Cheers, > >>> >> >>>>>> > >>> >> >>>>>> On Wednesday, 30 September 2015 14:33:14 UTC+10, Anssi > >>> >> >>>>>> Kääriäinen > >>> >> >>>>>> wrote: > >>> >> >>>>>>> > >>> >> >>>>>>> On the core ORM side we need to make > >>> >> >>>>>>> .exclude(LessThan(F('friends__age'), 30)) do a subquery. > This > >>> >> >>>>>>> way > >>> >> >>>>>>> .exclude(friends__age__lt=30) does the same thing as the > >>> >> >>>>>>> expression > >>> >> >>>>>>> version. This isn't that easy to do. If we just use > >>> >> >>>>>>> resolve_expression, then the friends relation will generate > a > >>> >> >>>>>>> join, > >>> >> >>>>>>> and then as second step do a negated filter on the joined > >>> >> >>>>>>> value. > >>> >> >>>>>>> Instead we want to detect that the LessThan expression > needs > >>> >> >>>>>>> to be > >>> >> >>>>>>> pushed in to a subquery. > >>> >> >>>>>>> > >>> >> >>>>>>> So, we need to solve: > >>> >> >>>>>>> > >>> >> >>>>>>> A) A way to ask an expression if it is referencing a > multijoin > >>> >> >>>>>>> (possible approach is to just have a method > >>> >> >>>>>>> "refs_multi_valued_relation(query)") > >>> >> >>>>>>> B) When the ORM sees an expression that is reffing a > multijoin > >>> >> >>>>>>> in > >>> >> >>>>>>> an > >>> >> >>>>>>> exclude filter, then we need to push the expression in to a > >>> >> >>>>>>> subquery. > >>> >> >>>>>>> > >>> >> >>>>>>> A) requires some new work. This shouldn't be that hard to > >>> >> >>>>>>> implement, > >>> >> >>>>>>> we just recursively ask subexpressions if they reference a > >>> >> >>>>>>> multijoin. > >>> >> >>>>>>> > >>> >> >>>>>>> Something like https://github.com/django/django/pull/4385 > will > >>> >> >>>>>>> make > >>> >> >>>>>>> B) > >>> >> >>>>>>> much easier to implement. > >>> >> >>>>>>> > >>> >> >>>>>>> I've been working on making Q-objects responsible for > >>> >> >>>>>>> resolving > >>> >> >>>>>>> themselves. See https://github.com/django/django/pull/4801. > > >>> >> >>>>>>> This > >>> >> >>>>>>> should solve 3). > >>> >> >>>>>>> > >>> >> >>>>>>> We don't seem to be missing any major parts. I (or some > >>> >> >>>>>>> volunteer) > >>> >> >>>>>>> just need to finish the PRs, and then we should be really > >>> >> >>>>>>> close to > >>> >> >>>>>>> full support for expressions in filter. > >>> >> >>>>>>> > >>> >> >>>>>>> Josh: do you think we could get Lookup as expressions in to > >>> >> >>>>>>> 1.10 > >>> >> >>>>>>> instead of 1.11? > >>> >> >>>>>>> > >>> >> >>>>>>> - Anssi > >>> >> >>>>>>> > >>> >> >>>>>>> On Wed, Sep 30, 2015 at 3:46 AM, Josh Smeaton > >>> >> >>>>>>> <[email protected]> > >>> >> >>>>>>> wrote: > >>> >> >>>>>>>> 1. Lookups should become Expressions, just as Transforms > have > >>> >> >>>>>>>> become > >>> >> >>>>>>>> Expressions. This will let us process Lookup arguments as > >>> >> >>>>>>>> Expressions > >>> >> >>>>>>>> all > >>> >> >>>>>>>> the way the way through. I think this should be a major > goal > >>> >> >>>>>>>> for > >>> >> >>>>>>>> version > >>> >> >>>>>>>> 1.11. > >>> >> >>>>>>>> > >>> >> >>>>>>>> 2. Chaining transforms is now possible since they are just > >>> >> >>>>>>>> Func > >>> >> >>>>>>>> expressions. > >>> >> >>>>>>>> Func(Func(Func('field_name'))) is no issue. > >>> >> >>>>>>>> > >>> >> >>>>>>>> 3. Sounds like an OK idea, but I haven't looked into the > >>> >> >>>>>>>> details > >>> >> >>>>>>>> enough > >>> >> >>>>>>>> to > >>> >> >>>>>>>> really comment. I do think we should create the correct > form > >>> >> >>>>>>>> as > >>> >> >>>>>>>> early > >>> >> >>>>>>>> as > >>> >> >>>>>>>> possible (parsing into a chain of Lookup/Transform > >>> >> >>>>>>>> expressions) > >>> >> >>>>>>>> so > >>> >> >>>>>>>> we > >>> >> >>>>>>>> don't > >>> >> >>>>>>>> have to do parsing in multiple places. The entry points to > >>> >> >>>>>>>> .filter() > >>> >> >>>>>>>> and > >>> >> >>>>>>>> .exclude(), or their direct counterparts in sql.query > sound > >>> >> >>>>>>>> ideal. > >>> >> >>>>>>>> Anssi has > >>> >> >>>>>>>> mentioned elsewhere that WhereNode's should only contain > >>> >> >>>>>>>> fully > >>> >> >>>>>>>> resolved > >>> >> >>>>>>>> expressions, so resolving will need to be done directly > after > >>> >> >>>>>>>> parsing > >>> >> >>>>>>>> (or > >>> >> >>>>>>>> during). > >>> >> >>>>>>>> > >>> >> >>>>>>>> Part 1 above can be started now if you have the time or > >>> >> >>>>>>>> interest. > >>> >> >>>>>>>> We > >>> >> >>>>>>>> can > >>> >> >>>>>>>> nail down the particulars of part 3 while we're solving > part > >>> >> >>>>>>>> 1. > >>> >> >>>>>>>> Part 1 > >>> >> >>>>>>>> may > >>> >> >>>>>>>> drive some of part 3. > >>> >> >>>>>>>> > >>> >> >>>>>>>> Cheers > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> On Wednesday, 30 September 2015 04:49:54 UTC+10, Alexey > >>> >> >>>>>>>> Zankevich > >>> >> >>>>>>>> wrote: > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> Here is a list of issues to solve to support explicit > >>> >> >>>>>>>>> transforms > >>> >> >>>>>>>>> and > >>> >> >>>>>>>>> lookups by filter (and exclude) methods. > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> 1. Make Lookup.__init__ signature to support > initialization > >>> >> >>>>>>>>> with > >>> >> >>>>>>>>> F > >>> >> >>>>>>>>> objects > >>> >> >>>>>>>>> or string path (e.g. GreaterThan(F('user__id'), 10) or > >>> >> >>>>>>>>> GreaterThan('user__id', 10)), not sure it's possible to > use > >>> >> >>>>>>>>> simultaneously > >>> >> >>>>>>>>> with the current approach with lhs, rhs initialization > (even > >>> >> >>>>>>>>> with > >>> >> >>>>>>>>> moving it > >>> >> >>>>>>>>> to a separate class method, e.g Lookup.build(lhs, rhs)), > so > >>> >> >>>>>>>>> I > >>> >> >>>>>>>>> assume > >>> >> >>>>>>>>> creating so-called util classes which will delegate > >>> >> >>>>>>>>> SQL-related > >>> >> >>>>>>>>> functionality to existing Lookup classes. > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> 2. Chain transforms by passing them as argument: > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> Lower(Unaccent(F('user__name))) > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> 3. Decide if Q objects shall support explicit > >>> >> >>>>>>>>> lookups/transforms > >>> >> >>>>>>>>> as > >>> >> >>>>>>>>> argument as well - it's a kind of logical step, as > without Q > >>> >> >>>>>>>>> objects > >>> >> >>>>>>>>> it will > >>> >> >>>>>>>>> not be possible to perform complicated conditions (AND, > OR, > >>> >> >>>>>>>>> NOT). > >>> >> >>>>>>>>> In that case lookup/transform parsing should be moved > from > >>> >> >>>>>>>>> QuerySet > >>> >> >>>>>>>>> object > >>> >> >>>>>>>>> to Q object - filter will take already parsed lookup > tree. > >>> >> >>>>>>>>> Example: > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> Q(user__name__lower__unaccent__icontains='Bob') will > >>> >> >>>>>>>>> internally > >>> >> >>>>>>>>> parse > >>> >> >>>>>>>>> it > >>> >> >>>>>>>>> and build next structure: > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> Q(Icontains(Lower(Unaccent(F('user__name')))), 'Bob') > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> On Sunday, August 16, 2015 at 4:18:26 PM UTC+3, Alexey > >>> >> >>>>>>>>> Zankevich > >>> >> >>>>>>>>> wrote: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Hi all, > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> This topic is related to the current ORM query syntax > with > >>> >> >>>>>>>>>> underscores. > >>> >> >>>>>>>>>> There are lots of arguing related to it, anyway it has > pros > >>> >> >>>>>>>>>> and > >>> >> >>>>>>>>>> cons. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Let's take a concrete example of querying a model: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Pros: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 1. The syntax is easy to understand > >>> >> >>>>>>>>>> 2. Can be extended with custom transforms and lookups > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> However, there are several cons: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 1. Long strings is hard to read, especially if we have > >>> >> >>>>>>>>>> fields > >>> >> >>>>>>>>>> with > >>> >> >>>>>>>>>> underscores. > >>> >> >>>>>>>>>> It's really easy to make a mistake by missing one: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > GameSession.objects.filter(user_profile__last_login_date__gte=yesterday) > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Not easy to catch missing underscore between user and > >>> >> >>>>>>>>>> profile, > >>> >> >>>>>>>>>> is > >>> >> >>>>>>>>>> it? > >>> >> >>>>>>>>>> Even > >>> >> >>>>>>>>>> though, it's not easy to say whether it should be > >>> >> >>>>>>>>>> "user_profile" > >>> >> >>>>>>>>>> attribute or > >>> >> >>>>>>>>>> user.profile foreign key. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 2. Query strings can't be reused, thus the approach > >>> >> >>>>>>>>>> violates > >>> >> >>>>>>>>>> DRY > >>> >> >>>>>>>>>> principle. > >>> >> >>>>>>>>>> For example, we need to order results by > last_login_date: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) \ > >>> >> >>>>>>>>>> .order_by('user__profile__last_login_date') > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> We can't keep user__profile_login_date as a variable as > in > >>> >> >>>>>>>>>> the > >>> >> >>>>>>>>>> first > >>> >> >>>>>>>>>> part > >>> >> >>>>>>>>>> of the > >>> >> >>>>>>>>>> expression we use a keyword argument, meanwhile in the > >>> >> >>>>>>>>>> second > >>> >> >>>>>>>>>> part - > >>> >> >>>>>>>>>> just > >>> >> >>>>>>>>>> a > >>> >> >>>>>>>>>> string. And thus we just have to type query path twice. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 3. Lookup names not natural to Python language and > require > >>> >> >>>>>>>>>> to > >>> >> >>>>>>>>>> be > >>> >> >>>>>>>>>> remembered or > >>> >> >>>>>>>>>> looked up in documentation. For example, "__gte" or > "__lte" > >>> >> >>>>>>>>>> lookups > >>> >> >>>>>>>>>> tend > >>> >> >>>>>>>>>> to be > >>> >> >>>>>>>>>> confused with "ge" and "le" due to similarity to methods > >>> >> >>>>>>>>>> "__ge__" > >>> >> >>>>>>>>>> and > >>> >> >>>>>>>>>> "__le__". > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 4. Lookup keywords limited to a single argument only, > very > >>> >> >>>>>>>>>> inconvenient > >>> >> >>>>>>>>>> when > >>> >> >>>>>>>>>> necessary to filter objects by range. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> I was thinking a lot trying to solve those issues, > keeping > >>> >> >>>>>>>>>> in > >>> >> >>>>>>>>>> mind > >>> >> >>>>>>>>>> Django > >>> >> >>>>>>>>>> approaches. Finally I came up with solution to extend Q > >>> >> >>>>>>>>>> objects > >>> >> >>>>>>>>>> with > >>> >> >>>>>>>>>> dot > >>> >> >>>>>>>>>> expression syntax: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> > GameSession.objecs.filter(Q.user.profile.last_login_date > >>> >> >>>>>>>>>>>>> >= > >>> >> >>>>>>>>>>>>> yesterday) > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Q is a factory instance for old-style Q objects. > Accessing > >>> >> >>>>>>>>>> attribute > >>> >> >>>>>>>>>> by > >>> >> >>>>>>>>>> dot > >>> >> >>>>>>>>>> returns a child factory, calling factory will > instantiate > >>> >> >>>>>>>>>> old-style Q > >>> >> >>>>>>>>>> object. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> Q > >>> >> >>>>>>>>>> <QFactory object at 0x7f407298ee10> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> Q.user.profile > >>> >> >>>>>>>>>> <QFactory object at 0x7f40765da310> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> Q(user__name='Bob') > >>> >> >>>>>>>>>> <Q: (AND: ('user__name', 'Bob'))> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> It overrides operators, so comparing factory with value > >>> >> >>>>>>>>>> returns > >>> >> >>>>>>>>>> a > >>> >> >>>>>>>>>> related > >>> >> >>>>>>>>>> Q > >>> >> >>>>>>>>>> object: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> Q.user.name == 'Bob' > >>> >> >>>>>>>>>> <Q: (AND: ('user__name', 'Bob'))> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Factory has several helper functions for lookups which > >>> >> >>>>>>>>>> aren't > >>> >> >>>>>>>>>> related > >>> >> >>>>>>>>>> to > >>> >> >>>>>>>>>> any > >>> >> >>>>>>>>>> Python operators directly: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> Q.user.name.icontains('Bob') > >>> >> >>>>>>>>>> <Q: (AND: ('user__name__icontains', 'Bob'))> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> And helper to get query path as string, which requred by > >>> >> >>>>>>>>>> order_by > >>> >> >>>>>>>>>> or > >>> >> >>>>>>>>>> select_related queryset methods: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> Q.user.profile.last_login_date.get_path() > >>> >> >>>>>>>>>> 'user__profile__last_login_date' > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> You can check implementation and more examples here > >>> >> >>>>>>>>>> https://github.com/Nepherhotep/django-orm-sugar > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> How it solves issues: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> #1. Dots hard to confuse with underscores > >>> >> >>>>>>>>>> #2. Query paths can be reused: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> factory = Q.user.profile.last_login_date > >>> >> >>>>>>>>>>>>> query = GameSession.objects.filter(factory >= > yesterday) > >>> >> >>>>>>>>>>>>> query = query.order_by(factory.get_path()) > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> #3. Not neccessary to remember most of lookup names and > use > >>> >> >>>>>>>>>> comparison > >>> >> >>>>>>>>>> operators > >>> >> >>>>>>>>>> instead. > >>> >> >>>>>>>>>> #4. Possible to use multiple keyword arguments: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> Q.user.profile.last_login_date.in_range(from_date, > >>> >> >>>>>>>>>>>>> to_date) > >>> >> >>>>>>>>>> <Q: (AND: ('user__profile__last_login_date__lte', > >>> >> >>>>>>>>>> from_date), > >>> >> >>>>>>>>>> ('user__profile__last_login_date__gte', to_date))> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> This approach looked the best for me due to several > >>> >> >>>>>>>>>> reasons: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 1. It's explicit - it doesn't do anything but generating > >>> >> >>>>>>>>>> appropriate > >>> >> >>>>>>>>>> Q > >>> >> >>>>>>>>>> object. > >>> >> >>>>>>>>>> The result of comparison can be saved as Q object > variable. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 2. It's short - variants with using model for that will > >>> >> >>>>>>>>>> look > >>> >> >>>>>>>>>> much > >>> >> >>>>>>>>>> longer, > >>> >> >>>>>>>>>> when > >>> >> >>>>>>>>>> joining two or more filters: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> GameSession.objects.user.profile_last_login_date >= > >>> >> >>>>>>>>>>>>> yesterday > >>> >> >>>>>>>>>>>>> # > >>> >> >>>>>>>>>>>>> awkward > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 3. Implementation will not require to change querset > >>> >> >>>>>>>>>> manager or > >>> >> >>>>>>>>>> model > >>> >> >>>>>>>>>> classes > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> 4. Will still allow to use filters and Q class in the > old > >>> >> >>>>>>>>>> way: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> q = Q(user__profile__last_login_date__gte=yesterday) > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> or > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> I'd like to make it as a part of Django ORM syntax and > it > >>> >> >>>>>>>>>> will > >>> >> >>>>>>>>>> not be > >>> >> >>>>>>>>>> hard to > >>> >> >>>>>>>>>> do, especially taking into account the library is > already > >>> >> >>>>>>>>>> done > >>> >> >>>>>>>>>> and > >>> >> >>>>>>>>>> working. > >>> >> >>>>>>>>>> Anyway, I need your thought about the idea in general, > as > >>> >> >>>>>>>>>> well > >>> >> >>>>>>>>>> as > >>> >> >>>>>>>>>> about > >>> >> >>>>>>>>>> particular things like chosen method names - "get_path", > >>> >> >>>>>>>>>> "in_range" > >>> >> >>>>>>>>>> and > >>> >> >>>>>>>>>> etc. > >>> >> >>>>>>>>>> As next step I can create a ticket in the issue tracker, > or > >>> >> >>>>>>>>>> prepare > >>> >> >>>>>>>>>> DEP > >>> >> >>>>>>>>>> first. > >>> >> >>>>>>>>>> In latter case I need to find a shepherd to work with. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Best regards, > >>> >> >>>>>>>>>> Alexey > >>> >> >>>>>>>> > >>> >> >>>>>>>> -- > >>> >> >>>>>>>> You received this message because you are subscribed to > the > >>> >> >>>>>>>> Google > >>> >> >>>>>>>> Groups > >>> >> >>>>>>>> "Django developers (Contributions to Django itself)" > group. > >>> >> >>>>>>>> To unsubscribe from this group and stop receiving emails > from > >>> >> >>>>>>>> it, > >>> >> >>>>>>>> send > >>> >> >>>>>>>> an > >>> >> >>>>>>>> email to [email protected]. > >>> >> >>>>>>>> To post to this group, send email to > >>> >> >>>>>>>> [email protected]. > >>> >> >>>>>>>> Visit this group at > >>> >> >>>>>>>> http://groups.google.com/group/django-developers. > >>> >> >>>>>>>> To view this discussion on the web visit > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > https://groups.google.com/d/msgid/django-developers/3bad8371-f9b4-47ff-a681-0108b320e9b5%40googlegroups.com. > > > >>> >> >>>>>>>> > >>> >> >>>>>>>> For more options, visit https://groups.google.com/d/optout. > > >>> >> >>>>> > >>> >> >>>>> -- > >>> >> >>>>> You received this message because you are subscribed to the > >>> >> >>>>> Google > >>> >> >>>>> Groups > >>> >> >>>>> "Django developers (Contributions to Django itself)" group. > >>> >> >>>>> To unsubscribe from this group and stop receiving emails from > >>> >> >>>>> it, > >>> >> >>>>> send > >>> >> >>>>> an > >>> >> >>>>> email to [email protected]. > >>> >> >>>>> To post to this group, send email to > >>> >> >>>>> [email protected]. > >>> >> >>>>> Visit this group at > >>> >> >>>>> http://groups.google.com/group/django-developers. > >>> >> >>>>> To view this discussion on the web visit > >>> >> >>>>> > >>> >> >>>>> > >>> >> >>>>> > >>> >> >>>>> > https://groups.google.com/d/msgid/django-developers/5e7ee87b-6253-420b-9688-51c39d657cab%40googlegroups.com. > > > >>> >> >>>>> > >>> >> >>>>> For more options, visit https://groups.google.com/d/optout. > >>> >> >> > >>> >> >> -- > >>> >> >> You received this message because you are subscribed to the > Google > >>> >> >> Groups > >>> >> >> "Django developers (Contributions to Django itself)" group. > >>> >> >> To unsubscribe from this group and stop receiving emails from > it, > >>> >> >> send > >>> >> >> an > >>> >> >> email to [email protected]. > >>> >> >> To post to this group, send email to > [email protected]. > >>> >> >> Visit this group at > >>> >> >> http://groups.google.com/group/django-developers. > >>> >> >> To view this discussion on the web visit > >>> >> >> > >>> >> >> > >>> >> >> > https://groups.google.com/d/msgid/django-developers/b97ea866-514c-4d03-bf60-e46a06e5fc5a%40googlegroups.com. > > > >>> >> >> > >>> >> >> For more options, visit https://groups.google.com/d/optout. > >>> >> > > >>> >> > -- > >>> >> > You received this message because you are subscribed to the > Google > >>> >> > Groups "Django developers (Contributions to Django itself)" > group. > >>> >> > To unsubscribe from this group and stop receiving emails from it, > >>> >> > send > >>> >> > an email to [email protected]. > >>> >> > To post to this group, send email to [email protected]. > > >>> >> > Visit this group at > >>> >> > http://groups.google.com/group/django-developers. > >>> >> > To view this discussion on the web visit > >>> >> > > >>> >> > > https://groups.google.com/d/msgid/django-developers/CALMtK1H8OGkXJys4Z-XiZ9kFspZ_naqNS%2BbOigsHCK40Snhj1g%40mail.gmail.com. > > > >>> >> > For more options, visit https://groups.google.com/d/optout. > >>> >> > >>> > -- > >>> > You received this message because you are subscribed to the Google > >>> > Groups > >>> > "Django developers (Contributions to Django itself)" group. > >>> > To unsubscribe from this group and stop receiving emails from it, > send > >>> > an > >>> > email to [email protected]. > >>> > To post to this group, send email to [email protected]. > >>> > Visit this group at http://groups.google.com/group/django-developers. > > >>> > To view this discussion on the web visit > >>> > > >>> > > https://groups.google.com/d/msgid/django-developers/9ea58a11-ce31-4d80-be45-913dff947878%40googlegroups.com. > > > >>> > > >>> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Django developers (Contributions to Django itself)" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > To post to this group, send email to [email protected] > <javascript:>. > > Visit this group at http://groups.google.com/group/django-developers. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/django-developers/17bc7731-8668-4652-a4d3-442fcba48fe1%40googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/92356f24-a388-4735-84ca-dbd9de8c9f4e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
