Re: should manage.py test run system checks?

[David Filipovic]

Sorry for the double post, but I think allowing two levels of control, by 
possibly even adding a setting: TEST_SKIP_CHECKS (in addition to the 
--skip-checks management command flag) which defaults to False would be 
ideal. That way any developer worried about test performance can just add 
this flag to their settings file and still use their old workflows.


On Tuesday, October 20, 2015 at 1:02:12 AM UTC-4, David Filipovic wrote:
>
> How about making them opt-out instead of opt-in (for instance with a 
> --skip-checks flag)? That way anybody who is aware of the fact that checks 
> are being run every time tests are run and is seeking an increase in 
> performance can choose to opt-out, whereas anybody not aware (or at least 
> not fully aware) of it could probably benefit from having them run anyways.
>
>
>
> On Monday, October 19, 2015 at 8:48:29 PM UTC-4, Tim Graham wrote:
>>
>> A ticket [1] and pull request [2] note that `manage.py test` ran the 
>> system checks in Django 1.7 (as a side effect of call_command('migrate')), 
>> but this is no longer the case in Django 1.8 since call_command() doesn't 
>> trigger the system checks anymore.
>>
>> me: I'm of the opinion that running the system checks as part of the 
>> manage.py 
>> test command should be opt-in (for example, by writing a test that 
>> asserts the call_command('check') output is empty. For example, when 
>> debugging a single test, it doesn't seem necessary to have the overhead of 
>> running check. As more options are added to check (e.g. #25500 
>> ), a default implementation 
>> as currently proposed could become increasing inflexible. I'd like to 
>> document the change in the 1.8 release notes and suggest the alternative.
>>
>>
>> Adam: I don't think they should be optional, or if they are, they should 
>> be opt-out. The checks are a brilliant guard against error, but not running 
>> them as part of test invites them not being run at all in a TDD 
>> workflow, as often code can be developed with nothing but running the 
>> tests. It is also surprising that *only* test doesn't run them, since 
>> every other manage command does.
>> At YPlan we couldn't do without them as part of tests. Our aforementioned 
>> 'installed packages' check saves a lot of time that would otherwise be 
>> wasted understanding confusing error messages about imports not working, 
>> and our other custom checks do verification similar to Django's, for issues 
>> that without resolution it does not make sense to even attempt do any 
>> tests. Also we don't notice any real overhead, we can still get a single 
>> test to run in 1 second (with --keepdb :) ) despite all our extra 
>> messing around with pip freeze etc.
>>
>> Other opinions?
>>
>> [1] https://code.djangoproject.com/ticket/25415
>> [2] https://github.com/django/django/pull/5293
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8fa54964-ee76-453e-8f22-a8fa272093c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Contributing to a module that does not seem to be covered by tests extensively (djanog.db.models.expressions)

[David Filipovic]

Hey, Josh,

The 1st Beta was released today. Should we still try to make it into 1.9, 
or is it too late? 

I can get started on the move tomorrow (Tuesday) night. I suggest:

1. moving all date/time related transforms into lookups/datetime.py, and 
leaving all the other ones in lookups/__init__.py.
(this is important because these transforms still have to be registered as 
lhs lookups in __init__.py, in order not to break any existing 
functionality)

2. Renaming DateTransform to DateTimeExtract (to match lower-level API) 
[possibly split into DateExtract and TimeExtract] and make everything work 
through that interface. Regarding this, you mentioned we don't have an 
opportunity to do Extract(lookup='month') type API. Is there a reason for 
this? It could be easily done with the __new__ magic method and if we need 
all the other transform classes available, for some reason, we could just 
initialize them with a DateTimeExtractFactory, for instance. So something 
like: MonthTransform = DateTimeExtractFactory('month').

That way we can just make the DateTimeTransform public, it would be easy to 
document and testing would be a breeze. What do you think?

I'll create a new ticket and get started on this tomorrow after work. I 
expect to have a PR compiled by Wednesday night, but take a little longer.


On Sunday, October 18, 2015 at 9:56:51 PM UTC-4, Josh Smeaton wrote:
>
> > However, the way I see it here, the right hand side lookup interface is 
> all broken up and scattered across multiple modules and classes, many of 
> which are undocumented.
>
> Yes, that's what we should try to fix in 1.9 before it reaches Beta. I'm 
> mostly interested in the DateTransform classes for the moment, since they 
> were only recently added and we have an opportunity to improve them.
>
> Firstly, I think they should be renamed. I'm not sure if we should go for 
> Month() or ExtractMonth() or something else entirely. We don't have the 
> opportunity to do an Extract(lookup='month') type of API just yet which may 
> be a better overall API. Does anyone have thoughts on naming?
>
> Once they've been named, we should consider where the best place for them 
> to reside should be. I'm guessing db/functions.py with the rest of the 
> dbfunctions, but they could easily go into a submodule type arrangement, 
> either lookups/datetime.py or functions/datetime.py.
>
> Then the ones we're making public need to be documented. This should be 
> fairly straightforward once we've achieved the above.
>
> We need to get this done quickly in order to make 1.9. The beta will be 
> released next week, so the timeline is the next few days. Opinions 
> definitely welcome.
>
> > MyModel.objects.filter(date1__month=F('date2__month'))
>
> This is an option in the future. Charettes has linked to a discussion on 
> alternative syntax. So, for the moment, let's just get the object based 
> right hand side documented, and we can work on alternatives later.
>
> > Additionally, is there a reason why Combinable/F couldn't figure out its 
> own output_field type (at least in some cases? e.g (DateField - DateField) 
> --> DurationField)
>
> If you can find an appropriate place for this logic to live then I'm all 
> for it. ExpressionWrapper is cumbersome, but it's also necessary for user 
> based combinations where the base F/Combinable/Expression has no knowledge 
> of output_type combinations.
>
> Cheers
>
> On Saturday, 17 October 2015 14:08:49 UTC+11, David Filipovic wrote:
>>
>> Hi Josh,
>>
>> Thanks so much for your input, both here and on the ticket I referenced.
>>
>> You were correct, MonthTransform did work. Essentially, this piece of 
>> code worked in django 1.9:
>>
>> MyModel.objects.filter(date1__month=MonthTransform('date2'))
>>
>>
>> However, the way I see it here, the right hand side lookup interface is 
>> all broken up and scattered across multiple modules and classes, many of 
>> which are undocumented.
>>
>> I am of the opinion that there should be a unified right hand side lookup 
>> interface, whether it resided in lookups or expressions, but it should be 
>> possible to just use one class for all types of right hand side 
>> expressions/lookups.
>> The left hand side lookups work just fine and are able to figure out 
>> everything about the type of field and the type of lookup and translate 
>> that into sql, so why shouldn't the right hand side ones be able to do the 
>> same? 
>>
>> The documentation does say that the F() object represents the value of a 
>> model field, but is there a reason why it couldn't be the universal right 
>> hand side lookup operator, inclusive of 'subfields' (in a manner of 
>> speaking) such as year, month and day on DateFields (essentially, a subset 
>> of lookups allowable on the left hand side, such that gt would not be 
>> allowed, for instance), so that the following would work:
>>
>> MyModel.objects.filter(date1__month=F('date2__month'))
>>
>>
>> Additionally, is there a reason why 

makemigrations --exit; exit status feels backwards from conventions

[Jon Dufresne]

Before posting this, I've read through the full thread "sys.exit(1)
from makemigrations if no changes found" at
.

I fully agree with the spirit of the change. I already find the
feature useful in CI. However, after using this feature on a CI
server, I find the exit status backwards compared to typical commands.
The makemigrations command returns status 0 to indicate CI failure
(migrations missing) and 1 to indicate CI pass (continue to the next
CI stage).

Typically status 0 indicates pass and non-zero indicates failure. By
following the typical exit status conventions, commands can explicitly
return a non-zero status when detecting a failure or the Python
runtime may return a non-zero status if something goes terribly wrong;
such as an unhandled exception.

Someone that is accustomed to typical exit status conventions might
naively use the makemigrations command:

./manage.py makemigrations --dry-run --exit

The expectation: the next stage should continue if there are no new
migrations (the developer did everything they were supposed to do and
included migrations). However, the above command will return status 1,
not 0, if there are no new migrations.

OK, we can test for that. Maybe change the command to:

! ./manage.py makemigrations --dry-run --exit

That is, interpret the exit status opposite of what one would
typically expect. Immediately, this looks backwards compared to
typical shell scripting. But what happened to the "terribly wrong"
scenario? For example, what if a developer mistakenly added an
incorrect setting that caused an ImproperlyConfigured error? If this
were to happen, I would want the above command to fail and stop the CI
pipeline.

So maybe the next step would be to check explicitly for exit code 1.

./manage.py makemigrations --dry-run --exit; test $? -eq 1

Now it looks like we're hacking around something.

Additionally, Python exits with status 1 for unhandled exceptions. So
the above command would still pass the CI stage for an unhandled
exception. Further Django's BaseCommand.run_from_argv() also exits
with status code 1 on CommandError, so again, it would pass the CI
stage for anything that triggers this sort of error.

It seems exit status 1 is overloaded to mean "all migrations are
accounted for, continue to the next stage of CI", and "something went
terribly wrong".

This is why I feel the exit status is backwards from what is typically expected.

I would like to suggest we find a better way to interface with CI
servers. That is return 0 when there are no migrations (continue to
the next stage) and non-zero for both "migrations missing" and
"something went terribly wrong".

I suggest maybe adding a system check for missing migrations. The
check could report an error, when they are missing.  The check
framework seems like a natural command to be used by CI servers
anyway, so this seems like a good place. The missing migration
detection already exists, so the same code could be leveraged for this
check.

I'm also open to other suggestions on creating a more convention exit status.

If there is agreement and the proposal sounds good, I can follow
through with a ticket and code changes.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5YDr-HB5sUdwKK-K2awQZk7qUhJJdaU%2B4SH_6nx9x%3D5w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fellow Report - October 17, 2015

[Michael Manfre]

Tim gave a good overview of the week. I'll focus my response on the MSSQL
roadmap.

On Sat, Oct 17, 2015 at 8:36 PM, Tim Graham  wrote:

> * Discussed the long-term roadmap for MSSQL support in Django. I'll let
> Michael speak to the details of this.
>

I had the opportunity to speak with several engineers that had intimate
knowledge of Microsoft's various SQL Server drivers; past, current, and
future. The ADO based drivers used by django-mssql will not receive any
future love from Microsoft. They don't have plans on removing them from
Windows, but the advice was to use either ODBC or FreeTDS. The fastest and
most feature rich drivers for connecting to SQL Server from Windows are the
ODBC drivers. Microsoft is actively working on updating the ODBC drivers
for Linux and expect them to be released this year. ODBC drivers for Mac
are planned for sometime next year. The next best way of connecting to SQL
Server is to use FreeTDS (without the ODBC interface), which has the
benefit of having drivers available for all OSes right now.

The short-term plan for django-mssql is to get it to pass the Django 1.8
test suite, without making any substantial changes. This will allow any
existing django-mssql users to upgrade to the Django 1.8 LTS. My long term
plan is to switch out ADO for replaceable ODBC and FreeTDS. This may follow
the current pattern Aymeric started when he created django-pymssql, or have
them both in django-mssql. After 1.8 is fully supported, I need to think,
discuss, and investigate more before making a decision.

ODBC seems to be the most performant and best option, but I have concerns
about the current lack of good drivers for Linux and Mac. If Microsoft
follows through with their commitment, then it should be a non-issue, but
I'd rather hedge my bets and ensure an easier time with drivers in the long
run. Supporting both also provides for a good way of comparing the
performance of the underlying drivers.

I plan on evaluating django-mssql, django-pyodbc-azure, and starting from
scratch to see which would be the best route for this new mssql database
backend. There are pros and cons to each. Django-mssql has a lot of legacy
cruft. Django-pyodbc-azure targets Azure SQL Server, which has some subtle
differences with Microsoft's non-cloud version of the database server.
Starting from scratch is potentially more work. Regardless of the decision,
the repo will be hosted on github.

Regards,
Michael Manfre

-- 
GPG Fingerprint: 74DE D158 BAD0 EDF8
keybase.io/manfre

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAGdCwBvWHFj3qoyUWOwU-r8Ng_ME4MhF%2Bw%2BOqa1J0PkxBkxWxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[ANNOUNCE] Django 1.9 beta 1 released

[Tim Graham]

We've made the second release on the way to Django's next major release, 
Django 1.9! With a month and a half until the scheduled final release, 
we'll need timely testing from the community to ensure an on-time and 
stable release. Check out the blog post:

https://www.djangoproject.com/weblog/2015/oct/19/django-19-beta-1-released/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/68e360e2-917b-4f36-953a-64fce919457b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

should manage.py test run system checks?

[Tim Graham]

A ticket [1] and pull request [2] note that `manage.py test` ran the system 
checks in Django 1.7 (as a side effect of call_command('migrate')), but 
this is no longer the case in Django 1.8 since call_command() doesn't 
trigger the system checks anymore.

me: I'm of the opinion that running the system checks as part of the manage.py 
test command should be opt-in (for example, by writing a test that asserts 
the call_command('check') output is empty. For example, when debugging a 
single test, it doesn't seem necessary to have the overhead of running check. 
As more options are added to check (e.g. #25500 
), a default implementation as 
currently proposed could become increasing inflexible. I'd like to document 
the change in the 1.8 release notes and suggest the alternative.


Adam: I don't think they should be optional, or if they are, they should be 
opt-out. The checks are a brilliant guard against error, but not running 
them as part of test invites them not being run at all in a TDD workflow, 
as often code can be developed with nothing but running the tests. It is 
also surprising that *only* test doesn't run them, since every other manage 
command does.
At YPlan we couldn't do without them as part of tests. Our aforementioned 
'installed packages' check saves a lot of time that would otherwise be 
wasted understanding confusing error messages about imports not working, 
and our other custom checks do verification similar to Django's, for issues 
that without resolution it does not make sense to even attempt do any 
tests. Also we don't notice any real overhead, we can still get a single 
test to run in 1 second (with --keepdb :) ) despite all our extra messing 
around with pip freeze etc.

Other opinions?

[1] https://code.djangoproject.com/ticket/25415
[2] https://github.com/django/django/pull/5293

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4a18c3ad-0194-40fe-b872-180db835b2d7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django ORM query syntax enhancement

[Alexey Zankevich]

Here is a link https://github.com/django/django/pull/5443, sorry :)

Making F object as expression wasn't required, but I still had to create 
FieldExpression wrapper, which wrapped F object. Eventually, I thought it 
would confuse people, especially taking in account F objects referenced in 
doc as "F Expression". So, I removed FieldExpression wrapper and made F 
object as real expression.


On Monday, October 19, 2015 at 2:09:14 PM UTC+3, Josh Smeaton wrote:
>
> I think you forgot the PR link Alexey. I'd like to have a look at the 
> changes you've made. I'm not sure about the extensions to F() just yet, I'm 
> hoping they aren't actually needed for your patch to work, but will need to 
> review to be sure.
>
> Cheers
>
> On Monday, 19 October 2015 21:49:37 UTC+11, Alexey Zankevich wrote:
>
> Hi all,
>
> Here is a raw pull request, allowing lookup instances passing to filter 
> method 
> There are two new test cases working:
>
> 1. Passing lookup with F object: 
> Article.objects.filter(GreaterThan(F('id'), Value(5)))
> 2. And passing lookup with transforms: 
> Article.objects.filter(Contains(Upper(Lower(Upper(F('headline', 
> 'ARTICLE 5'))
>
> All the existing tests are passing successfully, by the way (at least for 
> SQLite backend).
>
> Several comments:
>
>- F expressions became real expressions (Expression subclass).
>- Expression got a new method - get_source_f_object, which walks 
>through children classes and find if it has F object passed. In that case 
>output_field-related code should be postponed, until compiler resolves it 
>(which cannot be done during lookup init time).
>- some code related to query.build_filter method (responsible to 
>lookups/transform parsing, and calculating joins), extracted into separate 
>helper classes - QueryKeywordLookupHelper and QueryObjectLookupHelper. 
> They 
>have common code base, but some of methods work in a different way.
>
> Except review, it would be really great if someone help with writing test 
> cases (and I'll get them working in case of any issues). Also, there is a 
> workaround field=None -> isnull=True for oracle database, can anybody check 
> if it is working correctly?
>
> Regards,
> Alexey
>
>
> On Thursday, October 1, 2015 at 11:38:39 AM UTC+3, Anssi Kääriäinen wrote:
>
> Lets concentrate on making expressions in filter calls and lookups as 
> expressions reality. When we have done that we can try out different 
> syntax approaches in 3rd party apps. Finally, after we have field 
> tested the approaches, we can merge something in to Django. 
>
>  - Anssi 
>
> On Thu, Oct 1, 2015 at 10:49 AM, Alexey Zankevich 
>  wrote: 
> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for 
> >> core. 
> > 
> > When Django supports expression lookups, it will be possible to build 
> syntax 
> > sugar third-party library on top of it. 
> > For example, it will be possible to implement dotted syntax for F 
> objects: 
> > 
> > UserModel.objects.filter(Decode(F.user.name, 'utf-8') == u'Бармаглот') 
> > 
> > alternatively it can be even combined with your syntax 
> > 
> > UserModel.objects.filter(F.user.name.decode('utf-8') == u'Бармаглот') 
> > 
> > However, the second solution will be more complicated from technical 
> point 
> > of view (as will require lazy object, collecting all the func or 
> transform 
> > calls), also it will require to be really careful with transform names 
> not 
> > to conflict with possible field names. 
> > For example, it's hardly possible to have "decode" field in the model, 
> but 
> > easily - "date". 
> > 
> > 
> > On Thursday, October 1, 2015 at 10:18:40 AM UTC+3, Loïc Bistuer wrote: 
> >> 
> >> > On Oct 1, 2015, at 13:38, Anssi Kääriäinen  
> wrote: 
> >> > 
> >> > +1 to not requiring all transforms to handle __underscore__ syntax. 
> >> 
> >> +1 
> >> 
> >> > I think what we want to do is allow users choose which syntax they 
> >> > prefer. The idea is that Django will support both JSONExtract('data', 
> >> > path=['owner', 'other_pets', 0, 'name']) and 
> >> > data__owner__other_pets__0__name out of the box. 
> >> > 
> >> > We will also make it possible to support callable transforms. For 
> >> > example: 
> >> >filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот'))) 
> >> > is equivalent to 
> >> >filter(F('name').decode('utf-8') == Value(u'Бармаглот')) 
> >> > 
> >> > The callable transforms syntax will not be a part of Django, but it 
> >> > will be possible to create an extension for this (it is actually 
> >> > surprisingly easy to do once we have support for expressions in 
> >> > filter). 
> >> 
> >> I'm pretty convinced we need a better API / sugar as part of core in 
> the 
> >> long run. 
> >> 
> >> The `filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))` 
> >> syntax is not pythonic and very hard to decipher, and we've reached the 
> >> limits of our historical double underscore 

Re: Django ORM query syntax enhancement

[Asif Saifuddin]

this is the PR josh

https://github.com/django/django/pull/5443

On Monday, October 19, 2015 at 5:09:14 PM UTC+6, Josh Smeaton wrote:
>
> I think you forgot the PR link Alexey. I'd like to have a look at the 
> changes you've made. I'm not sure about the extensions to F() just yet, I'm 
> hoping they aren't actually needed for your patch to work, but will need to 
> review to be sure.
>
> Cheers
>
> On Monday, 19 October 2015 21:49:37 UTC+11, Alexey Zankevich wrote:
>
> Hi all,
>
> Here is a raw pull request, allowing lookup instances passing to filter 
> method 
> There are two new test cases working:
>
> 1. Passing lookup with F object: 
> Article.objects.filter(GreaterThan(F('id'), Value(5)))
> 2. And passing lookup with transforms: 
> Article.objects.filter(Contains(Upper(Lower(Upper(F('headline', 
> 'ARTICLE 5'))
>
> All the existing tests are passing successfully, by the way (at least for 
> SQLite backend).
>
> Several comments:
>
>- F expressions became real expressions (Expression subclass).
>- Expression got a new method - get_source_f_object, which walks 
>through children classes and find if it has F object passed. In that case 
>output_field-related code should be postponed, until compiler resolves it 
>(which cannot be done during lookup init time).
>- some code related to query.build_filter method (responsible to 
>lookups/transform parsing, and calculating joins), extracted into separate 
>helper classes - QueryKeywordLookupHelper and QueryObjectLookupHelper. 
> They 
>have common code base, but some of methods work in a different way.
>
> Except review, it would be really great if someone help with writing test 
> cases (and I'll get them working in case of any issues). Also, there is a 
> workaround field=None -> isnull=True for oracle database, can anybody check 
> if it is working correctly?
>
> Regards,
> Alexey
>
>
> On Thursday, October 1, 2015 at 11:38:39 AM UTC+3, Anssi Kääriäinen wrote:
>
> Lets concentrate on making expressions in filter calls and lookups as 
> expressions reality. When we have done that we can try out different 
> syntax approaches in 3rd party apps. Finally, after we have field 
> tested the approaches, we can merge something in to Django. 
>
>  - Anssi 
>
> On Thu, Oct 1, 2015 at 10:49 AM, Alexey Zankevich 
>  wrote: 
> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for 
> >> core. 
> > 
> > When Django supports expression lookups, it will be possible to build 
> syntax 
> > sugar third-party library on top of it. 
> > For example, it will be possible to implement dotted syntax for F 
> objects: 
> > 
> > UserModel.objects.filter(Decode(F.user.name, 'utf-8') == u'Бармаглот') 
> > 
> > alternatively it can be even combined with your syntax 
> > 
> > UserModel.objects.filter(F.user.name.decode('utf-8') == u'Бармаглот') 
> > 
> > However, the second solution will be more complicated from technical 
> point 
> > of view (as will require lazy object, collecting all the func or 
> transform 
> > calls), also it will require to be really careful with transform names 
> not 
> > to conflict with possible field names. 
> > For example, it's hardly possible to have "decode" field in the model, 
> but 
> > easily - "date". 
> > 
> > 
> > On Thursday, October 1, 2015 at 10:18:40 AM UTC+3, Loïc Bistuer wrote: 
> >> 
> >> > On Oct 1, 2015, at 13:38, Anssi Kääriäinen  
> wrote: 
> >> > 
> >> > +1 to not requiring all transforms to handle __underscore__ syntax. 
> >> 
> >> +1 
> >> 
> >> > I think what we want to do is allow users choose which syntax they 
> >> > prefer. The idea is that Django will support both JSONExtract('data', 
> >> > path=['owner', 'other_pets', 0, 'name']) and 
> >> > data__owner__other_pets__0__name out of the box. 
> >> > 
> >> > We will also make it possible to support callable transforms. For 
> >> > example: 
> >> >filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот'))) 
> >> > is equivalent to 
> >> >filter(F('name').decode('utf-8') == Value(u'Бармаглот')) 
> >> > 
> >> > The callable transforms syntax will not be a part of Django, but it 
> >> > will be possible to create an extension for this (it is actually 
> >> > surprisingly easy to do once we have support for expressions in 
> >> > filter). 
> >> 
> >> I'm pretty convinced we need a better API / sugar as part of core in 
> the 
> >> long run. 
> >> 
> >> The `filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))` 
> >> syntax is not pythonic and very hard to decipher, and we've reached the 
> >> limits of our historical double underscore syntax (doesn't support 
> multiple 
> >> arguments, limited to ascii, etc.). 
> >> 
> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for 
> >> core: 
> >> 
> >> - It's a small extension to the existing F objects, so it's easy to 
> grasp 
> >> for existing Django users. 
> >> - It's just a thin sugar on 

Re: Django ORM query syntax enhancement

[Josh Smeaton]

I think you forgot the PR link Alexey. I'd like to have a look at the 
changes you've made. I'm not sure about the extensions to F() just yet, I'm 
hoping they aren't actually needed for your patch to work, but will need to 
review to be sure.

Cheers

On Monday, 19 October 2015 21:49:37 UTC+11, Alexey Zankevich wrote:
>
> Hi all,
>
> Here is a raw pull request, allowing lookup instances passing to filter 
> method 
> There are two new test cases working:
>
> 1. Passing lookup with F object: 
> Article.objects.filter(GreaterThan(F('id'), Value(5)))
> 2. And passing lookup with transforms: 
> Article.objects.filter(Contains(Upper(Lower(Upper(F('headline', 
> 'ARTICLE 5'))
>
> All the existing tests are passing successfully, by the way (at least for 
> SQLite backend).
>
> Several comments:
>
>- F expressions became real expressions (Expression subclass).
>- Expression got a new method - get_source_f_object, which walks 
>through children classes and find if it has F object passed. In that case 
>output_field-related code should be postponed, until compiler resolves it 
>(which cannot be done during lookup init time).
>- some code related to query.build_filter method (responsible to 
>lookups/transform parsing, and calculating joins), extracted into separate 
>helper classes - QueryKeywordLookupHelper and QueryObjectLookupHelper. 
> They 
>have common code base, but some of methods work in a different way.
>
> Except review, it would be really great if someone help with writing test 
> cases (and I'll get them working in case of any issues). Also, there is a 
> workaround field=None -> isnull=True for oracle database, can anybody check 
> if it is working correctly?
>
> Regards,
> Alexey
>
>
> On Thursday, October 1, 2015 at 11:38:39 AM UTC+3, Anssi Kääriäinen wrote:
>>
>> Lets concentrate on making expressions in filter calls and lookups as 
>> expressions reality. When we have done that we can try out different 
>> syntax approaches in 3rd party apps. Finally, after we have field 
>> tested the approaches, we can merge something in to Django. 
>>
>>  - Anssi 
>>
>> On Thu, Oct 1, 2015 at 10:49 AM, Alexey Zankevich 
>>  wrote: 
>> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for 
>> >> core. 
>> > 
>> > When Django supports expression lookups, it will be possible to build 
>> syntax 
>> > sugar third-party library on top of it. 
>> > For example, it will be possible to implement dotted syntax for F 
>> objects: 
>> > 
>> > UserModel.objects.filter(Decode(F.user.name, 'utf-8') == u'Бармаглот') 
>> > 
>> > alternatively it can be even combined with your syntax 
>> > 
>> > UserModel.objects.filter(F.user.name.decode('utf-8') == u'Бармаглот') 
>> > 
>> > However, the second solution will be more complicated from technical 
>> point 
>> > of view (as will require lazy object, collecting all the func or 
>> transform 
>> > calls), also it will require to be really careful with transform names 
>> not 
>> > to conflict with possible field names. 
>> > For example, it's hardly possible to have "decode" field in the model, 
>> but 
>> > easily - "date". 
>> > 
>> > 
>> > On Thursday, October 1, 2015 at 10:18:40 AM UTC+3, Loïc Bistuer wrote: 
>> >> 
>> >> > On Oct 1, 2015, at 13:38, Anssi Kääriäinen  
>> wrote: 
>> >> > 
>> >> > +1 to not requiring all transforms to handle __underscore__ syntax. 
>> >> 
>> >> +1 
>> >> 
>> >> > I think what we want to do is allow users choose which syntax they 
>> >> > prefer. The idea is that Django will support both 
>> JSONExtract('data', 
>> >> > path=['owner', 'other_pets', 0, 'name']) and 
>> >> > data__owner__other_pets__0__name out of the box. 
>> >> > 
>> >> > We will also make it possible to support callable transforms. For 
>> >> > example: 
>> >> >filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот'))) 
>> >> > is equivalent to 
>> >> >filter(F('name').decode('utf-8') == Value(u'Бармаглот')) 
>> >> > 
>> >> > The callable transforms syntax will not be a part of Django, but it 
>> >> > will be possible to create an extension for this (it is actually 
>> >> > surprisingly easy to do once we have support for expressions in 
>> >> > filter). 
>> >> 
>> >> I'm pretty convinced we need a better API / sugar as part of core in 
>> the 
>> >> long run. 
>> >> 
>> >> The `filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))` 
>> >> syntax is not pythonic and very hard to decipher, and we've reached 
>> the 
>> >> limits of our historical double underscore syntax (doesn't support 
>> multiple 
>> >> arguments, limited to ascii, etc.). 
>> >> 
>> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for 
>> >> core: 
>> >> 
>> >> - It's a small extension to the existing F objects, so it's easy to 
>> grasp 
>> >> for existing Django users. 
>> >> - It's just a thin sugar on top of the canonical API. 
>> >> - All the required machinery 

Re: Django ORM query syntax enhancement

[Alexey Zankevich]

Hi all,

Here is a raw pull request, allowing lookup instances passing to filter 
method 
There are two new test cases working:

1. Passing lookup with F object: 
Article.objects.filter(GreaterThan(F('id'), Value(5)))
2. And passing lookup with transforms: 
Article.objects.filter(Contains(Upper(Lower(Upper(F('headline', 
'ARTICLE 5'))

All the existing tests are passing successfully, by the way (at least for 
SQLite backend).

Several comments:

   - F expressions became real expressions (Expression subclass).
   - Expression got a new method - get_source_f_object, which walks through 
   children classes and find if it has F object passed. In that case 
   output_field-related code should be postponed, until compiler resolves it 
   (which cannot be done during lookup init time).
   - some code related to query.build_filter method (responsible to 
   lookups/transform parsing, and calculating joins), extracted into separate 
   helper classes - QueryKeywordLookupHelper and QueryObjectLookupHelper. They 
   have common code base, but some of methods work in a different way.

Except review, it would be really great if someone help with writing test 
cases (and I'll get them working in case of any issues). Also, there is a 
workaround field=None -> isnull=True for oracle database, can anybody check 
if it is working correctly?

Regards,
Alexey


On Thursday, October 1, 2015 at 11:38:39 AM UTC+3, Anssi Kääriäinen wrote:
>
> Lets concentrate on making expressions in filter calls and lookups as 
> expressions reality. When we have done that we can try out different 
> syntax approaches in 3rd party apps. Finally, after we have field 
> tested the approaches, we can merge something in to Django. 
>
>  - Anssi 
>
> On Thu, Oct 1, 2015 at 10:49 AM, Alexey Zankevich 
>  wrote: 
> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for 
> >> core. 
> > 
> > When Django supports expression lookups, it will be possible to build 
> syntax 
> > sugar third-party library on top of it. 
> > For example, it will be possible to implement dotted syntax for F 
> objects: 
> > 
> > UserModel.objects.filter(Decode(F.user.name, 'utf-8') == u'Бармаглот') 
> > 
> > alternatively it can be even combined with your syntax 
> > 
> > UserModel.objects.filter(F.user.name.decode('utf-8') == u'Бармаглот') 
> > 
> > However, the second solution will be more complicated from technical 
> point 
> > of view (as will require lazy object, collecting all the func or 
> transform 
> > calls), also it will require to be really careful with transform names 
> not 
> > to conflict with possible field names. 
> > For example, it's hardly possible to have "decode" field in the model, 
> but 
> > easily - "date". 
> > 
> > 
> > On Thursday, October 1, 2015 at 10:18:40 AM UTC+3, Loïc Bistuer wrote: 
> >> 
> >> > On Oct 1, 2015, at 13:38, Anssi Kääriäinen  
> wrote: 
> >> > 
> >> > +1 to not requiring all transforms to handle __underscore__ syntax. 
> >> 
> >> +1 
> >> 
> >> > I think what we want to do is allow users choose which syntax they 
> >> > prefer. The idea is that Django will support both JSONExtract('data', 
> >> > path=['owner', 'other_pets', 0, 'name']) and 
> >> > data__owner__other_pets__0__name out of the box. 
> >> > 
> >> > We will also make it possible to support callable transforms. For 
> >> > example: 
> >> >filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот'))) 
> >> > is equivalent to 
> >> >filter(F('name').decode('utf-8') == Value(u'Бармаглот')) 
> >> > 
> >> > The callable transforms syntax will not be a part of Django, but it 
> >> > will be possible to create an extension for this (it is actually 
> >> > surprisingly easy to do once we have support for expressions in 
> >> > filter). 
> >> 
> >> I'm pretty convinced we need a better API / sugar as part of core in 
> the 
> >> long run. 
> >> 
> >> The `filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))` 
> >> syntax is not pythonic and very hard to decipher, and we've reached the 
> >> limits of our historical double underscore syntax (doesn't support 
> multiple 
> >> arguments, limited to ascii, etc.). 
> >> 
> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for 
> >> core: 
> >> 
> >> - It's a small extension to the existing F objects, so it's easy to 
> grasp 
> >> for existing Django users. 
> >> - It's just a thin sugar on top of the canonical API. 
> >> - All the required machinery being already in place (lookup & transform 
> >> registration) the change to Django is minimal. 
> >> 
> >> > - Anssi 
> >> > 
> >> > On Thu, Oct 1, 2015 at 4:00 AM, Josh Smeaton  
> >> > wrote: 
> >> >> No, not all Lookups or Transforms are required to handle 
> __underscore__ 
> >> >> syntax. The entire point of supporting object based lookups is to 
> >> >> handle 
> >> >> cases that get more complex than a single argument transform or a 
> left 
> >>