Re: DJANGO_SETTINGS_FILE

[James Pic]

There are indeed a nice handful of apps out there that help doing this
in different manners. While we're at it, my own way is to make every
project a python package with a setup.py. So the settings can always
be imported. Then I let docker swarm or kubernetes handle environment
variables which are (surprise) environment-specific (the cherry on the
top being the entry point that points to manage.py).

I'm sure everybody here has a favorite way, which works, even if it's
not as pretty as my favourite way which is the prettiest in my opinion
of course ;) Let alone aesthetics,
DJANGO_SETTINGS_MODULE=production_settings
PYTHONPATH=/etc/yourapp:$PYTHONPATH is basically a lot more
error-prone than DJANGO_SETTINGS_FILE=/etc/yourapp/production.py, is
it even necessary to demonstrate this ? This somewhat reminds my
favorite talk this year, "Don't fight the controls", by Daniele
Procida.

Is there a problem to fix for any of us ? Probably not, we've been
deploying django for more than a decade with DJANGO_SETTINGS_MODULE.
We know are way in there, so nothing is going to change if our needs
don't change or if we don't take a step back to look at the bigger
picture.

Do I need this to deploy my projects ? No of course, because I use the
prettiest way hhihihi ;) I'm more than happy to win a debate
aesthetics of automated deployment any time with gitlab ci and docker
swarm or kubernetes of course, but actually that's not why that thread
was even about, sorry if I misguided you, this thread is neither about
you neither about me actually.

Do I need this in projects which I'm not the only one to deploy ? Well
now we're getting somewhere. You see, when someone makes settings.py
parse a json file, or read an environment file (yes, django-environ
actually has the feature to parse an environment file), it reminds of
E. W. Dijkstra's quote "The computing scientist’s main challenge is
not to get confused by the complexities of his own making".

Do you think this kind of situation would still have happened if
Django supported DJANGO_SETTINGS_FILE ? If so, would it at least be
less likely ? They wouldn't need to mess with the settings file that
we all share and that should not do anything more complicated than
reading the existing environment according to 12factor (and my taste
:)), they could just extend it and pass the path.

Do I ear about projects where they invent a new per-environment
configuration files ? Yeah, a lot, and you Adam, you actually have
never heard about it to the point where you call it "esoteric" ? How
is defining a configuration file esoteric ? What's esotheric here, is
PYTHONPATH=/etc/yourapp DJANGO_SETTINGS_MODULE=production (or should
it be production_settings ? doesn't matter, it'll smell a mile away
anyway). DJANGO_SETTINGS_FILE=/etc/yourapp/production.py is logical
and explicit. How me, kubernetes or docker swarm do
env-specific-file-less configuration, can be called esoteric, but it's
certainly not the other way around.

I'm sure many of you maintain open source django projects which are to
be installed by non technical people. I mean, I assume most of us are
either making big money in big corporations either dreaming of it, but
I also know most of us also dedicate a huge amount of time to
contribute to NGOs with Django, or just teach django for the pleasure
to share a passion. In any of this case, the project developer (us)
faces a bunch of people who are really willing to learn, python,
django, linux, bash, and so on, but have no experience. And this is
how the drama begins, I can imagine them in front of Tobias.

"How do I configure this project ?", they ask.

"Just **prepend** the configuration **directory** to PYTHONPATH, and
use the file name without the .py extension as
DJANGO_SETTINGS_MODULE", Adam says.

The user adds a file in /etc, called "memopol.py", to store their
settings. Then after they do what Adam told them, itsnotworking again.
"ModuleNotFoundError: No module named 'memopol.wsgi' 'memopol' is not
a package". I'm sure it won't take long before Tobias begins a
technical explanation (instead of working on the project itself):

"Well django couldn't import WSGI anymore because it was in
memopol/wsgi.py in the package, and python imported memopol from /etc
now that it's first in the PYTHONPATH".

"", (it doesn't matter, the user is lost, let's talk him into a
solution to cheer them up).

"So what you can do for example, is to prepend something unique to
your configuration file, such as _settings"

"Same error"

"What ??"

It won't take long before Tobias figures that this project also
depends on a module named memopol_settings.

And that's not even the worst case scenario.

Once you've been through this for a while, sharing your django
projects with non technical people, you start thinking "I won't forget
to ask django-dev if my friends can haz DJANGO_SETTINGS_FILE next
time". And here I am today, at their service, at your service, thanks
for reading.

-- 
You received this message 

Paginator Class - Refactor page function for better abstraction

[Robert Roskam]

So the existing Paginator class is great for the standard use case. 

But recently I wanted to be have a Paginator that would return *at least 
per_page *x items. I called it FlexPaginator. My use case was that I had a 
group_by value that I wanted to keep together on the same page, and so the 
rows per page needed to grow. When it was all said an done, I had a 
completely different looking page function on my FlexPaginator.

Here's the default one for reference:

def page(self, number):
"""
Returns a Page object for the given 1-based page number.
"""
number = self.validate_number(number)
bottom = (number - 1) * self.per_page
top = bottom + self.per_page
if top + self.orphans >= self.count:
top = self.count
return self._get_page(self.object_list[bottom:top], number, self)



I'd like to refactor several things about it that would have made it more 
abstract and easier for me, but still keep it backwards compatible:

   - rename bottom and top variables, instead self.first and self.last 
   respectively
   - turn their inline calculations into function calls to  a new, 
   self.first() and self.last()
   - move the entire self.object_list[bottom:top] into its own line
   - move that specific logic into a function called self.get_page_objects
   - store that function's value into  self.page_objects

Any comments?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b11166ec-0800-48f6-a647-62f41b67631c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [feature request] including HttpResponse(status=204) as an HttpResponse subclasses

[Philip Lee]

I think you'd better write the decision in the document to stop Django 
users from making the same feature request in future !

On Thursday, April 6, 2017 at 12:35:41 AM UTC+8, Tim Graham wrote:
>
> Hi, this was already wontfixed here:
> https://code.djangoproject.com/ticket/3362
>
> with the rationale, "We've decided in the past not to add a new class for 
> every single response code. You can already pass the status code in when 
> creating the HttpResponse class, so that can be used in this case."
>
> (found with this google search: httpresponse 204 site:
> code.djangoproject.com)
>
> On Wednesday, April 5, 2017 at 11:25:44 AM UTC-4, Philip Lee wrote:
>>
>> Every Django view function MUST return an HttpResponse object, sometimes we 
>> just want to send data to the server and  don't want to send data back to 
>> the client , this probably lead Python users want to return None instead, 
>> however , this is not allowed in Django view function, I found  
>> HttpResponse(status=204) 
>>  may come for 
>> rescue in this case,  thus it would be better to include 
>> HttpResponse(status=204) as an HttpResponse subclasses for convenience so 
>> that could save Django users from asking those returning-null-response 
>> questions:http://stackoverflow.com/questions/17557618/post-without-response-in-django-javascript-interactionhttp://stackoverflow.com/questions/2131203/django-no-redirectionshttp://stackoverflow.com/questions/4123155/how-do-i-send-empty-response-in-django-without-templates
>>
>> BTW, someone already implemented this feature  here, better to adopt it in 
>> Django http://django-extras.readthedocs.io/en/latest/ref/http-response.html
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2f619780-d6f1-4bce-8d34-40a18b87d798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: DJANGO_SETTINGS_FILE

[Chris Church]

Look through https://code.djangoproject.com/wiki/SplitSettings for many
different approaches to loading settings from files that aren't importable
as Python modules.

My personal favorite is django-split-settings, which would let you
accomplish exactly what you're trying to do.


On Thu, Apr 6, 2017 at 9:50 AM, Tobias McNulty 
wrote:

> I don't see Django making a change like this. If it's really impossible to
> get your production settings file on an already-importable Python path, you
> *could* modify your PYTHONPATH
> 
> environment variable to add the directory containing your Django settings
> file. That being said, I think you're better off keeping all your settings
> files in version control where they belong, and changing the things you
> need to change via environment variables.
>
> On Thu, Apr 6, 2017 at 9:20 AM, James Pic  wrote:
>
>> Hi all!
>>
>> Life with DJANGO_SETTINGS_MODULE has been great. However, it requires the
>> settings to be an importable module. In most cases, this is a straight
>> forward "simple" python script.
>>
>> Sometimes it looks like it could make sense to add an environment
>> variable that could use a file that is not importable as a python module,
>> ie.:
>>
>> DJANGO_SETTINGS_FILE=/etc/yourapp/production.py
>> DJANGO_SETTINGS_FILE=~/yourapp_production.py
>> DJANGO_SETTINGS_FILE=settings.py  # relative to pwd
>>
>> Then, in this file, we could override the default project settings there
>> without having to put the new file in an importable module, ie. in
>> /etc/yourapp/production.py:
>>
>> from yourapp.settings import *
>> DATABASES=...
>> CACHES=...
>> SECRET_KEY=...
>>
>> I'm sure this has been discussed before but didn't find anything by
>> searching DJANGO_SETTINGS_FILE in the ML.
>>
>> What's your opinion on this ? Is it time to upgrade this ?
>>
>> Best,
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/django-developers/CAC6Op19-0JeL8Jdcq70-pvwz%2Bi2hfKtopEv
>> ihca_rGfH1DrPww%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
>
>
> *Tobias McNulty*Chief Executive Officer
>
> tob...@caktusgroup.com
> www.caktusgroup.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CAMGFDKThapJkEykuhxFEuTF%2BZNjHNr7t6p%
> 3Dssenb0qM85DVXyQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAH%2BKTJ6nisgP7AC99V0S0zkYGPVu_5oKQCF%3DQ39XEz_0%3Dn9B_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: DJANGO_SETTINGS_FILE

[Tobias McNulty]

I don't see Django making a change like this. If it's really impossible to
get your production settings file on an already-importable Python path, you
*could* modify your PYTHONPATH

environment variable to add the directory containing your Django settings
file. That being said, I think you're better off keeping all your settings
files in version control where they belong, and changing the things you
need to change via environment variables.

On Thu, Apr 6, 2017 at 9:20 AM, James Pic  wrote:

> Hi all!
>
> Life with DJANGO_SETTINGS_MODULE has been great. However, it requires the
> settings to be an importable module. In most cases, this is a straight
> forward "simple" python script.
>
> Sometimes it looks like it could make sense to add an environment variable
> that could use a file that is not importable as a python module, ie.:
>
> DJANGO_SETTINGS_FILE=/etc/yourapp/production.py
> DJANGO_SETTINGS_FILE=~/yourapp_production.py
> DJANGO_SETTINGS_FILE=settings.py  # relative to pwd
>
> Then, in this file, we could override the default project settings there
> without having to put the new file in an importable module, ie. in
> /etc/yourapp/production.py:
>
> from yourapp.settings import *
> DATABASES=...
> CACHES=...
> SECRET_KEY=...
>
> I'm sure this has been discussed before but didn't find anything by
> searching DJANGO_SETTINGS_FILE in the ML.
>
> What's your opinion on this ? Is it time to upgrade this ?
>
> Best,
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CAC6Op19-0JeL8Jdcq70-pvwz%
> 2Bi2hfKtopEvihca_rGfH1DrPww%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 


*Tobias McNulty*Chief Executive Officer

tob...@caktusgroup.com
www.caktusgroup.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMGFDKThapJkEykuhxFEuTF%2BZNjHNr7t6p%3Dssenb0qM85DVXyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: DJANGO_SETTINGS_FILE

[Adam Johnson]

You could do this already by putting a file on PYTHONPATH that just reads
the contents of the file specified by DJANGO_SETTINGS_FILE and puts it in
its own module dict, e.g.

$ cat ~/tmp/test.py
foobar = 12
$ TEST_SETTINGS_FILE=~/tmp/test.py ipython
...

In [1]: import imp, os

In [2]: test = imp.load_source('test', os.environ['TEST_SETTINGS_FILE'])

In [3]: globals().update(test.__dict__)

In [4]: foobar
Out[4]: 12

You could make this a third party package too, and just point django to it.

I don't see why Django needs to change to make this somewhat esoteric
usecase possible.

On 6 April 2017 at 14:20, James Pic  wrote:

> Hi all!
>
> Life with DJANGO_SETTINGS_MODULE has been great. However, it requires the
> settings to be an importable module. In most cases, this is a straight
> forward "simple" python script.
>
> Sometimes it looks like it could make sense to add an environment variable
> that could use a file that is not importable as a python module, ie.:
>
> DJANGO_SETTINGS_FILE=/etc/yourapp/production.py
> DJANGO_SETTINGS_FILE=~/yourapp_production.py
> DJANGO_SETTINGS_FILE=settings.py  # relative to pwd
>
> Then, in this file, we could override the default project settings there
> without having to put the new file in an importable module, ie. in
> /etc/yourapp/production.py:
>
> from yourapp.settings import *
> DATABASES=...
> CACHES=...
> SECRET_KEY=...
>
> I'm sure this has been discussed before but didn't find anything by
> searching DJANGO_SETTINGS_FILE in the ML.
>
> What's your opinion on this ? Is it time to upgrade this ?
>
> Best,
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/django-developers/CAC6Op19-0JeL8Jdcq70-pvwz%2Bi2hfKtopEv
> ihca_rGfH1DrPww%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0PsDncjCq745B-GuBsc7hWp1qemineA9BUx%2BEn1VJvFQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django 1.8 - django 1.7 differences to read DecimalField

[Maximiliano Robaina]

Hi Tim,

Thanks for the tip.

I finally found the problem [1], which was obviously between the chair and 
the keyboard.

Regards.
Maxi

[1] https://github.com/maxirobaina/django-firebird/issues/69


El miércoles, 5 de abril de 2017, 20:27:11 (UTC-3), Tim Graham escribió:
>
> I'd suggest to use git bisect to find the Django commit where the behavior 
> changed:
>
> https://docs.djangoproject.com/en/dev/internals/contributing/triaging-tickets/#bisecting-a-regression
>
 

>
>
> On Wednesday, April 5, 2017 at 5:57:03 PM UTC-4, Maximiliano Robaina wrote:
>>
>> Hi,
>>
>>
>> I'm the maintener of django-firebird database backend.
>>
>> I've an strange behavior (a different result) when reading a value from a 
>> DecimalField between Django 1.7 and Django 1.8 and I'm trying to figured 
>> out where is my mistake.
>>
>> I've a django project which I update from 1.7 to 1.8.
>> In this project I've a model with a DecimalField.  (by example, price 
>> = DecimalField(max_digits=18, decimal_places=2)
>>
>> Then, in django 1.7
>>
>> >> obj = ModelWithDecimalField.objects.first()
>> >> obj.price
>> >>  Decimal("1990.00")
>>
>> In django 1.8
>>
>> >> obj = ModelWithDecimalField.objects.first()
>> >> obj.price
>> >>  Decimal("1990")
>>
>>
>> So, I don't know where look to found this change. What change between 1.7 
>> and 1.8
>> Maybe I forget to implement something in django-firebird 1.8
>>
>>
>> Thanks in advance
>> Maxi
>>
>>
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8dcbecbc-4edb-4eb8-b019-d659df7c7921%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

DJANGO_SETTINGS_FILE

[James Pic]

Hi all!

Life with DJANGO_SETTINGS_MODULE has been great. However, it requires the
settings to be an importable module. In most cases, this is a straight
forward "simple" python script.

Sometimes it looks like it could make sense to add an environment variable
that could use a file that is not importable as a python module, ie.:

DJANGO_SETTINGS_FILE=/etc/yourapp/production.py
DJANGO_SETTINGS_FILE=~/yourapp_production.py
DJANGO_SETTINGS_FILE=settings.py  # relative to pwd

Then, in this file, we could override the default project settings there
without having to put the new file in an importable module, ie. in
/etc/yourapp/production.py:

from yourapp.settings import *
DATABASES=...
CACHES=...
SECRET_KEY=...

I'm sure this has been discussed before but didn't find anything by
searching DJANGO_SETTINGS_FILE in the ML.

What's your opinion on this ? Is it time to upgrade this ?

Best,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAC6Op19-0JeL8Jdcq70-pvwz%2Bi2hfKtopEvihca_rGfH1DrPww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding generated common table expressions

[Anssi Kääriäinen]

On Thursday, April 6, 2017 at 11:53:32 AM UTC+3, Marc Tamlyn wrote:
>
> Regarding Anssi's comments about SubQuery, we do now have that in core as 
> of 1.11 [0]. It does look like an .attach() approach might actually have 
> been a nicer version of this, but on the other hand it's currently 
> implementable solely with the Expressions API. It seems like the OuterRef 
> is very similar to your queryset.ref(). An even nicer approach using attach 
> could be to say qs.attach(q1=some_qs).filter(a=F('q1__b'))?
>

Hmmh, we have one form of SubQuery, but that's actually for SELECT clause, 
not for FROM clause. I believe the same class won't work for the CTE or 
subquery in FROM clause case.

As for the attach(), seems like a really nice syntax. We do need something 
for generating the join clause for the JOIN. If you look at an example:
q1 = Book.objects.values('author_id').annotate(avg_price=Avg('price'))
q2 = Author.objects.attach(q1=q1)
it needs to create something like:
WITH q1 AS (
SELECT author_id, avg(price) FROM book GROUP BY author_id
)
SELECT author.id, author.name
   FROM author
   LEFT JOIN q1 ON author.id = q1.author_id;

Or, equivalently without the CTE:

SELECT author.id, author.name
   FROM author
   LEFT JOIN ( SELECT author_id, avg(price) FROM book GROUP BY author_id) 
ON author.id = q1.author_id;

Now, the main points are:
   1. There is no need to design this to be about CTEs. That just limits 
the feature from backends that don't have CTEs without any real benefit. 
>From Django's perspective the two above queries are the same.
   2. We do need something for the JOIN ON condition. In some cases Django 
could guess this, but there needs to be an explicit way to express the join 
condition.

If we allow usage of expressions from the attached queryset, but don't try 
to go for cases where model instance are created from the attached 
queryset, this will be both possible to implement without having to write a 
change-everything patch, and this will also be a really nice feature.

For recursive CTEs, I'd leave that strictly as a later step. The only thing 
we need to check right now is that we don't do something that prevents a 
good recursive CTEs implementation later on.

 - Anssi

>
> Looking forwards to seeing a DEP!
>
> [0] 
> https://docs.djangoproject.com/en/1.11/ref/models/expressions/#subquery-expressions
>
> On 22 March 2017 at 01:32, Ashley Waite  > wrote:
>
>> Here's the code changes I've made, noting that some of them were to shove 
>> in a generalised VALUES clause that mocks being a queryset, so that it 
>> plays with the same interface.
>>
>>
>> https://github.com/django/django/compare/master...ashleywaite:cte-dev#files_bucket
>>
>> I've had a glance at cte-trees/cte-forest and once general CTEs are 
>> worked out expanding that to include recursive CTEs wouldn't be too 
>> difficult, and that would greatly simplify the implementation of cte-forest 
>> to the extent that it might be viable as a django data/reference type.
>>
>> - Ashley
>>
>>
>> On Saturday, March 18, 2017 at 8:28:53 PM UTC+11, Josh Smeaton wrote:
>>>
>>> Thanks for bringing this up Ashley, and for all of the detail you 
>>> provided. I'd certainly like to see CTEs make their way into Django, 
>>> provided we could come up with a nice enough API. From the look of it, 
>>> you've already got something that works with an okay API so I'm hopeful.
>>>
>>> I'd be very interested in seeing your POC too if you're able to share.
>>>
>>> From looking very briefly at django-cte-trees it doesn't aim to support 
>>> user defined CTEs for anything other than recursive queries. I'd be 
>>> interested in seeing, as part of a DEP, how CTE inclusion in django core 
>>> could support the cte-trees project from an API perspective.
>>>
>>> On Friday, 17 March 2017 22:28:17 UTC+11, Ashley Waite wrote:

 Hey all,


 I'd like to suggest adding Common Table Expression (CTE) query 
 generation as a feature to Django.

 I've been working on a project that required manipulation of many 
 records at once, and as with many ORMs found that this wasn't an ideal 
 use-case in Django. As the rest of our code base and related projects are 
 in Django, there was a strong preference to find a way to do it and keep 
 to 
 the same model-is-the-truth design.

 I first did this by writing some hackish functions using raw querysets 
 and generating my own CTE based queries, but it lacked ideal flexibility 
 and maintainability. So I've now written some modifications into my Django 
 to do this in a more Django-esque way and think that this functionality 
 would be beneficial within the project itself, but am unsure exactly where 
 to start the conversation about that.


 *Why generate CTE based queries from querysets?*

 By allowing querysets to be attached to each other, and setting 
 appropriate WHERE clauses, 

Re: Adding generated common table expressions

[Ashley Waite]

Oh right, I feel silly for not thinking of named arguments as a resolution
there, that's a damn good idea!

It resolves a couple of minor issues I'd anticipated in a complete
implementation, so consider the idea stolen!  ;)

I mean, ummm, that was totally what I had planned!

- Ashley

On 06/04/2017 6:53 PM, "Marc Tamlyn"  wrote:

> Hi!
>
> This looks generally very good, and I'm quite excited about it.
>
> In terms of taking it forwards, I think a DEP is a very good idea, and
> there are at least 3 core devs who are keen to see a solution. Whether you
> have the right solution yet I'm not so sure, but it sounds like you're not
> either.
>
> Regarding Anssi's comments about SubQuery, we do now have that in core as
> of 1.11 [0]. It does look like an .attach() approach might actually have
> been a nicer version of this, but on the other hand it's currently
> implementable solely with the Expressions API. It seems like the OuterRef
> is very similar to your queryset.ref(). An even nicer approach using attach
> could be to say qs.attach(q1=some_qs).filter(a=F('q1__b'))?
>
> Looking forwards to seeing a DEP!
>
> [0] https://docs.djangoproject.com/en/1.11/ref/
> models/expressions/#subquery-expressions
>
> On 22 March 2017 at 01:32, Ashley Waite  wrote:
>
>> Here's the code changes I've made, noting that some of them were to shove
>> in a generalised VALUES clause that mocks being a queryset, so that it
>> plays with the same interface.
>>
>> https://github.com/django/django/compare/master...ashleywait
>> e:cte-dev#files_bucket
>>
>> I've had a glance at cte-trees/cte-forest and once general CTEs are
>> worked out expanding that to include recursive CTEs wouldn't be too
>> difficult, and that would greatly simplify the implementation of cte-forest
>> to the extent that it might be viable as a django data/reference type.
>>
>> - Ashley
>>
>>
>> On Saturday, March 18, 2017 at 8:28:53 PM UTC+11, Josh Smeaton wrote:
>>>
>>> Thanks for bringing this up Ashley, and for all of the detail you
>>> provided. I'd certainly like to see CTEs make their way into Django,
>>> provided we could come up with a nice enough API. From the look of it,
>>> you've already got something that works with an okay API so I'm hopeful.
>>>
>>> I'd be very interested in seeing your POC too if you're able to share.
>>>
>>> From looking very briefly at django-cte-trees it doesn't aim to support
>>> user defined CTEs for anything other than recursive queries. I'd be
>>> interested in seeing, as part of a DEP, how CTE inclusion in django core
>>> could support the cte-trees project from an API perspective.
>>>
>>> On Friday, 17 March 2017 22:28:17 UTC+11, Ashley Waite wrote:

 Hey all,


 I'd like to suggest adding Common Table Expression (CTE) query
 generation as a feature to Django.

 I've been working on a project that required manipulation of many
 records at once, and as with many ORMs found that this wasn't an ideal
 use-case in Django. As the rest of our code base and related projects are
 in Django, there was a strong preference to find a way to do it and keep to
 the same model-is-the-truth design.

 I first did this by writing some hackish functions using raw querysets
 and generating my own CTE based queries, but it lacked ideal flexibility
 and maintainability. So I've now written some modifications into my Django
 to do this in a more Django-esque way and think that this functionality
 would be beneficial within the project itself, but am unsure exactly where
 to start the conversation about that.


 *Why generate CTE based queries from querysets?*

 By allowing querysets to be attached to each other, and setting
 appropriate WHERE clauses, arbitrary and nested SQL queries can be
 generated. Where the results of the queries are only necessary for the
 execution of following queries this saves a very substantial amount of time
 and database work. Once these features exist, other functionality can also
 transparently use these to generate more efficient queries (such as large
 IN clauses).

 This allows several powerful use cases I think Django would benefit
 from:


 *Large 'IN' clauses*, can be implemented as CTEs reducing expensive
 lookups to a single CTE INNER JOIN. For sets of thousands to match from
 tables of millions of records this can be a very substantial gain.


 *Composite 'IN' conditions,* where multiple fields must match and
 you're matching against a large set of condition rows. In my usage this was
 "where the md5/sha hashes match one of the million md5/sha tuples in my
 match set". This is simply a CTE JOIN with two clauses in the WHERE.


 *Nested data creation*, where the parent doesn't yet exist. Django
 doesn't currently do this as the primary keys are needed, and this makes

Re: Adding generated common table expressions

[Marc Tamlyn]

Hi!

This looks generally very good, and I'm quite excited about it.

In terms of taking it forwards, I think a DEP is a very good idea, and
there are at least 3 core devs who are keen to see a solution. Whether you
have the right solution yet I'm not so sure, but it sounds like you're not
either.

Regarding Anssi's comments about SubQuery, we do now have that in core as
of 1.11 [0]. It does look like an .attach() approach might actually have
been a nicer version of this, but on the other hand it's currently
implementable solely with the Expressions API. It seems like the OuterRef
is very similar to your queryset.ref(). An even nicer approach using attach
could be to say qs.attach(q1=some_qs).filter(a=F('q1__b'))?

Looking forwards to seeing a DEP!

[0]
https://docs.djangoproject.com/en/1.11/ref/models/expressions/#subquery-expressions

On 22 March 2017 at 01:32, Ashley Waite  wrote:

> Here's the code changes I've made, noting that some of them were to shove
> in a generalised VALUES clause that mocks being a queryset, so that it
> plays with the same interface.
>
> https://github.com/django/django/compare/master...
> ashleywaite:cte-dev#files_bucket
>
> I've had a glance at cte-trees/cte-forest and once general CTEs are worked
> out expanding that to include recursive CTEs wouldn't be too difficult, and
> that would greatly simplify the implementation of cte-forest to the extent
> that it might be viable as a django data/reference type.
>
> - Ashley
>
>
> On Saturday, March 18, 2017 at 8:28:53 PM UTC+11, Josh Smeaton wrote:
>>
>> Thanks for bringing this up Ashley, and for all of the detail you
>> provided. I'd certainly like to see CTEs make their way into Django,
>> provided we could come up with a nice enough API. From the look of it,
>> you've already got something that works with an okay API so I'm hopeful.
>>
>> I'd be very interested in seeing your POC too if you're able to share.
>>
>> From looking very briefly at django-cte-trees it doesn't aim to support
>> user defined CTEs for anything other than recursive queries. I'd be
>> interested in seeing, as part of a DEP, how CTE inclusion in django core
>> could support the cte-trees project from an API perspective.
>>
>> On Friday, 17 March 2017 22:28:17 UTC+11, Ashley Waite wrote:
>>>
>>> Hey all,
>>>
>>>
>>> I'd like to suggest adding Common Table Expression (CTE) query
>>> generation as a feature to Django.
>>>
>>> I've been working on a project that required manipulation of many
>>> records at once, and as with many ORMs found that this wasn't an ideal
>>> use-case in Django. As the rest of our code base and related projects are
>>> in Django, there was a strong preference to find a way to do it and keep to
>>> the same model-is-the-truth design.
>>>
>>> I first did this by writing some hackish functions using raw querysets
>>> and generating my own CTE based queries, but it lacked ideal flexibility
>>> and maintainability. So I've now written some modifications into my Django
>>> to do this in a more Django-esque way and think that this functionality
>>> would be beneficial within the project itself, but am unsure exactly where
>>> to start the conversation about that.
>>>
>>>
>>> *Why generate CTE based queries from querysets?*
>>>
>>> By allowing querysets to be attached to each other, and setting
>>> appropriate WHERE clauses, arbitrary and nested SQL queries can be
>>> generated. Where the results of the queries are only necessary for the
>>> execution of following queries this saves a very substantial amount of time
>>> and database work. Once these features exist, other functionality can also
>>> transparently use these to generate more efficient queries (such as large
>>> IN clauses).
>>>
>>> This allows several powerful use cases I think Django would benefit from:
>>>
>>>
>>> *Large 'IN' clauses*, can be implemented as CTEs reducing expensive
>>> lookups to a single CTE INNER JOIN. For sets of thousands to match from
>>> tables of millions of records this can be a very substantial gain.
>>>
>>>
>>> *Composite 'IN' conditions,* where multiple fields must match and
>>> you're matching against a large set of condition rows. In my usage this was
>>> "where the md5/sha hashes match one of the million md5/sha tuples in my
>>> match set". This is simply a CTE JOIN with two clauses in the WHERE.
>>>
>>>
>>> *Nested data creation*, where the parent doesn't yet exist. Django
>>> doesn't currently do this as the primary keys are needed, and this makes
>>> normalised data structures unappealing. Using INSERTs as CTEs that supply
>>> those keys to following statements means that entire nested data structures
>>> of new information can be recreated in the database at once, efficiently
>>> and atomically.
>>>
>>>
>>> *Non-uniform UPDATE*s, such that a modified set of objects can all be
>>> updated with different data at the same time by utilising a CTE values
>>> statement JOINed to the UPDATE statement. As there's